a bg9@stddlZddlZddlmZmZmZmZmZddlm Z m Z ddl m Z ddl mZddlmZGddde ZdS) N)AnyDictListOptionalcast) AgentAction AgentFinish)BaseCallbackHandler) LLMResult)parsecseZdZUdZdZeed<edZeed<dZeed<dZ eed <d3ee ee ee ed d fd d Z e ee feee d dddZee d dddZee d dddZee d dddZe ee fe ee fe d dddZe ee fe d dddZee d dd d!Ze ee fee d d"d#d$Zee e d%d&d'Zd4e e ee ee d d(d)d*Zee d dd+d,Zee d d-d.d/Zee d d0d1d2ZZ S)5ArgillaCallbackHandleraCallback Handler that logs into Argilla. Args: dataset_name: name of the `FeedbackDataset` in Argilla. Note that it must exist in advance. If you need help on how to create a `FeedbackDataset` in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html. workspace_name: name of the workspace in Argilla where the specified `FeedbackDataset` lives in. Defaults to `None`, which means that the default workspace will be used. api_url: URL of the Argilla Server that we want to use, and where the `FeedbackDataset` lives in. Defaults to `None`, which means that either `ARGILLA_API_URL` environment variable or the default will be used. api_key: API Key to connect to the Argilla Server. Defaults to `None`, which means that either `ARGILLA_API_KEY` environment variable or the default will be used. Raises: ImportError: if the `argilla` package is not installed. ConnectionError: if the connection to Argilla fails. FileNotFoundError: if the `FeedbackDataset` retrieval from Argilla fails. Examples: >>> from langchain_community.llms import OpenAI >>> from langchain_community.callbacks import ArgillaCallbackHandler >>> argilla_callback = ArgillaCallbackHandler( ... dataset_name="my-dataset", ... workspace_name="my-workspace", ... api_url="http://localhost:6900", ... api_key="argilla.apikey", ... ) >>> llm = OpenAI( ... temperature=0, ... callbacks=[argilla_callback], ... verbose=True, ... openai_api_key="API_KEY_HERE", ... ) >>> llm.generate([ ... "What is the best NLP-annotation tool out there? (no bias at all)", ... ]) "Argilla, no doubt about it." z%https://github.com/argilla-io/argillaREPO_URLz/issues ISSUES_URLzphttps://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.htmlBLOG_URLzhttp://localhost:6900DEFAULT_API_URLN) dataset_nameworkspace_nameapi_urlapi_keyreturnc sptzddl}|j|_Wnty8tdYn0t|jtdkr^td|jd|durtddurt d|j d |j }|durtd durt|jtd krd nd |_ t d|j d|j }z|j ||dWn@ty,}z&td|d|jd|WYd}~n d}~00||_|p@||_zXi}t|jtdkr|t d|jdtddi}|jjf|j|jd||_WnXty}z>td|d|jd|jd|jd|jd |WYd}~n d}~00dd g}|d!d"|jjDkrRtd#|jd|jd$|d%d&d"|jjDd'|jd( i|_t d)|jddS)*a)Initializes the `ArgillaCallbackHandler`. Args: dataset_name: name of the `FeedbackDataset` in Argilla. Note that it must exist in advance. If you need help on how to create a `FeedbackDataset` in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html. workspace_name: name of the workspace in Argilla where the specified `FeedbackDataset` lives in. Defaults to `None`, which means that the default workspace will be used. api_url: URL of the Argilla Server that we want to use, and where the `FeedbackDataset` lives in. Defaults to `None`, which means that either `ARGILLA_API_URL` environment variable or the default will be used. api_key: API Key to connect to the Argilla Server. Defaults to `None`, which means that either `ARGILLA_API_KEY` environment variable or the default will be used. Raises: ImportError: if the `argilla` package is not installed. ConnectionError: if the connection to Argilla fails. FileNotFoundError: if the `FeedbackDataset` retrieval from Argilla fails. rNzTo use the Argilla callback manager you need to have the `argilla` Python package installed. Please install it with `pip install argilla`z1.8.0z#The installed `argilla` version is z} but `ArgillaCallbackHandler` requires at least version 1.8.0. Please upgrade `argilla` with `pip install --upgrade argilla`.ZARGILLA_API_URLz[Since `api_url` is None, and the env var `ARGILLA_API_URL` is not set, it will default to `z6`, which is the default API URL in Argilla Quickstart.ZARGILLA_API_KEYz1.11.0z admin.apikeyz owner.apikeyz[Since `api_key` is None, and the env var `ARGILLA_API_KEY` is not set, it will default to `z6`, which is the default API key in Argilla Quickstart.)rrz.Could not connect to Argilla with exception: 'z'. Please check your `api_key` and `api_url`, and make sure that the Argilla server is up and running. If the problem persists please report it to z as an `integration` issue.1.14.0zYou have Argilla z., but Argilla 1.14.0 or higher is recommended.Z with_recordsF)nameZ workspacez@`FeedbackDataset` retrieval from Argilla failed with exception `z+`. Please check that the dataset with name=z in the workspace=zz exists in advance. If you need help on how to create a `langchain`-compatible `FeedbackDataset` in Argilla, please visit z.. If the problem persists please report it to promptresponsecSsg|] }|jqSr.0fieldrr|/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/callbacks/argilla_callback.py z3ArgillaCallbackHandler.__init__..z`FeedbackDataset` with name=z] had fields that are not supported yet for the`langchain` integration. Supported fields are: z/, and the current `FeedbackDataset` fields are cSsg|] }|jqSrrrrrrr r!zl. For more information on how to create a `langchain`-compatible `FeedbackDataset` in Argilla, please visit .zThe `ArgillaCallbackHandler` is currently in beta and is subject to change based on updates to `langchain`. Please report any issues to )super__init__Zargilla __version__ARGILLA_VERSION ImportErrorr osgetenvwarningswarnrZDEFAULT_API_KEYinit ExceptionConnectionErrorrrZ get_workspacer UserWarningZFeedbackDatasetZ from_argilladatasetFileNotFoundErrorrfields ValueErrorprompts) selfrrrrZrge extra_argsZsupported_fields __class__rrr$=s          zArgillaCallbackHandler.__init__) serializedr4kwargsrcKs$|jt|dp|d|idS)z.Save the prompts in memory when an LLM starts. parent_run_idrun_idN)r4updatestr)r5r:r4r;rrr on_llm_startsz#ArgillaCallbackHandler.on_llm_start)tokenr;rcKsdS)z)Do nothing when a new token is generated.Nr)r5rAr;rrron_llm_new_tokensz'ArgillaCallbackHandler.on_llm_new_token)rr;rc s|dr dS|jt|d}t||jD]$\}|jjfdd|Ddq*|jt|dt|jtdkr|j dS)z(Log records to Argilla when an LLM ends.r<Nr=cs g|]}d|jdiqS)r2rr)textstrip)rZ generationrrrr s z5ArgillaCallbackHandler.on_llm_end..recordsr) r4r?zip generationsr0 add_recordspopr r&push_to_argilla)r5rr;r4rJrrFr on_llm_ends  z!ArgillaCallbackHandler.on_llm_end)errorr;rcKsdS)z%Do nothing when LLM outputs an error.Nrr5rOr;rrr on_llm_errorsz#ArgillaCallbackHandler.on_llm_error)r:inputsr;rcKsHd|vrD|jt|dp|dt|dtr6|dn|dgidS)aIf the key `input` is in `inputs`, then save it in `self.prompts` using either the `parent_run_id` or the `run_id` as the key. This is done so that we don't log the same input prompt twice, once when the LLM starts and once when the chain starts. inputr<r=N)r4r>r? isinstancelist)r5r:rRr;rrron_chain_starts z%ArgillaCallbackHandler.on_chain_start)outputsr;rc s.tfddt|dt|dfDs.dSjt|dp\ttjt|dg}|D]V\}}t|trj j ddt ||Ddqfj j d d || d igdqft|djvrjt|dt|djvr jt|dtjtd kr*j dS) zIf either the `parent_run_id` or the `run_id` is in `self.prompts`, then log the outputs to Argilla, and pop the run from `self.prompts`. The behavior differs if the output is a list or not. c3s|]}|jvVqdS)N)r4)rkeyr5rr sz6ArgillaCallbackHandler.on_chain_end..r<r=NcSs&g|]\}}d||ddiqS)r2rDrC)rE)rroutputrrrr s  z7ArgillaCallbackHandler.on_chain_end..rGr2 rCr)anyr?r4getrritemsrTrUr0rKrIjoinrErLr r&rM)r5rWr;r4Zchain_output_keyZchain_output_valrrYr on_chain_ends8    z#ArgillaCallbackHandler.on_chain_endcKsdS)z+Do nothing when LLM chain outputs an error.NrrPrrron_chain_error8sz%ArgillaCallbackHandler.on_chain_error)r: input_strr;rcKsdS)zDo nothing when tool starts.Nr)r5r:rcr;rrr on_tool_start<sz$ArgillaCallbackHandler.on_tool_start)actionr;rcKsdS)z.Do nothing when agent takes a specific action.Nr)r5rer;rrron_agent_actionEsz&ArgillaCallbackHandler.on_agent_action)r[observation_prefix llm_prefixr;rcKsdS)zDo nothing when tool ends.Nr)r5r[rgrhr;rrr on_tool_endIsz"ArgillaCallbackHandler.on_tool_endcKsdS)z&Do nothing when tool outputs an error.NrrPrrr on_tool_errorSsz$ArgillaCallbackHandler.on_tool_error)rDr;rcKsdSz Do nothingNr)r5rDr;rrron_textWszArgillaCallbackHandler.on_text)finishr;rcKsdSrkr)r5rmr;rrron_agent_finish[sz&ArgillaCallbackHandler.on_agent_finish)NNN)NN)!__name__ __module__ __qualname____doc__r r?__annotations__rrrrr$rrrr@rBr rN BaseExceptionrQrVrarbrdrrfrirjrlrrn __classcell__rrr8rr sZ +      2   r )r(r*typingrrrrrZlangchain_core.agentsrrZlangchain_core.callbacksr Zlangchain_core.outputsr Zpackaging.versionr r rrrrs