a
bg76 @ s d dl Z d dlZd dlmZ d dlmZ d dlmZmZmZm Z m
Z
m Z d dl m
Z
d dlmZmZ d dlmZ d dlmZmZ d d lmZmZ G d
d d eZe eef e
eef d d
dZG dd deZ dS ) N)datetime)Enum)AnyDictListOptionalTupleUnion)UUID) AgentAction AgentFinish)BaseCallbackHandler) BaseMessage ChatMessage)
Generation LLMResultc @ s e Zd ZdZdZdZdS )LabelStudioModez Label Studio mode enumerator.promptZchatN)__name__
__module__ __qualname____doc__PROMPTCHAT r r /var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/callbacks/labelstudio_callback.pyr s r )modereturnc C s4 t jjdt jjdi}t| tr&t | } || j | fS )zGet default Label Studio configs for the given mode.
Parameters:
mode: Label Studio mode ("prompt" or "chat")
Returns: Tuple of Label Studio config and mode
a
a
)r r valuer
isinstancestr)r Z_default_label_configsr r r get_default_label_configs s )
r! c
s e Zd ZU dZdZeed< dddedejfe e e e e e
ee e e eef d fdd
Z ee
e
e ddd d
Zeeef e
e edd d d
ZeedddZddddeeef e
e
e ee e e e
e e eeef eedddZeeddddZeeddddZeeddd d Zeeef eeef edd d d Z eeef edd!d"d#Z eeddd$d%Z eeef eedd&d'd(Z e eed)d*d+Z!d7ee e e e edd,d-d.Z"eeddd/d0Z#eedd1d2d3Z$e%edd4d5d6Z& Z'S )8LabelStudioCallbackHandlera Label Studio callback handler.
Provides the ability to send predictions to Label Studio
for human evaluation, feedback and annotation.
Parameters:
api_key: Label Studio API key
url: Label Studio URL
project_id: Label Studio project ID
project_name: Label Studio project name
project_config: Label Studio project config (XML)
mode: Label Studio mode ("prompt" or "chat")
Examples:
>>> from langchain_community.llms import OpenAI
>>> from langchain_community.callbacks import LabelStudioCallbackHandler
>>> handler = LabelStudioCallbackHandler(
... api_key='',
... url='http://localhost:8080',
... project_name='LangChain-%Y-%m-%d',
... mode='prompt'
... )
>>> llm = OpenAI(callbacks=[handler])
>>> llm.invoke('Tell me a story about a dog.')
zLangChain-%Y-%m-%dDEFAULT_PROJECT_NAMEN)api_keyurl
project_id project_nameproject_configr c
sT t z dd l}W n& ty< td| jj dY n0 |sztdr\ttd}n t d| jj d| jj d|| _
|stdrtd}n$t d|j
d | jj d
|j
}|| _i | _|j| j| j
d | _|| _|r|| _d | _nt|\| _| _|ptd | _| jd ur:| jt| j| _nXt | j}| jj |d
} | rt| d | _| jj | _n | jj || jd| _| jj | _| jj | _ d\| _!| _"| _#| _$| j % D ]V\}
} | d dkr|
| _!| d d | _"| d d d | _#| d d d | _$ qq| j!sPd| j d} | jt&j'kr@| d7 } n| d7 } t | d S )Nr z
You're using z in your code, but you don't have the LabelStudio SDK Python package installed or upgraded to the latest version. Please run `pip install -U label-studio-sdk` before using this callback.ZLABEL_STUDIO_API_KEYz in your code, Label Studio API key is not provided. Please provide Label Studio API key: go to the Label Studio instance, navigate to Account & Settings -> Access Token and copy the key. Use the key as a parameter for the callback: zr(label_studio_api_key='', ...) or set the environment variable LABEL_STUDIO_API_KEY=ZLABEL_STUDIO_URLz5Label Studio URL is not provided, using default URL: z8If you want to provide your own URL, use the parameter: zj(label_studio_url='', ...) or set the environment variable LABEL_STUDIO_URL=)r% r$ ZLABEL_STUDIO_PROJECT_ID)title)r) Z label_config)NNNNtypeZTextAreato_nameinputsr zLabel Studio project "zI" does not have a TextArea tag. Please add a TextArea tag to the project.z
HINT: go to project Settings -> Labeling Interface -> Browse Templates and select "Generative AI -> Supervised Language Model Fine-tuning" template.z
HINT: go to project Settings -> Labeling Interface -> Browse Templates and check available templates under "Generative AI" section.)(super__init__Zlabel_studio_sdk ImportError __class__r osgetenvr
ValueErrorr$ warningswarnZLABEL_STUDIO_DEFAULT_URLr% payloadZClientZ ls_clientr' r( r r! r& Z get_projectint
ls_projectr todaystrftimeZ get_projectsidZcreate_projectZparsed_label_config from_namer+ r
input_typeitemsr r )
selfr$ r% r& r' r( r ZlsZ
project_titleZexisting_projectsZtag_nameZtag_info
error_messager0 r r r. j s
z#LabelStudioCallbackHandler.__init__)run_id generationsr c
C s g }| j | d }| j | d di d}t||D ]H\}}|| j|d|i| j| jdddd |D id
g|d gd q8| j| d S )
NpromptskwargsZinvocation_paramsZ
model_namerB Ztextareatextc S s g | ]
}|j qS r )rF ).0gr r r
zFLabelStudioCallbackHandler.add_prompts_generations..)r< r+ r* r )result
model_version)dataZ predictions) r6 getzipappendr r< r+ r8 Z import_tasks)r? rB rC tasksrD rL r Z
generationr r r add_prompts_generations s2 z2LabelStudioCallbackHandler.add_prompts_generations)
serializedrD rE r c K sD | j dkr$td| j d| j dt|d }||d| j|< dS ).Save the prompts in memory when an LLM starts.Text
Label Studio project "" has an input type . To make it work with the mode="chat", the input type should be .
Read more here https://labelstud.io/tags/textrB )rD rE N)r= r3 r' r r6 )r? rS rD rE rB r r r on_llm_start s
z'LabelStudioCallbackHandler.on_llm_start)messager c C s t |tr|jS |jjS dS )z Get the role of the message.N)r r roler0 r )r? rY r r r _get_message_role s
z,LabelStudioCallbackHandler._get_message_role)
parent_run_idtagsmetadata)rS messagesrB r\ r] r^ rE r c K s | j dkr$td| j d| j dg }|D ]4} g }
| D ] } |
| | | jd q8||
q,||||||d| jt|< dS )rT Z
ParagraphsrV rW z>. To make it work with the mode="chat", the input type should be .
Read more here https://labelstud.io/tags/paragraphs)rZ content)rD r] r^ rB r\ rE N)r= r3 r' rP r[ r` r6 r ) r? rS r_ rB r\ r] r^ rE rD Z message_listZdialogrY r r r on_chat_model_start s.
z.LabelStudioCallbackHandler.on_chat_model_start)tokenrE r c K s dS )z)Do nothing when a new token is generated.Nr )r? rb rE r r r on_llm_new_tokenG s z+LabelStudioCallbackHandler.on_llm_new_token)responserE r c K s* t |d }| ||j | j| dS )z>Create a new Label Studio task for each prompt and generation.rB N)r rR rC r6 pop)r? rd rE rB r r r
on_llm_endK s z%LabelStudioCallbackHandler.on_llm_end)errorrE r c K s dS )z%Do nothing when LLM outputs an error.Nr r? rg rE r r r on_llm_errorU s z'LabelStudioCallbackHandler.on_llm_error)rS r, rE r c K s d S Nr )r? rS r, rE r r r on_chain_startY s z)LabelStudioCallbackHandler.on_chain_start)outputsrE r c K s d S rj r )r? rl rE r r r on_chain_end^ s z'LabelStudioCallbackHandler.on_chain_endc K s dS )z+Do nothing when LLM chain outputs an error.Nr rh r r r on_chain_errora s z)LabelStudioCallbackHandler.on_chain_error)rS input_strrE r c K s dS )z Do nothing when tool starts.Nr )r? rS ro rE r r r
on_tool_starte s z(LabelStudioCallbackHandler.on_tool_start)actionrE r c K s dS )z.Do nothing when agent takes a specific action.Nr )r? rq rE r r r on_agent_actionn s z*LabelStudioCallbackHandler.on_agent_action)outputobservation_prefix
llm_prefixrE r c K s dS )zDo nothing when tool ends.Nr )r? rs rt ru rE r r r on_tool_endr s z&LabelStudioCallbackHandler.on_tool_endc K s dS )z&Do nothing when tool outputs an error.Nr rh r r r
on_tool_error| s z(LabelStudioCallbackHandler.on_tool_error)rF rE r c K s dS z
Do nothingNr )r? rF rE r r r on_text s z"LabelStudioCallbackHandler.on_text)finishrE r c K s dS rx r )r? rz rE r r r on_agent_finish s z*LabelStudioCallbackHandler.on_agent_finish)NN)(r r r r r# r __annotations__r r r r7 r r. r r rR r r rX r r[ r
ra rc r rf
BaseExceptionri rk rm rn rp r rr rv rw ry r r{
__classcell__r r rA r r" N s
y %
)
r" ) r1 r4 r enumr typingr r r r r r uuidr
Zlangchain_core.agentsr r Zlangchain_core.callbacksr
Zlangchain_core.messagesr r Zlangchain_core.outputsr r r r r! r" r r r r s
9