a ag@sFdZddlmZddlZddlZddlZddlZddlZddl Z ddl m Z m Z ddl m Z mZmZmZmZmZmZmZmZddlmZddlmZddlmZdd lmZmZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%dd l"m&Z'dd l"m(Z)ddl*m+Z+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2m3Z3ddl4m5Z5m6Z6ddl4m7Z8ddl9m:Z:m;Z;ddlZ>m?Z?m@Z@mAZAddlBmCZCddlDmEZEddlFmGZGddlHmIZIddlJmKZKddlLmMZMmNZNmOZOddlPmQZRdd lSm&ZTddlSmUZUmVZVe rddlWZXeYeZZ[eegeeIe#ffeee\gefe#eIfZ]eegeeIe#ffefZ^Gddde_Z`Gd d!d!e\ZaGd"d#d#e\Zbdd%d&d'd(d)d*Zcd+d&d,d-d.ZdGd/d0d0eGZed+d1d,d2d3Zfd4d5d6d7d8d9Zgd4d:d5d6d;dd?d@Zid'dAdBdCdDdEdFdGZjdHdIdJdKdLdMZkdHdIdJdNdOdPZldHdIdJdQdRdSZmdTdUd&dCdIdJdJdJdVdW dXdYZndHdIdIdIdZd[d\d]ZodHd&dCdIdIdId^d_d`daZpdddddbdcd+dIddd5dedfdgdhdiZqddddjdkd+dddId5dedldmdndoZrddpd4dqd'd5drdsdtduZsddddjdcd+dddId5dedfdvdwdxZtddddjdkd+dddId5dedydmdzd{Zuddpd4dqd'd5drdsd|d}Zvdd~d&d%d&dedIdddddZwGdddeGddZxejyGdddZzddddZ{dd6dddZ|dZ}dddddddddd&d%dBdddJdeddJdd+d ddZ~dddddddddd&d%dBdddJdeddJdd+d ddZdZee_edde~_dS)z>Utilities for running language models or Chains over datasets.) annotationsN)datetimetimezone) TYPE_CHECKINGAnyCallableDictListOptionalTupleUnioncast)warn_deprecated) Callbacks)BaseLanguageModel) BaseMessagemessages_from_dict) ChatResult LLMResult)RunnableRunnableConfigRunnableLambdaconfig)utils)EvaluatorCallbackHandlerwait_for_all_evaluators)LangChainTracer)Client) get_git_infoget_langchain_env_var_metadata)EvaluationResult RunEvaluator) run_evaluator) as_runnableis_traceable_function)DatasetDataTypeExampleRun TracerSession)LangSmithError) HTTPError) TypedDict)Chain)load_evaluator) EvaluatorTypePairwiseStringEvaluatorStringEvaluator) evaluation)name_generationprogressc@seZdZdZdS)InputFormatErrorz(Raised when the input format is invalid.N)__name__ __module__ __qualname____doc__r;r;u/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain/smith/evaluation/runner_utils.pyr6Nsr6c@s,eZdZdZddddZddddZdS) TestResultz1A dictionary of the results of a single test run. pd.DataFramereturncCs.|}dd|jD}|jddj|ddS)zReturn quantiles for the feedback scores. This method calculates and prints the quantiles for the feedback scores across all feedback keys. Returns: A DataFrame containing the quantiles for each feedback key. cSs6g|].}|ds.|ds.|dvs.|dr|qS)inputs.outputs.>inputoutput reference) startswith).0colr;r;r< es    z5TestResult.get_aggregate_feedback..all)include)Zaxis) to_dataframecolumnsZdescribeZdrop)selfZdfZto_dropr;r;r<get_aggregate_feedbackXs z!TestResult.get_aggregate_feedbackc CsTz ddl}Wn.ty:}ztd|WYd}~n d}~00g}g}|dD]\}}|d}|d}t|trdd|D} n|duri} nd|i} id d|d D| } d |vrt|d tr| d d|d Dn |d | d <| id d|D|d|d|dd|| ||qP|j||dS)z#Convert the results to a dataframe.rNzfPandas is required to convert the results to a dataframe. to install pandas, run `pip install pandas`.resultsfeedbackrDcSsi|]\}}d||qS)rBr;rGkvr;r;r< z+TestResult.to_dataframe..cSsi|]\}}d||qS)rAr;rSr;r;r<rVrWrCrEcSsi|]\}}d||qS)z reference.r;rSr;r;r<rVrWcSsi|]}d|j|jqS)z feedback.)keyZscore)rGfr;r;r<rVrWErrorexecution_timerun_id)errorr[r\)index) pandas ImportErroritemsget isinstancedictupdateappendZ DataFrame) rOpdeindicesrecords example_idresultrRZoutput_rDrr;r;r<rMosR       zTestResult.to_dataframeN)r7r8r9r:rPrMr;r;r;r<r=Usr=cs:eZdZdZddddfdd Zddd d d ZZS) EvalErrorz"Your architecture raised an error. BaseExceptionrNone)rZkwargsr@c stjfd|i|dS)NrZ)super__init__)rOrZrq __class__r;r<rsszEvalError.__init__str)namer@cCs2z ||WSty,td|dYn0dS)Nz%'EvalError' object has no attribute '')KeyErrorAttributeError)rOrwr;r;r< __getattr__s  zEvalError.__getattr__)r7r8r9r:rsr{ __classcell__r;r;rtr<rnsrn MODEL_OR_CHAIN_FACTORYrvMCF)llm_or_chain_factory dataset_namer@csdt|trR|jj}|jdurFjjj}td|d|d|dfddSt|tr`|St|trz|fddSt|r`t |rt t t |fd dSz |}WnNt yt t |}t|}td |d t|fd dYS0t t |t|tr|St t t |rDt t t |fd dSt|ts\fddSS|S)zForgive the user if they pass in a chain without memory instead of a chain factory. It's a common mistake. Raise a more helpful error message as well.Na$Cannot directly evaluate a chain with stateful memory. To evaluate this chain, pass in a chain constructor that initializes fresh memory each time it is called. This will safegaurd against information leakage between dataset examples. For example: def chain_constructor(): new_memory = z(...) return z*(memory=new_memory, ...) run_on_dataset("z", chain_constructor, ...)csSNr;r;)chainr;r<rWz(_wrap_in_chain_factory..csSrr;r;)lcfr;r<rrWcsSrr;r; runnable_r;r<rrWzWrapping function z as RunnableLambda.csSrr;r;)wrappedr;r<rrWcsSrr;r;rr;r<rrWcstSr)rr;) constructorr;r<rrW)rcr.rur7Zmemory ValueErrorrrcallabler%r$r r TypeErrorinspect signatureloggerinfor)rrZ chain_classZ memory_class_modelZ user_funcsigr;)rrrrrr<_wrap_in_chain_factorysT                  rzDict[str, Any])inputsr@cCs4|s tdg}d|vrJt|dts>tdt|dj|dg}nd|vrt|dtrvtdd|dDstdt|dj|d}nnt|dkrtt | }t|tr|g}n0t|trtd d|Dr|}ntd |ntd |t|dkr|d Std t|ddS)zGet prompt from inputs. Args: inputs: The input dictionary. Returns: A string prompt. Raises: InputFormatError: If the input format is invalid. Inputs should not be empty.promptz"Expected string for 'prompt', got promptscss|]}t|tVqdSrrcrvrGir;r;r< sz_get_prompt..z,Expected list of strings for 'prompts', got rLcss|]}t|tVqdSrrrr;r;r<r rWz)LLM Run expects string prompt input. Got z5LLM Run expects 'prompt' or 'prompts' in inputs. Got rz)LLM Run expects single prompt input. Got z prompts.N) r6rcrvtyper7listrJlennextitervalues)rrZprompt_r;r;r< _get_promptsD      rc@seZdZUdZded<dS)ChatModelInputzVInput for a chat model. Parameters: messages: List of chat messages. zList[BaseMessage]messagesNr7r8r9r:__annotations__r;r;r;r<rs rrdcCs|s td|}d|vr,|d|d<n t|dkrLtt||d<d|vr|d}t|tr~t dd|Dr~|g}t|dkrt |d|d<ntd|Std |d S) zGet Chat Messages from inputs. Args: inputs: The input dictionary. Returns: A list of chat messages. Raises: InputFormatError: If the input format is invalid. rrrCrLcss|]}t|tVqdSr)rcrdrr;r;r<r8sz _get_messages..rzGBatch messages not supported. Please provide a single list of messages.zMChat Run expects single List[dict] or List[List[dict]] 'messages' input. Got N) r6copypoprrrrrcrrJr)rZ input_copyZ raw_messagesr;r;r< _get_messages$s0   rr(Optional[Callable[[Dict], Any]]rp) first_example input_mapperr@c Cs|rP||j}t|tst|tr4tdd|Dstd|dt|dnVzt|jWnFtyzt|jWn$tytd|jdYn0Yn0dS)Ncss|]}t|tVqdSrrcrrGmsgr;r;r<rTrWz>_validate_example_inputs_for_language_model..zWhen using an input_mapper to prepare dataset example inputs for an LLM or chat model, the output must a single string or a list of chat messages. Got: of type .zvExample inputs do not match language model input format. Expected a dictionary with messages or a single prompt. Got: z Please update your dataset OR provide an input_mapper to convert the example.inputs to a compatible format for the llm or chat model you wish to evaluate.) rrcrvrrJr6rrr)rrZ prompt_inputr;r;r<+_validate_example_inputs_for_language_modelLs2    rr.)rrrr@cCs|rb||j}t|j|}t|tsBtd|dt|d|rtd|jd|nP|j}t|j|}t |dkrt |jdkrn|rtd|jd|dS) z  r)examplerrr@cCsRt|trt||n8|}t|tr4t|||nt|trNtd|dS)z9Validate that the example inputs are valid for the model.zSkipping input validation for N)rcrrr.rrrdebug)rrrrr;r;r<_validate_example_inputss    r List[Example]"Optional[smith_eval.RunEvalConfig]r'zOptional[List[RunEvaluator]])rexamplesr3 data_typer@c Cs|rzt|trd\}}d}n2d}|}t|tr6|jnd}t|trJ|jnd}t||||djrnt|djnd||}nd}|S)z.)rrrzUnsupported custom evaluator: z+. Expected RunEvaluator or StringEvaluator.) evaluatorscustom_evaluatorsanyrrrrfrcr"r2rrrrrr)rrrrrrrrrrrr#rZcustom_evaluatorr;r;r<rTsV          rr callbacksrmetadatarrOptional[Dict[str, Any]]zUnion[str, BaseMessage])rrrrrrr@c s|durn||}t|ts6t|tr\tdd|Dr\|j|t||pFg|pLiddIdHStd|dn~z2t|}|j|t||pg|piddIdH}WnJtyt|} |jfi| dt||pg|pidiIdH}Yn0|S) aAsynchronously run the language model. Args: llm: The language model to run. inputs: The input dictionary. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. input_mapper: Optional function to map inputs to the expected format. Returns: The LLMResult or ChatResult. Raises: ValueError: If the LLM type is unsupported. InputFormatError: If the input format is invalid. Ncss|]}t|tVqdSrrrr;r;r<rrWz_arun_llm..rrrrz%Input mapper returned invalid format 3 Expected a single string or list of chat messages.r) rcrvrrJainvokerr6rr) rrrrrrprompt_or_messagesr llm_output llm_inputsr;r;r< _arun_llmsJ  rrrrzUnion[Chain, Runnable]zUnion[dict, str])rrrrrrr@c s|dur |n||}t|trrt|trrt|dkrr|jrrtt|}|j|t ||pZg|p`iddIdH}n*t |pzg||pid} |j|| dIdH}|S)z%Run a chain asynchronously on inputs.NrLrrrrr) rcr.rdrrrrrrr rrrrrrZinputs_valrDrunnable_configr;r;r< _arun_chains*  r)rrz'Union[dict, str, LLMResult, ChatResult])rrrrr@c st|trdnd}d}znt|trNt||j|d|d||ddIdH}n0|}t||j|d|d||ddIdH}|}WnVty}z>t|d|j d |jd t |t |d }WYd}~n d}~00|S) aAsynchronously run the Chain or language model. Args: example: The example to run. llm_or_chain_factory: The Chain or language model constructor to run. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. input_mapper: Optional function to map the input to the expected format. Returns: A list of outputs. LLMr.Nrrrr failed for example with inputs  rZ) rcrrrrbr Exceptionrridreprrn) rrrr chain_or_llmrlrDrrhr;r;r<_arun_llm_or_chainsB    r)rrrrrrr@c Cs|durj||}t|ts6t|trXtdd|DrX|j|t||pFg|pLidd}qtd|dnlz,t|}|j|t||pg|pidd}Wn>tyt|} |jfi| dt||pid i}Yn0|S) a Run the language model on the example. Args: llm: The language model to run. inputs: The input dictionary. callbacks: The callbacks to use during the run. tags: Optional tags to add to the run. input_mapper: function to map to the inputs dictionary from an Example Returns: The LLMResult or ChatResult. Raises: ValueError: If the LLM type is unsupported. InputFormatError: If the input format is invalid. Ncss|]}t|tVqdSrrrr;r;r<rNrWz_run_llm..rrz'Input mapper returned invalid format: rr)rr) rcrvrrJinvokerr6rr) rrrrrrrrZ llm_promptsrr;r;r<_run_llm0sF  rzUnion[Dict, str]c Cs|dur |n||}t|trlt|trlt|dkrl|jrltt|}|j|t ||pZg|p`idd}n$t |ptg||p|id} |j|| d}|S)zRun a chain on inputs.NrLrrr) rcr.rdrrrrrrrrr;r;r< _run_chainns*  rc Cst|trdnd}d}zbt|trHt||j|d|d||dd}n*|}t||j|d|d||dd}|}Wnbty}zJt|j} t |d|j d |jd | d | t |d }WYd}~n d}~00|S) a Run the Chain or language model synchronously. Args: example: The example to run. llm_or_chain_factory: The Chain or language model constructor to run. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. Returns: Union[List[dict], List[str], List[LLMResult], List[ChatResult]]: The outputs of the model or chain. rr.Nrrrrrrz Error Type: z , Message: r) rcrrrrbrrrr7rrrrn) rrrrrrlrDrrhZ error_typer;r;r<_run_llm_or_chainsH    rrzOptional[Union[str, datetime]]z1Tuple[MCF, TracerSession, Dataset, List[Example]])clientrr project_nameproject_metadatardataset_versionr@c Cspt||}|j|d}t|j|j|d} | s>td|ddd| D} | rXt| nd} | rh| nd} zN|pti}t} | ri|d| i}| |d<|j ||j|rd |ini|d }Wnpt tt fy*}zPd t |vr|t }d |d |d|d}td|d|WYd}~n d}~00|jd|j}td|d|d|d|jdd|||| fS)N)r)Z dataset_idZas_ofzDataset z has no example rows.cSsg|]}|jr|jqSr;) modified_at)rGexr;r;r<rIrWz%_prepare_eval_run..gitrr)Zreference_dataset_idZ project_extrarzalready exists z+ run_on_dataset( ... project_name="z - z", # Update since z already exists ) z Test project z/ already exists. Please use a different name: z/compare?selectedSessions=z)View the evaluation results for project 'z' at: z View all tests for Dataset z at: T)flush)rZ read_datasetrZ list_examplesrrmax isoformatrZcreate_projectr,r+rvuuiduuid4urlprint)rrrrrrr wrapped_modeldatasetrrZmax_modified_atZinferred_versionZgit_infoprojectrhuidZ example_msgZcomparison_urlr;r;r<_prepare_eval_runsh    rc@s*eZdZUdZded<ded<ded<dS) _RowResultz5A dictionary of the results for a single example row.z Optional[List[EvaluationResult]]rRzOptional[float]r[rr\Nrr;r;r;r<rs rF)totalc@seZdZUdZded<ded<ded<ded <d ed <d Zd ed<ddddddZdddddZddddZddd d!d"Z d5dd$dd%d&d'Z e d6dd)d*d+d,d-d.d/d0d+d1dd2 d3d4Z d S)7_DatasetRunContainerz3A container to help manage the state of a eval run.rrr*rrrrrzList[RunnableConfig]configsNz6Optional[List[smith_eval_config.BATCH_EVALUATOR_LIKE]]batch_evaluatorsrzDict[str, _RowResult]rd) batch_resultsall_eval_resultsr@cCsi}t|j|D]\}}tt|t|ji}|j|dg|d|dd|t|j<t|t r~|j |t|jd<n||t|jd<|j r|j |t|jd<q|S)NrRr[r\)rCrRr[r\rZrDrE) ziprr rrbrvrrrcrnrZr)rOrrrQrrDZ row_resultr;r;r<_merge_test_outputss  z(_DatasetRunContainer._merge_test_outputszDict[str, Run]z List[dict])runsr@c s|j}|sgSfdd|jD}g}tj}|D]}zV|||j}t|trZ|}|t t||j |j j fi|d|j jdWq6ty}z&tdt|d|WYd}~q6d}~00q6Wdn1s0Y|S)Ncsg|]}t|jqSr;)rvrrGrrr;r<rI0rWz>_DatasetRunContainer._run_batch_evaluators..)r\Z project_idzError running batch evaluator z: )rr concurrentfuturesThreadPoolExecutorrcr!rdrfr submitrZcreate_feedbackrrrrr]r) rOrrZ runs_listaggregate_feedbackexecutorZ evaluatorrlrhr;rr<_run_batch_evaluators,s2   :z*_DatasetRunContainer._run_batch_evaluatorsz,Tuple[Dict[str, _RowResult], Dict[str, Run]]r?c Csi}i}|jD]}tt|dD]}t|trf|j}|D]&\\}}}|t|i d|iq.)rrrrrr)r4Z random_namererrrbrarfrrrr'kvrr5ZProgressBarCallbackrr)clsrrrrr3rrr:rr;rrrrrTrUrr;r>r<preparesJ  z_DatasetRunContainer.prepare)F)NNNr7NNN) r7r8r9r:rrrr%r/r0r6 classmethodrBr;r;r;r<r s*  rr1r?cCsHz.ddlm}|}|duo,dtt|vWStyBYdS0dS)Nr) get_ipythonZzmqshellF)ZIPythonrDrvrr`)rDresr;r;r<_is_jupyter_environments   rFr>)aggregate_resultsr@cCsRtr,ddlm}m}||d||n"|jdddd}tdt|dS) Nr)HTMLdisplayz

Experiment Results:

cSs|dS)Nz.2fr;)xr;r;r<rrWz,_display_aggregate_results..right)Z float_formatjustifyz Experiment Results:)rFIPython.displayrHrIZ to_stringr )rGrHrIZformatted_stringr;r;r<r3s  r3aThe input_mapper argument is deprecated and will be removed in a future release. Please add a RunnableLambda to your chain to map inputs to the expected format instead. Example: def construct_chain(): my_chain = ... input_mapper = {'other_key': 'MyOtherInput', 'my_input_key': x} return input_mapper | my_chain run_on_dataset(..., llm_or_chain_factory=construct_chain) (See https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.base.RunnableLambda.html)r7)r3rr:rrr2r;zOptional[Client]r9r8r) rrrr3rr:rrr2r;rqr@c s| dd} | rtdtdd| dur2td} | dd} | rPtdddd| rntdd | d dd |pvt}tj|||||| | ||| |d } t j | j d dgt t jt| j| d| j| j RIdH}| j||dS)Nr0.0.305Tmessagependingr;r0.1.9qThe tags argument is deprecated and will be removed in a future release. Please specify project_metadata instead.PThe following arguments are deprecated and will be removed in a future release: rrPZremovalrr;rrr=rrr2)rr_INPUT_MAPPER_DEP_WARNINGr rbrrrrBrunnable_utilsZgather_with_concurrencyrmap functoolspartialrrrr6)rrrr3rr:rrr2r;rqrr containerrr;r;r<arun_on_datasets`      r_c s$| ddrtdtdd| dd} | r.rWrX)rrrYr rbrrrrBrrrrZget_executor_for_configrr[r\r]rrr6)rrrr3rr:rrr2r;rqrrr$r;r`r<run_on_dataset8sf        $ raa1 Run the Chain or language model on a dataset and store traces to the specified project name. Args: dataset_name: Name of the dataset to run the chain on. llm_or_chain_factory: Language model or Chain constructor to run over the dataset. The Chain constructor is used to permit independent calls on each example without carrying over state. evaluation: Configuration for evaluators to run on the results of the chain concurrency_level: The number of async tasks to run concurrently. project_name: Name of the project to store the traces in. Defaults to {dataset_name}-{chain class name}-{datetime}. project_metadata: Optional metadata to add to the project. Useful for storing information the test variant. (prompt version, model version, etc.) client: LangSmith client to use to access the dataset and to log feedback and run traces. verbose: Whether to print progress. tags: Tags to add to each run in the project. revision_id: Optional revision identifier to assign this test run to track the performance of different versions of your system. Returns: A dictionary containing the run's project name and the resulting model outputs. For the (usually faster) async version of this function, see :func:`arun_on_dataset`. Examples -------- .. code-block:: python from langsmith import Client from langchain_openai import ChatOpenAI from langchain.chains import LLMChain from langchain.smith import smith_eval.RunEvalConfig, run_on_dataset # Chains may have memory. Passing in a constructor function lets the # evaluation framework avoid cross-contamination between runs. def construct_chain(): llm = ChatOpenAI(temperature=0) chain = LLMChain.from_string( llm, "What's the answer to {your_input_key}" ) return chain # Load off-the-shelf evaluators via config or the EvaluatorType (string or enum) evaluation_config = smith_eval.RunEvalConfig( evaluators=[ "qa", # "Correctness" against a reference answer "embedding_distance", smith_eval.RunEvalConfig.Criteria("helpfulness"), smith_eval.RunEvalConfig.Criteria({ "fifth-grader-score": "Do you have to be smarter than a fifth grader to answer this question?" }), ] ) client = Client() run_on_dataset( client, dataset_name="", llm_or_chain_factory=construct_chain, evaluation=evaluation_config, ) You can also create custom evaluators by subclassing the :class:`StringEvaluator ` or LangSmith's `RunEvaluator` classes. .. code-block:: python from typing import Optional from langchain.evaluation import StringEvaluator class MyStringEvaluator(StringEvaluator): @property def requires_input(self) -> bool: return False @property def requires_reference(self) -> bool: return True @property def evaluation_name(self) -> str: return "exact_match" def _evaluate_strings(self, prediction, reference=None, input=None, **kwargs) -> dict: return {"score": prediction == reference} evaluation_config = smith_eval.RunEvalConfig( custom_evaluators = [MyStringEvaluator()], ) run_on_dataset( client, dataset_name="", llm_or_chain_factory=construct_chain, evaluation=evaluation_config, ) zrun_on_dataset(zawait arun_on_dataset()r})NNN)r: __future__rconcurrent.futuresr dataclassesr\rloggingr rrtypingrrrrr r r r r Zlangchain_core._apirZ langchain_core.callbacks.managerrZlangchain_core.language_modelsrZlangchain_core.messagesrrZlangchain_core.outputsrrZlangchain_core.runnablesrrrrrrrZZ!langchain_core.tracers.evaluationrrZ langchain_core.tracers.langchainrZlangsmith.clientrZ langsmith.envrr Zlangsmith.evaluationr!r"r#rZlangsmith.run_helpersr$r%Zlangsmith.schemasr&r'r(r)r*Zlangsmith.utilsr+requestsr,Ztyping_extensionsr-Zlangchain.chains.baser.Zlangchain.evaluation.loadingr/Zlangchain.evaluation.schemar0r1r2Zlangchain.smithr3rZlangchain.smith.evaluationrr4r5r_rg getLoggerr7rrdr~rrr6r=rnrrrrrrrrrrrrrrrrrrrrrr dataclassrrFr3rYr_raZ_RUN_ON_DATASET_DOCSTRINGreplacer;r;r;r<s ,                 I=2 (!%C GE%>C%==C (F(Mk