a `goy@shdZddlmZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZddlmZddlmZzddlmZmZmZmZWn*eyddlmZmZmZmZYn0ddlZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%e&e'Z(Gd d d eZ)Gd ddeddZ*GdddeZ+GdddeddZ,GdddZ-ee+e,e.fZ/GdddeZ0ee0e.fZ1Gddde-Z2ddddZ3dZ4d d!d"d#Z5d$d$d%d&d'Z6Gd(d)d)Z7d*d)d+d,d-Z8d.d/d+d0d1Z9d.d2d+d3d4Z:d5d6d7d8d9Z;ee eej%eej$gee+e,ffe e ej%e ej$gee+e,fffZz@DynamicRunEvaluator._coerce_evaluation_result..)r.rr/zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r5z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got )r:r,r5 ValueErrorrdallr)rNrmr5rner$rrr%_coerce_evaluation_results8   z-DynamicRunEvaluator._coerce_evaluation_resultzUnion[dict, EvaluationResults]rH)rEr5rLcsPd|vr:|}fdd|dD|d<tfi|Sjtt|ddS)NrEcsg|]}j|dqS))r5)rx)rprrNr5r$r% szBDynamicRunEvaluator._coerce_evaluation_results..T)r5rn)copyrDrxr rC)rNrEr5cpr$rzr%_coerce_evaluation_resultss   z.DynamicRunEvaluator._coerce_evaluation_resultszMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list])rmr5rLcCs.t|tr|js||_|St|}|||Sro)r:r,r5_format_evaluator_resultr~)rNrmr5r$r$r%_format_results  z"DynamicRunEvaluator._format_resultrLcCs t|dSzCheck if the evaluator function is asynchronous. Returns: bool: True if the evaluator function is asynchronous, False otherwise. rZhasattrrNr$r$r%is_async'szDynamicRunEvaluator.is_asyncrrGrIcCst|ds6t}|r$tdn||||St}d|j i}t |ddrbt |j |d<|j ||||dd}|||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rYtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.r6 session_idN experimentrun_idmetadataZlangsmith_extra)rrPget_event_loop is_running RuntimeErrorrun_until_completerSuuiduuid4idrcrrrYr)rNrJrK running_loopr5rrmr$r$r%rO0s"   z DynamicRunEvaluator.evaluate_runrJrKcsrt|dst||IdHSt}d|ji}t|ddrJt|j|d<|j ||||ddIdH}| ||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rZNr6rrrr) rsuperrSrrrrcrrrZr)rNrJrKr5rrm __class__r$r%rSRs    z!DynamicRunEvaluator.aevaluate_runcCs |||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. )rOrMr$r$r%__call__mszDynamicRunEvaluator.__call__rcCsd|jdS))Represent the DynamicRunEvaluator object.zrdrr$r$r%__repr__~szDynamicRunEvaluator.__repr__)N)F)N)N)N)rr r!r"rjrxr~rpropertyrrOrSrr __classcell__r$r$rr%rVs6 "rVrWrYcCst|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rVrr$r$r% run_evaluators ri'r)objcCs,t|}t|tkr(|dtdd}|S)Nz...))reprlen_MAXSIZE)rsr$r$r% _maxsize_reprs rrC)inputsrLcCs&t|d}t|d}||dS)NrJrKr)rget)rZ run_truncatedZexample_truncatedr$r$r%rbsrbc@seZdZdZd"dddddZedd d d Zd#d d ddddZd$d d ddddZd%d d ddddZ dd ddZ e d ddddZ ddd ddd d!Z dS)&DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.NfCallable[[Sequence[Run], Optional[Example]], Union[_COMPARISON_OUTPUT, Awaitable[_COMPARISON_OUTPUT]]]zUOptional[Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]]rXcCst|}|rt|}t||ddlm}|durR|j|td|_t|dd|_t |r|durlt d|j|td|_t|dd|_n8|jt t ttttgtf|td|_t|dd|_dS)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. rr[Nr]rrVr^)$_normalize_comparison_evaluator_funcrr`r\rarbrZrcrdrerfrgr rr rr r_COMPARISON_OUTPUTrYrir$r$r%rjs>    z&DynamicComparisonRunEvaluator.__init__rlrcCs t|dSrrrr$r$r%rsz&DynamicComparisonRunEvaluator.is_async Sequence[Run]rGrTrunsrKrLcCslt|ds6t}|r$tdn||||St}| |}|j ||||dd}| |||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rYrrtagsr) rrPrrrr acompare_runsrr _get_tagsrY_format_results)rNrrKrr5rrmr$r$r% compare_runss"   z*DynamicComparisonRunEvaluator.compare_runscsRt|ds|||St}||}|j||||ddIdH}||||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rZrrN)rrrrrrZr)rNrrKr5rrmr$r$r%rs    z+DynamicComparisonRunEvaluator.acompare_runscCs |||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rNrrKr$r$r%rsz&DynamicComparisonRunEvaluator.__call__rcCsd|jdS)rzJrtzADynamicComparisonRunEvaluator._format_results..)rUr-r5r-zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r5zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got ) r:rTr5listziprdrCrur)rNrmr5rmsgrwr$r$r%r>s:     z-DynamicComparisonRunEvaluator._format_results)N)N)N)N)rr r!r"rjrrrrrr staticmethodrrr$r$r$r%rs8   rr)rYrLcCst|S)z.Create a comaprison evaluator from a function.)rrr$r$r%comparison_evaluatorcsrrz|Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]]csdt}dd|jDrHtfddDs^tdkr^dd}t|ntfd dDrd d gkrStrd d ddfdd }tdrt dn|j |_ |Sd dddfdd }tdrt dn|j |_ |SdS)N)rJrKroutputsreference_outputs attachmentscSs&g|]\}}|j|j|jfvr|qSr$kindPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrppnamepr$r$r%r{|sz-_normalize_evaluator_func..c3s|]}|vVqdSror$rprsupported_argsr$r%rsrtz,_normalize_evaluator_func..kInvalid evaluator function. Must have at least one positional argument. Supported positional arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc3s|]}|vVqdSror$rrr$r%rssrJrKrrGrhrIcs^|||r|jni|jpi|r&|jp(ini|r6|jp8inidfddD}|IdHS)NrJrKrrrrc3s|]}|VqdSror$rpargZarg_mapr$r%rsrtz>_normalize_evaluator_func..awrapper..rrrrJrKargsrYpositional_argsrr%awrappers z+_normalize_evaluator_func..awrapperrrcsP|||r|jni|jpi|jp i|r.|jp0inidfddD}|S)Nrc3s|]}|VqdSror$rrr$r%rsrtz=_normalize_evaluator_func..wrapper..rrrrr%wrappers z*_normalize_evaluator_func..wrapper re signature parametersitemsrvrrurfrrcrrYsigrrrr$rYrrr%r_msD        r_zUnion[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]]csdt}dd|jDrHtfddDs^tdkr^dd}t|ntfd dDrd d gkrStrd d ddfdd }tdrt dn|j |_ |Sd dddfdd }tdrt dn|j |_ |SdS)NrrKrrrcSs&g|]\}}|j|j|jfvr|qSr$rrr$r$r%r{sz8_normalize_comparison_evaluator_func..c3s|]}|vVqdSror$rrr$r%rsrtz7_normalize_comparison_evaluator_func..rrrc3s|]}|vVqdSror$rrr$r%rssrrKrrGrrcsR|||r|jnidd|D|r*|jp,inidfddD}|IdHS)NcSsg|]}|jpiqSr$rrprJr$r$r%r{rtzJ_normalize_comparison_evaluator_func..awrapper..rc3s|]}|VqdSror$rrr$r%rsrtzI_normalize_comparison_evaluator_func..awrapper..rrrrKrrrr%rs  z6_normalize_comparison_evaluator_func..awrapperrrcsL|||r|jnidd|D|r*|jp,inidfddD}|S)NcSsg|]}|jpiqSr$rrr$r$r%r{rtzI_normalize_comparison_evaluator_func..wrapper..rc3s|]}|VqdSror$rrr$r%rsrtzH_normalize_comparison_evaluator_func..wrapper..rrrrr%rs  z5_normalize_comparison_evaluator_func..wrapperrrr$rr%rsD         rz;Union[EvaluationResults, dict, str, int, bool, float, list]zUnion[EvaluationResults, dict])rmrLcCst|tttfrd|i}nx|s.td|ndt|trdtdd|DsZtd|dd|i}n.t|trxd|i}nt|trntd ||S) Nr.zdExpected a non-empty dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got css|]}t|tVqdSro)r:rC)rpxr$r$r%rsrtz+_format_evaluator_result..z8Expected a list of dicts or EvaluationResults. Received .rErzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) r:rlr<r;rurrvrrCrrr$r$r%r s0       rSUMMARY_EVALUATOR_Tcsdt}dd|jDrHtfddDsrtdkrrdd}rh|d d7}t|n^tfd dDrd d gkrSd dddfdd }tdrtdn|j |_ |SdS)NrexamplesrrrcSs&g|]\}}|j|j|jfvr|qSr$rrr$r$r%r{5sz0_normalize_summary_evaluator..c3s|]}|vVqdSror$rrr$r%rs;rtz/_normalize_summary_evaluator..rrrz Received positional arguments c3s|]}|vVqdSror$rrr$r%rsGsrrzSequence[schemas.Run]zSequence[schemas.Example]rH)rrrLcs^||dd|Ddd|Ddd|DdfddD}|}t|trV|St|S)NcSsg|] }|jqSr$)rrprKr$r$r%r{SrtzA_normalize_summary_evaluator..wrapper..cSsg|]}|jpiqSr$rrr$r$r%r{TrtcSsg|]}|jpiqSr$rrr$r$r%r{Urtrc3s|]}|VqdSror$rrr$r%rsWrtz@_normalize_summary_evaluator..wrapper..)r:r,r)rrrrmrrr%rMs    z-_normalize_summary_evaluator..wrapperr) rerrrrvrrurrcr)rYrrrr$rr%_normalize_summary_evaluator2s6     r)>r" __future__rrPrerabcrtypingrrrrrr r r r r Ztyping_extensionsrr`rZ pydantic.v1rrrr ImportErrorZpydanticlogging functoolsrZlangsmith.schemasrrrr getLoggerrr=rr&r,rDrFrCrhrTrrVrrrrbrrr_rrrrr$r$r$r%sd  0      /  W E SJ