a agG@sdZddlmZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZddlmZddlmZdd lmZdd lmZdd lmZeeZGd d d eeZGdddeZGdddZGdddeeZ GdddeeZ!GdddeeZ"dS)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Enum)AnyOptionalSequenceTupleUnion)warn) AgentAction)BaseLanguageModel)run_in_executor)Chainc@s`eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdS) EvaluatorTypezThe types of the evaluators.ZqaZcot_qaZ context_qaZpairwise_stringZ score_stringZlabeled_pairwise_stringZlabeled_score_stringZ trajectorycriteriaZlabeled_criteriaZstring_distanceZ exact_matchZ regex_matchZpairwise_string_distanceZembedding_distanceZpairwise_embedding_distanceZ json_validityZ json_equalityZjson_edit_distanceZjson_schema_validationN)__name__ __module__ __qualname____doc__ZQAZCOT_QAZ CONTEXT_QAZPAIRWISE_STRINGZ SCORE_STRINGZLABELED_PAIRWISE_STRINGZLABELED_SCORE_STRINGZAGENT_TRAJECTORYZCRITERIAZLABELED_CRITERIAZSTRING_DISTANCEZ EXACT_MATCHZ REGEX_MATCHZPAIRWISE_STRING_DISTANCEZEMBEDDING_DISTANCEZPAIRWISE_EMBEDDING_DISTANCEZ JSON_VALIDITYZ JSON_EQUALITYZJSON_EDIT_DISTANCEZJSON_SCHEMA_VALIDATIONrri/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain/evaluation/schema.pyrs,rc@s*eZdZdZeeddddddZdS) LLMEvalChainz,A base class for evaluators that use an LLM.r r)llmkwargsreturncKsdS)z#Create a new evaluator from an LLM.Nr)clsrrrrrfrom_llmMszLLMEvalChain.from_llmN)rrrr classmethodrrrrrrrJsrc@sleZdZdZeddddZeddddZeddd d Zeddd d ZdddddddZ d S)_EvalArgsMixinz(Mixin for checking evaluation arguments.boolrcCsdSz2Whether this evaluator requires a reference label.Frselfrrrrequires_referenceVsz!_EvalArgsMixin.requires_referencecCsdS)0Whether this evaluator requires an input string.Frr#rrrrequires_input[sz_EvalArgsMixin.requires_inputstrcCsd|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__rr#rrr_skip_input_warning`sz"_EvalArgsMixin._skip_input_warningcCsd|jjdS)z*Warning to show when reference is ignored.zIgnoring reference in r)r*r#rrr_skip_reference_warningesz&_EvalArgsMixin._skip_reference_warningN Optional[str]None) referenceinputrcCsx|jr"|dur"t|jjdn|dur:|js:t|j|jr\|dur\t|jjdn|durt|jstt|jdS)aCheck if the evaluation arguments are valid. Args: reference (Optional[str], optional): The reference label. input (Optional[str], optional): The input string. Raises: ValueError: If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided. Nz requires an input string.z requires a reference string.)r' ValueErrorr+rr r,r%r-)r$r0r1rrr_check_evaluation_argsls z%_EvalArgsMixin._check_evaluation_args)NN) rrrrpropertyr%r'r,r-r3rrrrrSsrc@seZdZdZeddddZeddddZed d d d d d d ddddZd d d d d d d ddddZ d d d dddd ddddZ d d d dddd ddddZ d S)StringEvaluatorzcGrade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.r(r!cCs|jjS)zThe name of the evaluation.r*r#rrrevaluation_nameszStringEvaluator.evaluation_namer cCsdSr"rr#rrrr%sz"StringEvaluator.requires_referenceNr0r1zUnion[str, Any]zOptional[Union[str, Any]]rdict) predictionr0r1rrcKsdS)a:Evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrr$r9r0r1rrrr_evaluate_stringss z!StringEvaluator._evaluate_stringscs"td|jf|||d|IdHS)aIAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr9r0r1)rr;r:rrr_aevaluate_stringssz"StringEvaluator._aevaluate_stringsr.cKs&|j||d|jf|||d|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r7r<)r3r;r:rrrevaluate_stringss z StringEvaluator.evaluate_stringscs,|j||d|jf|||d|IdHS)a Asynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r7r<N)r3r=r:rrraevaluate_stringss z!StringEvaluator.aevaluate_strings) rrrrr4r6r%rr;r=r>r?rrrrr5s$#r5c @seZdZdZeddddddddddd d Zddddddddddd d Zddddddddddd dZddddddddddddZdS)PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).Nr7r(r.rr8)r9 prediction_br0r1rrcKsdS)1Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nrr$r9rAr0r1rrrr_evaluate_string_pairss z.PairwiseStringEvaluator._evaluate_string_pairscs$td|jf||||d|IdHS)@Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr9rAr0r1)rrDrCrrr_aevaluate_string_pairssz/PairwiseStringEvaluator._aevaluate_string_pairscKs(|j||d|jf||||d|S)rBr7rF)r3rDrCrrrevaluate_string_pairs/sz-PairwiseStringEvaluator.evaluate_string_pairscs.|j||d|jf||||d|IdHS)rEr7rFN)r3rGrCrrraevaluate_string_pairsLsz.PairwiseStringEvaluator.aevaluate_string_pairs) rrrrrrDrGrHrIrrrrr@s#"r@c @seZdZdZeddddZedddd dd d d d ddZdddd dd d d d ddZdddd dd d d d ddZ dddd dd d d d ddZ dS)AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.r r!cCsdS)r&Trr#rrrr'msz'AgentTrajectoryEvaluator.requires_inputN)r0r(z!Sequence[Tuple[AgentAction, str]]r.rr8)r9agent_trajectoryr1r0rrcKsdS)Evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nrr$r9rKr1r0rrrr_evaluate_agent_trajectoryrs z3AgentTrajectoryEvaluator._evaluate_agent_trajectorycs$td|jf||||d|IdHS)Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)r9rKr0r1)rrNrMrrr_aevaluate_agent_trajectorysz4AgentTrajectoryEvaluator._aevaluate_agent_trajectorycKs(|j||d|jf||||d|S)rLr7r9r1rKr0)r3rNrMrrrevaluate_agent_trajectorysz2AgentTrajectoryEvaluator.evaluate_agent_trajectorycs.|j||d|jf||||d|IdHS)rOr7rQN)r3rPrMrrraevaluate_agent_trajectorysz3AgentTrajectoryEvaluator.aevaluate_agent_trajectory) rrrrr4r'rrNrPrRrSrrrrrJjs%$rJ)#r __future__rloggingabcrrenumrtypingrrrr r warningsr Zlangchain_core.agentsr Zlangchain_core.language_modelsr Zlangchain_core.runnables.configrZlangchain.chains.baser getLoggerrloggerr(rrrr5r@rJrrrrs"        6 1tr