a ag4@sTdZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZmZddlmZmZdd lmZmZmZdd lmZdd lmZdd l m!Z!m"Z"dd l#m$Z%eeeegeeee&ffZ'eeeeeegeeee&ffZ(GdddeZ)Gddde)Z*ee'ee"fZ+ee!e,e)fZ-GdddeZ.dS)z!Configuration for run evaluators.)AnyCallableDictListOptionalSequenceUnion) Embeddings)BaseLanguageModel)BasePromptTemplate) RunEvaluator)EvaluationResultEvaluationResults)ExampleRun) BaseModel ConfigDictField) CRITERIA_TYPE)EmbeddingDistance) EvaluatorTypeStringEvaluator)StringDistancec@s0eZdZUdZeed<eeefdddZ dS) EvalConfigaConfiguration for a given run evaluator. Parameters ---------- evaluator_type : EvaluatorType The type of evaluator to use. Methods ------- get_kwargs() Get the keyword arguments for the evaluator configuration. evaluator_typereturncCs4i}|D]&\}}|dkrqn |dur&q|||<q|S)zGet the keyword arguments for the load_evaluator call. Returns ------- Dict[str, Any] The keyword arguments for the load_evaluator call. rN)selfkwargsfieldvalrro/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain/smith/evaluation/config.py get_kwargs0s   zEvalConfig.get_kwargsN) __name__ __module__ __qualname____doc__r__annotations__rstrrr#rrrr"rs rcs`eZdZUdZdZeeed<dZeeed<dZ eeed<e ee fdfdd Z Z S) SingleKeyEvalConfigzBConfiguration for a run evaluator that only requires a single key.N reference_keyprediction_key input_keyrcs$t}dD]}||dq|S)N)r+r,r-)superr#pop)rrkey __class__rr"r#Qs zSingleKeyEvalConfig.get_kwargs)r$r%r&r'r+rr)r(r,r-rrr# __classcell__rrr1r"r*Cs r*c@sreZdZUdZeedZeee e fe d<dZ e ee e d<dZe eee d<dZe ee d<dZe ee d<dZe ee d <dZe ee d <ed d ZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZ Gddde!Z"Gdd d eZ#Gd!d"d"eZ$Gd#d$d$eZ%Gd%d&d&e%Z&dS)' RunEvalConfiga0Configuration for a run evaluation. Parameters ---------- evaluators : List[Union[EvaluatorType, EvalConfig, RunEvaluator, Callable]] Configurations for which evaluators to apply to the dataset run. Each can be the string of an :class:`EvaluatorType `, such as EvaluatorType.QA, the evaluator type string ("qa"), or a configuration for a given evaluator (e.g., :class:`RunEvalConfig.QA `). custom_evaluators : Optional[List[Union[RunEvaluator, StringEvaluator]]] Custom evaluators to apply to the dataset run. reference_key : Optional[str] The key in the dataset run to use as the reference string. If not provided, it will be inferred automatically. prediction_key : Optional[str] The key from the traced run's outputs dictionary to use to represent the prediction. If not provided, it will be inferred automatically. input_key : Optional[str] The key from the traced run's inputs dictionary to use to represent the input. If not provided, it will be inferred automatically. eval_llm : Optional[BaseLanguageModel] The language model to pass to any evaluators that use a language model. )default_factory evaluatorsNcustom_evaluatorsbatch_evaluatorsr+r,r-eval_llmTZarbitrary_types_allowedcs`eZdZUdZdZeeed<dZee ed<e j Z e ed<d eee ddfdd ZZS) zRunEvalConfig.Criteriaa"Configuration for a reference-free criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Ncriteriallmrr;rrc stjfd|i|dSNr;r.__init__rr;rr1rr"r@szRunEvalConfig.Criteria.__init__)N)r$r%r&r'r;rrr(r<r rZCRITERIArrr@r3rrr1r"Criterias  rBcs`eZdZUdZdZeeed<dZee ed<e j Z e ed<d eee ddfdd ZZS) zRunEvalConfig.LabeledCriteriaa,Configuration for a labeled (with references) criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Nr;r<rr=c stjfd|i|dSr>r?rAr1rr"r@sz&RunEvalConfig.LabeledCriteria.__init__)N)r$r%r&r'r;rrr(r<r rZLABELED_CRITERIArrr@r3rrr1r"LabeledCriterias  rCc@sJeZdZUdZejZeed<dZe e ed<dZ e e ed<e ddZdS)zRunEvalConfig.EmbeddingDistanceaIConfiguration for an embedding distance evaluator. Parameters ---------- embeddings : Optional[Embeddings] The embeddings to use for computing the distance. distance_metric : Optional[EmbeddingDistanceEnum] The distance metric to use for computing the distance. rN embeddingsdistance_metricTr:)r$r%r&r'rZEMBEDDING_DISTANCErr(rDrr rEEmbeddingDistanceEnumr model_configrrrr"rs  rc@s<eZdZUdZejZeed<dZe e ed<dZ e ed<dS)zRunEvalConfig.StringDistancezConfiguration for a string distance evaluator. Parameters ---------- distance : Optional[StringDistanceEnum] The string distance metric to use. rNdistanceTnormalize_score) r$r%r&r'rZSTRING_DISTANCErr(rHrStringDistanceEnumrIboolrrrr"rs   rc@s@eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)zRunEvalConfig.QAa-Configuration for a QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr<prompt) r$r%r&r'rQArr(r<rr rLr rrrr"rMs  rMc@s@eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)zRunEvalConfig.ContextQA<Configuration for a context-based QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr<rL r$r%r&r'rZ CONTEXT_QArr(r<rr rLr rrrr" ContextQAs  rPc@s@eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)zRunEvalConfig.CoTQArNrNr<rLrOrrrr"CoTQAs  rQc@s eZdZUdZejZeed<dS)zRunEvalConfig.JsonValidityz\Configuration for a json validity evaluator. Parameters ---------- rN)r$r%r&r'rZ JSON_VALIDITYrr(rrrr" JsonValidity&s rRc@s eZdZUdZejZeed<dS)z#RunEvalConfig.JsonEqualityEvaluatorz\Configuration for a json equality evaluator. Parameters ---------- rN)r$r%r&r'rZ JSON_EQUALITYrr(rrrr"JsonEqualityEvaluator/s rSc@sDeZdZUdZejZeed<dZe ed<dZ e ed<dZ e ed<dS)zRunEvalConfig.ExactMatchaxConfiguration for an exact match string evaluator. Parameters ---------- ignore_case : bool Whether to ignore case when comparing strings. ignore_punctuation : bool Whether to ignore punctuation when comparing strings. ignore_numbers : bool Whether to ignore numbers when comparing strings. rF ignore_caseignore_punctuationignore_numbersN) r$r%r&r'rZ EXACT_MATCHrr(rTrKrUrVrrrr" ExactMatch8s    rWc@s,eZdZUdZejZeed<dZe ed<dS)zRunEvalConfig.RegexMatchzConfiguration for a regex match string evaluator. Parameters ---------- flags : int The flags to pass to the regex. Example: re.IGNORECASE. rrflagsN) r$r%r&r'rZ REGEX_MATCHrr(rXintrrrr" RegexMatchJs rZcseZdZUdZejZeed<dZe e ed<dZ e e ed<dZ e eed<dZe eed<d e e e eeddfd d ZZS) zRunEvalConfig.ScoreStringaConfiguration for a score string evaluator. This is like the criteria evaluator but it is configured by default to return a score on the scale from 1-10. It is recommended to normalize these scores by setting `normalize_by` to 10. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. normalize_by: Optional[int] = None If you want to normalize the score, the denominator to use. If not provided, the score will be between 1 and 10 (by default). prompt : Optional[BasePromptTemplate] rNr;r< normalize_byrL)r;r[rrc stjf||d|dS)N)r;r[r?)rr;r[rr1rr"r@qsz"RunEvalConfig.ScoreString.__init__)NN)r$r%r&r'rZ SCORE_STRINGrr(r;rrr<r r[floatrLr rr@r3rrr1r" ScoreStringVs r]c@seZdZUejZeed<dS)z RunEvalConfig.LabeledScoreStringrN)r$r%r&rZLABELED_SCORE_STRINGrr(rrrr"LabeledScoreStringys r^)'r$r%r&r'rlistr6rrSINGLE_EVAL_CONFIG_TYPECUSTOM_EVALUATOR_TYPEr(r7rr8BATCH_EVALUATOR_LIKEr+r)r,r-r9r rrGr*rBrCrrrMrPrQrRrrSrWrZr]r^rrrr"r4]s> #    #r4N)/r'typingrrrrrrrZlangchain_core.embeddingsr Zlangchain_core.language_modelsr Zlangchain_core.promptsr Z langsmithr Zlangsmith.evaluation.evaluatorr rZlangsmith.schemasrrZpydanticrrrZ(langchain.evaluation.criteria.eval_chainrZ,langchain.evaluation.embedding_distance.baserrFZlangchain.evaluation.schemarrZ)langchain.evaluation.string_distance.baserrJdictZRUN_EVALUATOR_LIKErbrr*rar)r`r4rrrr"s2$        $