a ag(@s$UdZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZddlmZmZdd lmZmZdd lmZdd lmZmZdd lmZdd l m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0m1Z1e2eedddZ3e)j4e%e)j5e$e)j6e#e)j7ee)j8e.e)j9ee)j:e-e)j;ee)je1e)j?e0e)j@ee)jAee)jBee)jCee)jDee)jEe!e)jFe'e)jGeiZHee)eee*ee ee+ffeId<dde)ee eee e+fdddZJdddee)ee eeKeeee e+fdddZLdS) z Loading datasets and evaluators.)AnyDictListOptionalSequenceTypeUnion)BaseLanguageModel)Chain)TrajectoryEvalChain)PairwiseStringEvalChain)LabeledPairwiseStringEvalChain)CriteriaEvalChainLabeledCriteriaEvalChain)EmbeddingDistanceEvalChain"PairwiseEmbeddingDistanceEvalChain)ExactMatchStringEvaluator)JsonEqualityEvaluatorJsonValidityEvaluator)JsonEditDistanceEvaluator)JsonSchemaEvaluator)ContextQAEvalChainCotQAEvalChain QAEvalChain)RegexMatchStringEvaluator) EvaluatorType LLMEvalChainStringEvaluator)LabeledScoreStringEvalChainScoreStringEvalChain)PairwiseStringDistanceEvalChainStringDistanceEvalChain)urireturncCsLzddlm}Wnty*tdYn0|d|}dd|dDS)aLoad a dataset from the `LangChainDatasets on HuggingFace `_. Args: uri: The uri of the dataset to load. Returns: A list of dictionaries, each representing a row in the dataset. **Prerequisites** .. code-block:: shell pip install datasets Examples -------- .. code-block:: python from langchain.evaluation import load_dataset ds = load_dataset("llm-math") r) load_datasetzXload_dataset requires the `datasets` package. Please install with `pip install datasets`zLangChainDatasets/cSsg|]}|qSr%).0dr%r%j/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain/evaluation/loading.py Fz load_dataset..train)Zdatasetsr$ ImportError)r"r$Zdatasetr%r%r(r$'s  r$_EVALUATOR_MAPN)llm) evaluatorr.kwargsr#c Ks|tvr$td|dttt|}t|trzfzddlm}Wn>tyzddl m}WntytdYn0Yn0|p|dddd}Wn6t y}ztd |d |WYd }~n d }~00|j fd |i|S|fi|Sd S) a<Load the requested evaluation chain specified by a string. Parameters ---------- evaluator : EvaluatorType The type of evaluator to load. llm : BaseLanguageModel, optional The language model to use for evaluation, by default None **kwargs : Any Additional keyword arguments to pass to the evaluator. Returns ------- Chain The loaded evaluation chain. Examples -------- >>> from langchain.evaluation import load_evaluator, EvaluatorType >>> evaluator = load_evaluator(EvaluatorType.QA) zUnknown evaluator type: z Valid types are: r) ChatOpenAIzCould not import langchain_openai or fallback onto langchain_community. Please install langchain_openai or specify a language model explicitly. It's recommended to install langchain_openai AND specify a language model explicitly.zgpt-4*)modelseedZ temperaturezEvaluation with the z requires a language model to function. Failed to create the default 'gpt-4' model. Please manually provide an evaluation LLM or check your openai credentials.Nr.) r- ValueErrorlistkeys issubclassrZlangchain_openair1r,Z&langchain_community.chat_models.openai ExceptionZfrom_llm)r/r.r0Z evaluator_clsr1er%r%r(load_evaluatorcs:      r;)r.config) evaluatorsr.r<r0r#cKsHg}|D]:}|r||ini}|t|fd|ii||q|S)aeLoad evaluators specified by a list of evaluator types. Parameters ---------- evaluators : Sequence[EvaluatorType] The list of evaluator types to load. llm : BaseLanguageModel, optional The language model to use for evaluation, if none is provided, a default ChatOpenAI gpt-4 model will be used. config : dict, optional A dictionary mapping evaluator types to additional keyword arguments, by default None **kwargs : Any Additional keyword arguments to pass to all evaluators. Returns ------- List[Chain] The loaded evaluators. Examples -------- >>> from langchain.evaluation import load_evaluators, EvaluatorType >>> evaluators = [EvaluatorType.QA, EvaluatorType.CRITERIA] >>> loaded_evaluators = load_evaluators(evaluators, criteria="helpfulness") r.)getappendr;)r=r.r<r0Zloadedr/_kwargsr%r%r(load_evaluatorss !$rA)M__doc__typingrrrrrrrZlangchain_core.language_modelsr Zlangchain.chains.baser Z1langchain.evaluation.agents.trajectory_eval_chainr Zlangchain.evaluation.comparisonr Z*langchain.evaluation.comparison.eval_chainr Z(langchain.evaluation.criteria.eval_chainrrZ,langchain.evaluation.embedding_distance.baserrZ%langchain.evaluation.exact_match.baserZ!langchain.evaluation.parsing.baserrZ*langchain.evaluation.parsing.json_distancerZ(langchain.evaluation.parsing.json_schemarZlangchain.evaluation.qarrrZ%langchain.evaluation.regex_match.baserZlangchain.evaluation.schemarrrZ'langchain.evaluation.scoring.eval_chainrrZ)langchain.evaluation.string_distance.baser r!strr$ZQAZCOT_QAZ CONTEXT_QAZPAIRWISE_STRINGZ SCORE_STRINGZLABELED_PAIRWISE_STRINGZLABELED_SCORE_STRINGZAGENT_TRAJECTORYZCRITERIAZLABELED_CRITERIAZSTRING_DISTANCEZPAIRWISE_STRING_DISTANCEZEMBEDDING_DISTANCEZPAIRWISE_EMBEDDING_DISTANCEZ JSON_VALIDITYZ JSON_EQUALITYZJSON_EDIT_DISTANCEZJSON_SCHEMA_VALIDATIONZ REGEX_MATCHZ EXACT_MATCHr-__annotations__r;dictrAr%r%r%r(st$         % F