a bgf@sddlmZddlZddlmZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZGd d d eZdS) ) annotationsN)Path)AnyDictIterableListOptional)CallbackManagerForRetrieverRunDocument) BaseRetriever) ConfigDictc@seZdZUdZdZded<ded<dZded<dZd ed <ed d Z e d(d dddddddZ e dddddddddZ ddddddZ d)ddddd d!Ze d"dd#dd$ddd%d&d'ZdS)*TFIDFRetrieverz`TF-IDF` retriever. Largely based on https://github.com/asvskartheek/Text-Retrieval/blob/master/TF-IDF%20Search%20Engine%20(SKLEARN).ipynb Nr vectorizerzList[Document]docs tfidf_arrayintkT)Zarbitrary_types_allowedz Iterable[str]zOptional[Iterable[dict]]zOptional[Dict[str, Any]])texts metadatas tfidf_paramskwargsreturnc Kszddlm}Wnty*tdYn0|p2i}|fi|}||}|p\dd|D}ddt||D}|f|||d|S) Nr)TfidfVectorizerzNCould not import scikit-learn, please install with `pip install scikit-learn`.css|] }iVqdSN).0_rrr/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/retrievers/tfidf.py 4z,TFIDFRetriever.from_texts..cSsg|]\}}t||dqS)Z page_contentmetadatar )rtmrrr 5r!z-TFIDFRetriever.from_texts..rrr)Zsklearn.feature_extraction.textr ImportErrorZ fit_transformzip) clsrrrrrrrrrrr from_texts!s   zTFIDFRetriever.from_texts)rzIterable[Document]) documentsrrrcKs.tdd|D\}}|jf|||d|S)Ncss|]}|j|jfVqdSrr")rdrrrr @r!z0TFIDFRetriever.from_documents..)rrr)r)r+)r*r,rrrrrrrfrom_documents8s zTFIDFRetriever.from_documentsstrr )query run_managerrcs\ddlm}j|g}|j|d}fdd|j ddddD}|S)Nr)cosine_similarity)csg|]}j|qSr)r)riselfrrr&Pr!z:TFIDFRetriever._get_relevant_documents..r3)Zsklearn.metrics.pairwiser2rZ transformrZreshapeZargsortr)r6r0r1r2Z query_vecresultsZ return_docsrr5r_get_relevant_documentsEs  ,z&TFIDFRetriever._get_relevant_documentstfidf_vectorizerNone) folder_path file_namercCsz ddl}Wnty&tdYn0t|}|jddd||j||dt||dd$}t|j|j f|Wdn1s0YdS)NrBCould not import joblib, please install with `pip install joblib`.T)exist_okparents.joblib.pklwb) joblibr(rmkdirdumpropenpicklerr)r6r;r<rCpathfrrr save_localSs   zTFIDFRetriever.save_localF)allow_dangerous_deserializationr<bool)r;rKr<rc Csz ddl}Wnty&tdYn0|s4tdt|}|||d}t||dd}t|\}} Wdn1s0Y|||| dS) aLoad the retriever from local storage. Args: folder_path: Folder path to load from. allow_dangerous_deserialization: Whether to allow dangerous deserialization. Defaults to False. The deserialization relies on .joblib and .pkl files, which can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine. You will need to set this to `True` to use deserialization. If you do this, make sure you trust the source of the file. file_name: File name to load from. Defaults to "tfidf_vectorizer". Returns: TFIDFRetriever: Loaded retriever. rNr=aThe de-serialization of this retriever is based on .joblib and .pkl files.Such files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to load this retriever. If you do this, make sure you trust the source of the file, and you are responsible for validating the file came from a trusted source.r@rArbr')rCr( ValueErrorrloadrFrG) r*r;rKr<rCrHrrIrrrrr load_localis    ,zTFIDFRetriever.load_local)NN)r9)__name__ __module__ __qualname____doc__r__annotations__rrr Z model_config classmethodr+r.r8rJrPrrrrr s,     r) __future__rrGpathlibrtypingrrrrrZlangchain_core.callbacksr Zlangchain_core.documentsr Zlangchain_core.retrieversr Zpydanticr rrrrrs