a bg F@sddlmZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lm Z e!gd Z"dZ#ddddZ$GdddeZ%dS)) annotationsN) ConfigParser)Path)AnyCallableDictIterableListOptionalTuple)Document) Embeddings guard_import) VectorStore)Docstore)InMemoryDocstore)maximal_marginal_relevance)angularZ euclideanZ manhattanZhammingdotrrreturncCstdS)z1Import annoy if available, otherwise raise error.annoyrrrt/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/vectorstores/annoy.pydependable_annoy_importsrc @seZdZdZdddddddd Zed d d d ZdQdddddddZddddddZdRddddddd Z dSddddd!d"d#Z dTddddd$d%d&Z dUddddd'd(d)d*Z dVddddd'd+d,d-Z dWddddd'd.d/d0ZdXdddd3dd'd4d5d6ZdYdddd3dd'd7d8d9Zeded:dfdd;dd?Zeded:dfddszAnnoy.add_textsz List[int]z List[float]zList[Tuple[Document, float]])idxsdistsrcCs^g}t||D]J\}}|j|}|j|}t|tsJtd|d||||fq|S)aTurns annoy results into a list of documents and scores. Args: idxs: List of indices of the documents in the index. dists: List of distances of the documents in the index. Returns: List of Documents and scores. Could not find document for id , got )zipr#r"search isinstancer ValueErrorappend)r$r-r.docsidxdist_iddocrrrprocess_index_resultsHs    zAnnoy.process_index_resultsint) embeddingksearch_krcCs$|jj|||dd\}}|||Sa}Return docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query and score for each TrAZinclude_distances)r get_nns_by_vectorr;)r$r?r@rAr-r.rrr&similarity_search_with_score_by_vector\s  z,Annoy.similarity_search_with_score_by_vector)docstore_indexr@rArcCs$|jj|||dd\}}|||SrB)r Zget_nns_by_itemr;)r$rFr@rAr-r.rrr%similarity_search_with_score_by_indexns  z+Annoy.similarity_search_with_score_by_index)queryr@rArcCs||}||||}|S)a~Return docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query and score for each )rrE)r$rHr@rAr?r6rrrsimilarity_search_with_scores z"Annoy.similarity_search_with_scorezList[Document])r?r@rAr*rcKs||||}dd|DS)aReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the embedding. cSsg|] \}}|qSrr.0r:_rrr z5Annoy.similarity_search_by_vector..)rE)r$r?r@rAr*docs_and_scoresrrrsimilarity_search_by_vectorsz!Annoy.similarity_search_by_vector)rFr@rAr*rcKs||||}dd|DS)azReturn docs most similar to docstore_index. Args: docstore_index: Index of document in docstore k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the embedding. cSsg|] \}}|qSrrrJrrrrMrNz4Annoy.similarity_search_by_index..)rG)r$rFr@rAr*rOrrrsimilarity_search_by_indexsz Annoy.similarity_search_by_index)rHr@rAr*rcKs||||}dd|DS)alReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query. cSsg|] \}}|qSrrrJrrrrMrNz+Annoy.similarity_search..)rI)r$rHr@rAr*rOrrrsimilarity_searchszAnnoy.similarity_search?float)r?r@fetch_k lambda_multr*rc  sjj||dddfddD}ttj|gtjd|||d}fdd|D}g} |D]B} j| } j| } t | t st d | d | | | q^| S) aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. fetch_k: Number of Documents to fetch to pass to MMR algorithm. k: Number of Documents to return. Defaults to 4. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. r=FrCcsg|]}j|qSr)r Zget_item_vectorrKir&rrrMrNzAAnnoy.max_marginal_relevance_search_by_vector..)Zdtype)r@rWcsg|]}|dkr|qS)r=rrX)r-rrrMrNr/r0) r rDrnparrayZfloat32r#r"r2r3r r4r5) r$r?r@rVrWr*r'Z mmr_selectedZselected_indicesr6rYr9r:r)r-r$r'max_marginal_relevance_search_by_vectors&    z-Annoy.max_marginal_relevance_search_by_vector)rHr@rVrWr*rcKs ||}|j||||d}|S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. )rW)rr\)r$rHr@rVrWr*r?r6rrrmax_marginal_relevance_searchs  z#Annoy.max_marginal_relevance_searchdzList[List[float]]r ) r(r'r?r)r!treesn_jobsr*rc s|tvr td|dtttd} |s4tdt|d} | j| |d} t|D]\} } | | | qV| j||dg}t|D]*\} }|r|| ni}| t ||dqd d t t|Dt fd d t|D}||j | ||S) NzUnsupported distance metric: z. Expected one of rz/embeddings must be provided to build AnnoyIndexrr!)r`)Z page_contentmetadatacSsi|]}|ttqSr)ruuiduuid4rXrrr 6rNz Annoy.__from..csi|]\}}||qSrr)rKrYr:Z index_to_idrrre8rN) INDEX_METRICSr4listrlen AnnoyIndex enumerateZadd_itembuildr5r ranger embed_query)clsr(r'r?r)r!r_r`r*rfr rYZembZ documentstextrbr"rrfrZ__froms.  z Annoy.__from)r(r?r)r!r_r`r*rc Ks(||}|j|||||||fi|S)aConstruct Annoy wrapper from raw documents. Args: texts: List of documents to index. embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. metric: Metric to use for indexing. Defaults to "angular". trees: Number of trees to use for indexing. Defaults to 100. n_jobs: Number of jobs to use for indexing. Defaults to -1. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the Annoy database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import Annoy from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() index = Annoy.from_texts(texts, embeddings) )Zembed_documents _Annoy__from) ror(r?r)r!r_r`r*r'rrr from_texts<s $ zAnnoy.from_textszList[Tuple[str, List[float]]])text_embeddingsr?r)r!r_r`r*rc Ks:dd|D}dd|D} |j|| |||||fi|S)aConstruct Annoy wrapper from embeddings. Args: text_embeddings: List of tuples of (text, embedding) embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. metric: Metric to use for indexing. Defaults to "angular". trees: Number of trees to use for indexing. Defaults to 100. n_jobs: Number of jobs to use for indexing. Defaults to -1 This is a user friendly interface that: 1. Creates an in memory docstore with provided embeddings 2. Initializes the Annoy database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import Annoy from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = list(zip(texts, text_embeddings)) db = Annoy.from_embeddings(text_embedding_pairs, embeddings) cSsg|] }|dqS)rrrKtrrrrMrNz)Annoy.from_embeddings..cSsg|] }|dqS)rrurrrrMrN)rr) rortr?r)r!r_r`r*r(r'rrrfrom_embeddingses%zAnnoy.from_embeddingsFboolNone) folder_pathprefaultrcCst|}tj|ddt}|jj|jd|d<|jjt|d|dt |dd&}t |j |j |f|Wd n1s0Yd S) aSave Annoy index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to save index, docstore, and index_to_docstore_id to. prefault: Whether to pre-load the index into memory. T)exist_ok)rpr!ANNOY index.annoy)r| index.pklwbN)rosmakedirsrr rpr!saveropenpickledumpr"r#)r$r{r|path config_objectfilerrr save_locals zAnnoy.save_local)allow_dangerous_deserialization)r{r'rrc Cs|s tdt|}td}t|dd }t|\}}} Wdn1sP0Yt| dd} | dd} |j| | d } | t|d ||j | | ||S) aRLoad Annoy index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to load index, docstore, and index_to_docstore_id from. embeddings: Embeddings to use when generating queries. allow_dangerous_deserialization: whether to allow deserialization of the data which involves loading a pickle file. Pickle files can be modified by malicious actors to deliver a malicious payload that results in execution of arbitrary code on your machine. aBThe de-serialization relies loading a pickle file. Pickle files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to enable deserialization. If you do this, make sure that you trust the source of the data. For example, if you are loading a file that you created, and know that no one else has modified the file, then this is safe to do. Do not set this to `True` if you are loading a file from an untrusted source (e.g., some random site on the internet.).rrrbNr~rpr!rar) r4rrrrloadr>rjrrn) ror{r'rrrrr"r#rrpr!r rrr load_locals,   zAnnoy.load_local)N)r<r=)r<r=)r<r=)r<r=)r<r=)r<r=)r<rSrT)r<rSrT)F)__name__ __module__ __qualname____doc__r%propertyr'r,r;rErGrIrPrQrRr\r] classmethodDEFAULT_METRICrrrsrxrrrrrrrsb  2"% ( +r)& __future__rrrrc configparserrpathlibrtypingrrrrr r r numpyrZZlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.utilsrZlangchain_core.vectorstoresrZ!langchain_community.docstore.baserZ&langchain_community.docstore.in_memoryrZ&langchain_community.vectorstores.utilsr frozensetrgrrrrrrrs$   $