a bgt@s4dZddlmZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZdd lmZed gZd Zd Zd Zee d j!Z"e#e dj!Z$ej%j!Z&ddddZ'dddddZ(dddddZ)dddddZ*dddddZ+GdddeZ,dS) z&Wrapper around TileDB vector database.) annotationsN)AnyDictIterableListMappingOptionalTuple)Document) Embeddings guard_import) VectorStore)maximal_marginal_relevanceZ euclideanZ documentsvectorsuint64float32rreturncCstdtdfS)z@Import tiledb-vector-search if available, otherwise raise error.tiledb.vector_searchtiledbr rru/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/vectorstores/tiledb.pydependable_tiledb_importsrstr)grouprcCs |tjS) Get the URI of the vector index.)VECTOR_INDEX_NAMEurirrrrget_vector_index_uri_from_group#sr cCs |tjS)zGet the URI of the documents array from group. Args: group: TileDB group object. Returns: URI of the documents array. )DOCUMENTS_ARRAY_NAMErrrrr"get_documents_array_uri_from_group(s r")rrcCs|dtS)r/)rrrrrget_vector_index_uri4sr%cCs|dtS)z#Get the URI of the documents array.r#)r!r$rrrget_documents_array_uri9sr&c@sxeZdZdZddddddddddddd d d d d d ZeddddZddeddddddddddZddddddddd ddd d!Z ddddddddd dd"d#d$Z d\ddddd d%dd&d'Z d]ddddd d%d"d(d)Z ddd*dd+dddddd dd,d-d.Z d^dddddd d%d,d/d0Zd_dddddd d%d1d2d3Zed4dd5dddd6d dd7d8d9d:Zedded;ddd?ddd@dAddddd ddB dCdDZd`dAdd dEdFdGdHZdadId@dAdd d>dJdKdLZeddedMd;dddd@dAdddddd ddN dOdPZedded;ddsz0TileDB.process_index_results..c3s"|]\}}j||vVqdSrF)rSr>rW) result_docrr sz/TileDB.process_index_results..cs g|]\}}|kr||fqSrr)rXdocscore)rLrr z0TileDB.process_index_results..)r openr*r,r+zip MAX_UINT64 MAX_FLOAT_32lenr8r>r rpickleloadsnparraytolistastypeuint8tobytesrSitemsallappendr<)rBrOrPrJrKrLrdocs docs_arrayidxr_r^Zpickled_metadatarSr)r\rLrprocess_index_resultss@    zTileDB.process_index_resultsrJrKfetch_k)r/rJrKrxr2rc Kszd|vr|d}nt}|jjtt|tjgtjfd|durN|n|i|\}}|j|d|d|||dS)a[Return docs most similar to query. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, Any]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: nprobe: Optional, number of partitions to check if using IVF_FLAT index score_threshold: Optional, a floating point value to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and distance in float for each. Lower score represents more similarity. rLrJNrrOrPrKrJrL) pop MAX_FLOATr@queryrirjrlrru) rBr/rJrKrxr2rLdirrr&similarity_search_with_score_by_vectors   z-TileDB.similarity_search_with_score_by_vector)r|rJrKrxr2rcKs(||}|j|f|||d|}|S)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with Distance as float. Lower score represents more similarity. rw)r9r)rBr|rJrKrxr2r/rrrrrsimilarity_search_with_scores z#TileDB.similarity_search_with_scorezList[Document]cKs(|j|f|||d|}dd|DS)aReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. rwcSsg|] \}}|qSrrrXr^_rrrr`5raz6TileDB.similarity_search_by_vector..)r)rBr/rJrKrxr2docs_and_scoresrrrsimilarity_search_by_vectorsz"TileDB.similarity_search_by_vectorcKs(|j|f|||d|}dd|DS)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. rwcSsg|] \}}|qSrrrrrrr`Nraz,TileDB.similarity_search..)r)rBr|rJrKrxr2rrrrsimilarity_search7szTileDB.similarity_search?rJrx lambda_multrK)r/rJrxrrKr2rc sd|vr|d}nt}jjtt|tjgtjfd|durN|n|di|\}} j| d|d||dur|n|d|d} fdd| D} t tj|gtjd | ||d } g} | D]}| | |q| S) azReturn docs and their similarity scores selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents and similarity scores selected by maximal marginal relevance and score for each. rLrJNrrycs$g|]\}}j|jgdqSr)r/embed_documentsrTrrGrrr`}szMTileDB.max_marginal_relevance_search_with_score_by_vector..dtype)rJr) rzr{r@r|rirjrlrrurrq)rBr/rJrxrrKr2rLrPindicesresultsrHZ mmr_selectedrr~rrGr2max_marginal_relevance_search_with_score_by_vectorPs<    z9TileDB.max_marginal_relevance_search_with_score_by_vectorcKs*|j|f||||d|}dd|DS)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. rcSsg|] \}}|qSrrrrrrr`razBTileDB.max_marginal_relevance_search_by_vector..)r)rBr/rJrxrrKr2rrrr'max_marginal_relevance_search_by_vectorsz.TileDB.max_marginal_relevance_search_by_vector)r|rJrxrrKr2rc Ks*||}|j|f||||d|}|S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering (if needed) to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. r)r9r) rBr|rJrxrrKr2r/rrrrrmax_marginal_relevance_searchs z$TileDB.max_marginal_relevance_searchT) metadatasr+znp.dtypeNone)r0r5 dimensions vector_typerr+rc Cstdtd}}|j|dPz||Wn*|jyZ} z| WYd} ~ n d} ~ 00||d} t| j} t| j} |dkr|jj | |||dn|dkr|j j | |||d| j | t d|j d d td fttjd } || }|jd tddd}|g}|r,|jdtjdd}|||j|dd|d}|j | || j | td| Wdn1sx0YdS)Nrrr3wr6)rrrr+r7)nameidr)rdomainrrRZU1T)rrvarrSF)rsparseZallows_duplicatesattrs)r r:Z group_createZ TileDBErrorr;r%rr&r?createrAaddrZDimrdrirrDomainZAttrrmrqZ ArraySchemaZArrayr!r<)clsr0r5rrrr+rCrerrrr)docs_uridimdomZ text_attrr metadata_attrZschemarrrrs\        z TileDB.creater6r)rrOr1r5r+index_timestampz List[str]zList[List[float]]zOptional[List[dict]]zOptional[List[str]]) textsrHr/r0rrOr1r5r+rr2rc Ks|tvr td|dtttdtd} } t|tj}|j|||j d|j |du| d| j | dZ|stdt |}t |}|durd d |D}t|tj}| jjf||||| d kr| nd| d | | |d }|dur.tjt|tjd}tt|D]}|||<qi}t||d<|durtjt|gtd}d }|D](}tjt|tjd||<|d7}qf||d<|||<Wdn1s0YWdn1s0Y|f|||| d| S)NzUnsupported distance metric: z. Expected one of rrr)r0r5rrrr+r3z3embeddings must be provided to build a TileDB indexcSs g|]}ttdtdqSrrrrandomrandintrdrXrrrrr`?raz!TileDB.__from..r)r5r0 input_vectors external_idsrr+rrrRrS)r/r0r1r+) INDEX_METRICSr8rVr rirjrlrrshaperr:r%r&rZ ingestionZingestrbZzerosrfrangeemptyobject frombufferrgdumpsrm)rrrHr/r0rrOr1r5r+rr2rCrrr)rrAr~datarrSrrrZ__fromsz      Hz TileDB.__fromzOptional[bool])rOr,r2rcKs2t|tj}|jj||dkr&|ndddS)amDelete by vector ID or other criteria. Args: ids: List of ids to delete. timestamp: Optional timestamp to delete with. **kwargs: Other keyword arguments that subclasses might use. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. rN)rr,T)rirjrlrr@Z delete_batch)rBrOr,r2rrrrdeletees z TileDB.deletez Iterable[str])rrrOr,r2rcKs:td}|jt|}|dur.dd|D}t|tj}tjt |dd} t t |D]} tj|| tj d| | <q^|j j | ||dkr|nddi} t|| d <|durtjt |gtd} d} |D]&} tjt| tjd| | <| d 7} q| | d <|j|jd |dkr|nd|jd }| ||<||S)aRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional ids of each text object. timestamp: Optional timestamp to write new texts with. kwargs: vectorstore specific parameters Returns: List of ids from adding the texts into the vectorstore. rNcSs g|]}ttdtdqSrrrrrrr`raz$TileDB.add_texts..Orr)rrr,rRrrSrrQ)r r/rrVrirjrlrrrfrrr@Z update_batchrrrgrrmrbr*r+r<)rBrrrOr,r2rrHrrr~rrrrSrsrrr add_textszs>  zTileDB.add_textsz/tmp/tiledb_array) rr/rrOr1r0r5r+rr2rc Ks4g} ||} |jf|| |||||||| d | S)aConstruct a TileDB index from raw documents. Args: texts: List of documents to index. embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. ids: Optional ids of each text object. metric: Metric to use for indexing. Defaults to "euclidean". index_uri: The URI to write the TileDB arrays index_type: Optional, Vector index type ("FLAT", IVF_FLAT") config: Optional, TileDB config index_timestamp: Optional, timestamp to write new texts with. Example: .. code-block:: python from langchain_community import TileDB from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() index = TileDB.from_texts(texts, embeddings) rrHr/rrOr1r0r5r+r)r _TileDB__from) rrr/rrOr1r0r5r+rr2rHrrr from_textss #  zTileDB.from_textszList[Tuple[str, List[float]]]) text_embeddingsr/r0rrOr1r5r+rr2rc KsBdd|D} dd|D} |jf| | |||||||| d | S)aConstruct TileDB index from embeddings. Args: text_embeddings: List of tuples of (text, embedding) embedding: Embedding function to use. index_uri: The URI to write the TileDB arrays metadatas: List of metadata dictionaries to associate with documents. metric: Optional, Metric to use for indexing. Defaults to "euclidean". index_type: Optional, Vector index type ("FLAT", IVF_FLAT") config: Optional, TileDB config index_timestamp: Optional, timestamp to write new texts with. Example: .. code-block:: python from langchain_community import TileDB from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = list(zip(texts, text_embeddings)) db = TileDB.from_embeddings(text_embedding_pairs, embeddings) cSsg|] }|dqSrrrXtrrrr` raz*TileDB.from_embeddings..cSsg|] }|dqS)rrrrrrr` rar)r) rrr/r0rrOr1r5r+rr2rrHrrrfrom_embeddingss % zTileDB.from_embeddings)r1r+r,)r0r/r1r+r,r2rcKs|f|||||d|S)a}Load a TileDB index from a URI. Args: index_uri: The URI of the TileDB vector index. embedding: Embeddings to use when generating queries. metric: Optional, Metric to use for indexing. Defaults to "euclidean". config: Optional, TileDB config timestamp: Optional, timestamp to use for opening the arrays. )r/r0r1r+r,r)rr0r/r1r+r,r2rrrloadsz TileDB.load)r2rcKs|jjfi||_dSrF)r@consolidate_updates)rBr2rrrr7szTileDB.consolidate_updates)rINrv)rINrv)rIrvrN)rIrvrN)Nr)NNr)__name__ __module__ __qualname____doc__rEpropertyrHr{rurrrrrrr classmethodrDEFAULT_METRICrrrrrrrrrrrr'>s"J=+# >'% ?*Q7&2(5 r')-r __future__rrgrsystypingrrrrrrr numpyriZlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.utilsr Zlangchain_core.vectorstoresrZ&langchain_community.vectorstores.utilsr frozensetrrr!rZiinformaxrdZfinfore float_infor{rr r"r%r&r'rrrrs0 $