a bgR@sddlmZddlZddlZddlmZmZmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZerddlZeZdZGd d d eZdS) ) annotationsN) TYPE_CHECKINGAnyDictIterableListOptionalSetTupleType)Document) Embeddings) VectorStore)maximal_marginal_relevancec @seZdZUdZdZded<edddfddddd d d d d ZeddddZdTdddd ddddZ dd ddddZ e ddfddddd ddd d!Z e ddfddddd d"dd#d$Z dUddd d"d&d'd(Zde dddfd)dddd*d dd+d,d-ZdVdddd0ddd dd1d2d3ZdWd4ddd0ddd dd5d6d7ZdXd8ddd*d9d d:d;dd?d@ZdZdddd ddAdBdCZdd dddDdEZdd dddFdGZd ddHdIdJZd ddHdKdLZeddeddfdMddddddd ddN dOdPZededdfdMdddddd ddQdRdSZdS)[AwaDBz`AwaDB` vector store.Zlangchain_awadbstr_DEFAULT_TABLE_NAMENzOptional[Embeddings]z Optional[str]zOptional[awadb.Client]rNone) table_name embeddinglog_and_data_dirclientkwargsreturncKsz ddl}Wnty&tdYn0|dur8||_n |durN|||_n ||_||jkr|d7}|ttdd7}|j |i|_ |dur||j |<||_ dS)a@Initialize with AwaDB client. If table_name is not specified, a random table name of `_DEFAULT_TABLE_NAME + last segment of uuid` would be created automatically. Args: table_name: Name of the table created, default _DEFAULT_TABLE_NAME. embedding: Optional Embeddings initially set. log_and_data_dir: Optional the root directory of log and data. client: Optional AwaDB client. kwargs: Any possible extend parameters in the future. Returns: None. rNzRCould not import awadb python package. Please install it with `pip install awadb`._-) awadb ImportError awadb_clientZClientrruuiduuid4splitCreatetable2embeddingsusing_table_name)selfrrrrrrr(t/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/vectorstores/awadb.py__init__s&       zAwaDB.__init__)rcCs|j|jvr|j|jSdSN)r&r%)r'r(r(r) embeddingsKs  zAwaDB.embeddingsz Iterable[str]zOptional[List[dict]]zOptional[bool]z List[str])texts metadatasis_duplicate_textsrrcKsN|jdurtdd}|j|jvr8|j|jt|}|jdd||||S)aRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. is_duplicate_texts: Optional whether to duplicate texts. Defaults to True. kwargs: any possible extend parameters in the future. Returns: List of ids from adding the texts into the vectorstore. NAwaDB client is None!!!embedding_texttext_embedding)r ValueErrorr&r%Zembed_documentslistZAddTexts)r'r-r.r/rr,r(r(r) add_textsQs   zAwaDB.add_textsbool)rrrcKs|jdurtd|j|S)zLoad the local specified table. Args: table_name: Table name kwargs: Any possible extend parameters in the future. Returns: Success or failure of loading the local specified table Nr0)r r3Load)r'rrr(r(r) load_localts zAwaDB.load_localintzOptional[dict]zList[Document])queryktext_in_page_content meta_filterrrc Ksj|jdurtdd}|j|jvr6|j|j|}nddlm}||}hd}|j|||||dS)a8Return docs most similar to query. Args: query: Text query. k: The maximum number of documents to return. text_in_page_content: Filter by the text in page_content of Document. meta_filter (Optional[dict]): Filter by metadata. Defaults to None. E.g. `{"color" : "red", "price": 4.20}`. Optional. E.g. `{"max_price" : 15.66, "min_price": 4.20}` `price` is the metadata field, means range filter(4.20<'price'<15.66). E.g. `{"maxe_price" : 15.66, "mine_price": 4.20}` `price` is the metadata field, means range filter(4.20<='price'<=15.66). kwargs: Any possible extend parameters in the future. Returns: Returns the k most similar documents to the specified text query. Nr0r AwaEmbedding>scorer2_idr<r=not_include_fields_in_metadata) r r3r&r% embed_queryrr? Embeddingsimilarity_search_by_vector) r'r:r;r<r=rrr?not_include_fieldsr(r(r)similarity_searchs    zAwaDB.similarity_searchzList[Tuple[Document, float]]cKs|jdurtdd}|j|jvr6|j|j|}nddlm}||}g}ddh} |j||||| d} | D](} | j d} | j d=| | f} | | qr|S) aThe most k similar documents and scores of the specified query. Args: query: Text query. k: The k most similar documents to the text query. text_in_page_content: Filter by the text in page_content of Document. meta_filter: Filter by metadata. Defaults to None. kwargs: Any possible extend parameters in the future. Returns: The k most similar documents to the specified text query. 0 is dissimilar, 1 is the most similar. Nr0rr>r2rArBr@) r r3r&r%rDrr?rErFmetadataappend)r'r:r;r<r=rrr?resultsrGZretrieval_docsdocr@Z doc_tupler(r(r)similarity_search_with_scores,      z"AwaDB.similarity_search_with_scorer)r:r;rrcKs|j||fi|Sr+)rM)r'r:r;rr(r(r)(_similarity_search_with_relevance_scoressz.AwaDB._similarity_search_with_relevance_scoreszOptional[List[float]]zOptional[Set[str]])rr;r<r=rCrrc Ks|jdurtdg}|dur"|S|jj|||||d}|dkrH|S|ddD]Z} d} i} | D]6} | dkr|| | } qdn|dur| |vrqd| | | | <qd|t| | dqT|S) a Return docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. text_in_page_content: Filter by the text in page_content of Document. meta_filter: Filter by metadata. Defaults to None. not_incude_fields_in_metadata: Not include meta fields of each document. Returns: List of Documents which are the most similar to the query vector. Nr0)r<r=rGrZ ResultItemsr1 page_contentrI)r r3ZSearch__len__rJr ) r'rr;r<r=rCrrKZ show_resultsZ item_detailcontentZ meta_dataZitem_keyr(r(r)rFs6  z!AwaDB.similarity_search_by_vector?float)r:r;fetch_k lambda_multr<r=rrc Ksx|jdurtdg}|j|jvr6|j|j|}nddlm} | |}|dkr^gS|j ||||||d} | S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. text_in_page_content: Filter by the text in page_content of Document. meta_filter (Optional[dict]): Filter by metadata. Defaults to None. Returns: List of Documents selected by maximal marginal relevance. Nr0rr>)rXr<r=) r r3r&r%rDrr?rErR'max_marginal_relevance_search_by_vector) r'r:r;rWrXr<r=rrr?rKr(r(r)max_marginal_relevance_search)s$     z#AwaDB.max_marginal_relevance_searchz List[float])rr;rWrXr<r=rrcKs|jdurtdg}|dur"|Sddh} |j||||| d} g} | D]} | | jdqFttj|tjd| d} | D],}d| |jvrx| |jd=|| |qx|S) aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. text_in_page_content: Filter by the text in page_content of Document. meta_filter (Optional[dict]): Filter by metadata. Defaults to None. Returns: List of Documents selected by maximal marginal relevance. Nr0rAr@rBr2)Zdtype)Zembedding_list) r r3rFrJrIrnparrayZfloat32)r'rr;rWrXr<r=rrKrGZretrieved_docsZtop_embeddingsrLZ selected_docsZs_idr(r(r)rY^s0  z-AwaDB.max_marginal_relevance_search_by_vectorzOptional[List[str]]z Optional[int]zDict[str, Document])idsr<r=rGlimitrrcKs|jdurtd|jj|||||d}i}|D]`} d} i} | D]6} | dkrX| | } q@n| dks@| dkrjq@| | | | <q@t| | d} | || d<q0|S) aReturn docs according ids. Args: ids: The ids of the embedding vectors. text_in_page_content: Filter by the text in page_content of Document. meta_filter: Filter by any metadata of the document. not_include_fields: Not pack the specified fields of each document. limit: The number of documents to return. Defaults to 5. Optional. Returns: Documents which satisfy the input conditions. Nr0)r]r<r=rGr^rOr1r2rArP)r r3ZGetr )r'r]r<r=rGr^rZ docs_detailrKZ doc_detailrSZ meta_infofieldrLr(r(r)gets.  z AwaDB.get)r]rrcKs>|jdurtdd}|dus*|dkr.|S|j|}|S)aJDelete the documents which have the specified ids. Args: ids: The ids of the embedding vectors. **kwargs: Other keyword arguments that subclasses might use. Returns: Optional[bool]: True if deletion is successful. False otherwise, None if not implemented. Nr0r)r r3rRDelete)r'r]rretr(r(r)deletes  z AwaDB.delete)r]r-r.rrcKs&|jdurtd|jj|d||dS)a@Update the documents which have the specified ids. Args: ids: The id list of the updating embedding vector. texts: The texts of the updating documents. metadatas: The metadatas of the updating documents. Returns: the ids of the updated documents. Nr0r1)r]Ztext_field_namer-r.)r r3Z UpdateTexts)r'r]r-r.rr(r(r)updates  z AwaDB.updatecKs(|jdurdS|j|}|r$||_|S)zCreate a new table.NF)r r$r&r'rrrbr(r(r) create_tables   zAwaDB.create_tablecKs(|jdurdS|j|}|r$||_|S)zJUse the specified table. Don't know the tables, please invoke list_tables.NF)r ZUser&rer(r(r)use s   z AwaDB.use)rrcKs|jdurgS|jS)z*List all the tables created by the client.N)r Z ListAllTablesr'rr(r(r) list_tabless zAwaDB.list_tablescKs|jS)zGet the current table.)r&rhr(r(r)get_current_table(szAwaDB.get_current_tablez Type[AwaDB]) clsr-rr.rrrrrc Ks"|||||d}|j||d|S)a3Create an AwaDB vectorstore from a raw documents. Args: texts (List[str]): List of texts to add to the table. embedding (Optional[Embeddings]): Embedding function. Defaults to None. metadatas (Optional[List[dict]]): List of metadatas. Defaults to None. table_name (str): Name of the table to create. log_and_data_dir (Optional[str]): Directory of logging and persistence. client (Optional[awadb.Client]): AwaDB client Returns: AwaDB: AwaDB vectorstore. )rrrr)r-r.)r5) rkr-rr.rrrrr r(r(r) from_texts0szAwaDB.from_texts)rk documentsrrrrrrc Ks2dd|D}dd|D}|j||||||dS)avCreate an AwaDB vectorstore from a list of documents. If a log_and_data_dir specified, the table will be persisted there. Args: documents (List[Document]): List of documents to add to the vectorstore. embedding (Optional[Embeddings]): Embedding function. Defaults to None. table_name (str): Name of the table to create. log_and_data_dir (Optional[str]): Directory to persist the table. client (Optional[awadb.Client]): AwaDB client. Any: Any possible parameters in the future Returns: AwaDB: AwaDB vectorstore. cSsg|] }|jqSr()rQ.0rLr(r(r) jz(AwaDB.from_documents..cSsg|] }|jqSr()rIrnr(r(r)rpkrq)r-rr.rrr)rl) rkrmrrrrrr-r.r(r(r)from_documentsQszAwaDB.from_documents)NN)r)rrTrUNN)rrTrUNN)NNNNN)N)N)__name__ __module__ __qualname____doc__r__annotations__r*propertyr,r5r8 DEFAULT_TOPNrHrMrNrFrZrYr`rcrdrfrgrirj classmethodrlrrr(r(r(r)rs  1#17 :8@4 " r) __future__rloggingr!typingrrrrrrr r r numpyr[Zlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.vectorstoresrZ&langchain_community.vectorstores.utilsrr getLoggerloggerryrr(r(r(r)s ,