a bgWL@sddlmZddlZddlZddlZddlmZmZmZm Z m Z m Z m Z ddl ZddlmZddlmZddlmZddlmZddlmZd Zd Zd Zd Zd ZdZdZGdddeZ dS)) annotationsN)AnyDictListOptionalSequenceTupleType)Document) Embeddings)UpsertResponse) VectorStore)overrideZHNSWCSZ langchainiZlc_textuniqueidc seZdZeedddejdfddddddddd d fd d Zd d ddZedOddddddZ edddddZ edPddddddddZ edddddd d!Z d"d#d$d%d&Z dQd(dd)dd*d+d,ZedRd(dddd-d.d/ZedSdddd2ddd3d4d5ZedTd(ddd2ddd6d7d8ZeedUd9d:dd;ddd<d=d>Zeed9ddddd?d@dAZedd dBdCdDZed d dEdFZddd:dGdHdIZdJddKdLdMdNZZS)V ApertureDBNr strz Optional[int]z Optional[str]intzOptional[Dict]rNone) embeddingsdescriptor_set dimensionsenginemetric log_level propertieskwargsreturnc  s"zddlm} m} Wnty.tdYn0tjfi|tt|_ |j |||_ ||_ ||_ ||_||_||_|dur|j dtdzddlm} m} Wn ty|j dYn0| |_| |j|_z|jWn"ty|j dYn0|dS)aCreate a vectorstore backed by ApertureDB A single ApertureDB instance can support many vectorstores, distinguished by 'descriptor_set' name. The descriptor set is created if it does not exist. Different descriptor sets can use different engines and metrics, be supplied by different embedding models, and have different dimensions. See ApertureDB documentation on `AddDescriptorSet` https://docs.aperturedata.io/query_language/Reference/descriptor_commands/desc_set_commands/AddDescriptorSet for more information on the engine and metric options. Args: embeddings (Embeddings): Embeddings object descriptor_set (str, optional): Descriptor set name. Defaults to "langchain". dimensions (Optional[int], optional): Number of dimensions of the embeddings. Defaults to None. engine (str, optional): Engine to use. Defaults to "HNSW" for new descriptorsets. metric (str, optional): Metric to use. Defaults to "CS" for new descriptorsets. log_level (int, optional): Logging level. Defaults to logging.WARN. rUtilscreate_connectorzMApertureDB is not installed. Please install it using 'pip install aperturedb'NzNo embedding function provided.zFailed to connect to ApertureDB)aperturedb.Utilsr r! ImportErrorsuper__init__logging getLogger__name__loggersetLevelrembedding_functionrrrrfatal ValueError exception connectionutilsstatus Exception_find_or_add_descriptor_set) selfrrrrrrrrr r! __class__y/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/vectorstores/aperturedb.pyr%sB%        zApertureDB.__init__)rc Cs|j}d|dddddidig}|j|\}}|jsBJ|d|ddvrft|dddnd}|dkszJd|dkr|dddd}|jd |d |d }t|dksJd |jdur|d|_n.|j|dkr |jd |jd|d|d}t|dks*Jd|j durB|d|_ n.|j |dkrp|jd|j d|d|d} |j dur| |_ n&|j | kr|jd|j d| dd| D|_ n|jd |d|jdurt |_|j durt|_ |j durt|jd|_ |j dur>dd|j Dnd} |jj||j |j|j | d|jdd|jdd|jdtdS)NFindDescriptorSetTall_properties) with_nameenginesmetricsrresultsentitiesrz+Multiple descriptor sets with the same namezDescriptor set z already existsZ_engineszOnly one engine is supportedzEngine mismatch: z != Z_metricszOnly one metric is supportedzMetric mismatch: Z _dimensionszDimensions mismatch: cSs,i|]$\}}|tr|ttd|qSN) startswithPROPERTY_PREFIXlen.0kvr7r7r8 s z:ApertureDB._find_or_add_descriptor_set..z does not exist. Creating ittestcSsi|]\}}t||qSr7rCrEr7r7r8rI)namedimrrrZ _DescriptorZ _create_txnZ_DescriptorSet_name)rr/query last_query_okrDr)inforerrorrritemsrENGINEMETRICr+ embed_queryr0Zadd_descriptorsetZcreate_entity_indexUNIQUEID_PROPERTY) r4rZ find_ds_queryrbZ n_entitieser<r=rrr7r7r8r3ls              z&ApertureDB._find_or_add_descriptor_setzOptional[List[str]]zOptional[bool])idsrrcKs>|dusJdd|jtd|gidig}|j|\}}|S)zDelete documents from the vectorstore by id. Args: ids: List of ids to delete from the vectorstore. Returns: True if the deletion was successful, False otherwise Nzids must be providedZDeleteDescriptorin)set constraints)rrXr0execute)r4r\rrPresult_r7r7r8deletes   zApertureDB.deletez Sequence[str]zList[Document])r\rcsVdjtd|giddidig}j|\}}fdd|ddd gD}|S) zFind documents in the vectorstore by id. Args: ids: List of ids to find in the vectorstore. Returns: documents: List of Document objects found in the vectorstore. ZFindDescriptorr]r:T)r^r_r>csg|]}|qSr7_descriptor_to_documentrFdr4r7r8 sz)ApertureDB.get_by_ids..rr?)rrXr0r`get)r4r\rPr>rbdocsr7rhr8 get_by_idss    zApertureDB.get_by_ids)rPrGargsrrcOs8|jdusJd|j|}|j||g|Ri|S)aSearch for documents similar to the query using the vectorstore Args: query: Query string to search for. k: Number of results to return. Returns: List of Document objects ordered by decreasing similarity to the query. NzEmbedding function is not set)r+rWsimilarity_search_by_vector)r4rPrGrnr embeddingr7r7r8similarity_searchs  zApertureDB.similarity_searchzList[Tuple[Document, float]])rPrnrrcOs$|j|}|j|g|Ri|SrA)r+rW'_similarity_search_with_score_by_vector)r4rPrnrrpr7r7r8similarity_search_with_scores z'ApertureDB.similarity_search_with_scoredictr )rgrcCsVi}|D]&\}}|tr |||ttd<q |t}|t}t|||d}|S)N page_contentmetadataid)rTrBrCrD TEXT_PROPERTYrXr )r4rgrwrGrHrrdocr7r7r8re s z"ApertureDB._descriptor_to_documentFz List[float]bool)rprGvectorsrcs`ddlm}|j}t}|jj||ddjdt|dfdd|DS) Nr DescriptorsT)r^vector k_neighborsZ distances"ApertureDB similarity search took  secondscsg|]}||dfqS)Z _distancerdrfrhr7r8ri#rLzFApertureDB._similarity_search_with_score_by_vector..aperturedb.Descriptorsr~r/timeZ find_similarrr)rR)r4rprGr|r~ descriptors start_timer7rhr8rrs   z2ApertureDB._similarity_search_with_score_by_vector)rprGrrc s^ddlm}|j}t}|jj||djdt|dfdd|DS)a3Returns the k most similar documents to the given embedding vector Args: embedding: The embedding vector to search for k: The number of similar documents to return Returns: List of Document objects ordered by decreasing similarity to the query. rr})r^rrrrcsg|]}|qSr7rdrfrhr7r8ri<rLz:ApertureDB.similarity_search_by_vector..r)r4rprGrr~rrr7rhr8ro%s  z&ApertureDB.similarity_search_by_vector?float)rPrGfetch_k lambda_multrrcKs6|jd||j|}|j||||fi|S)aReturns similar documents to the query that also have diversity This algorithm balances relevance and diversity in the search results. Args: query: Query string to search for. k: Number of results to return. fetch_k: Number of results to fetch. lambda_mult: Lambda multiplier for MMR. Returns: List of Document objects ordered by decreasing similarity/diversty. z)Max Marginal Relevance search for query: )r)rRr+rW'max_marginal_relevance_search_by_vector)r4rPrGrrrrpr7r7r8max_marginal_relevance_search>s z(ApertureDB.max_marginal_relevance_search)rprGrrrrc  sbddlm}|j}t}|jj||||djdt|dfdd|DS)aReturns similar documents to the vector that also have diversity This algorithm balances relevance and diversity in the search results. Args: embedding: Embedding vector to search for. k: Number of results to return. fetch_k: Number of results to fetch. lambda_mult: Lambda multiplier for MMR. Returns: List of Document objects ordered by decreasing similarity/diversty. rr})r^rrrrz&ApertureDB similarity search mmr took rcsg|]}|qSr7rdrfrhr7r8ri~rLzFApertureDB.max_marginal_relevance_search_by_vector..)rr~r/rZfind_similar_mmrrr)rR) r4rprGrrrr~rrr7rhr8rZs  z2ApertureDB.max_marginal_relevance_search_by_vectorzType[ApertureDB]z List[str]zOptional[List[dict]])clstextsrp metadatasrrcKs"|fd|i|}||||S)aLCreates a new vectorstore from a list of texts Args: texts: List of text strings embedding: Embeddings object as for constructing the vectorstore metadatas: Optional list of metadatas associated with the texts. kwargs: Additional arguments to pass to the constructor r)Z add_texts)rrrprrstorer7r7r8 from_textss zApertureDB.from_texts)r documentsrprrcKs |fd|i|}|||S)aXCreates a new vectorstore from a list of documents Args: documents: List of Document objects embedding: Embeddings object as for constructing the vectorstore metadatas: Optional list of metadatas associated with the texts. kwargs: Additional arguments to pass to the constructor r) add_documents)rrrprrr7r7r8from_documentss zApertureDB.from_documents)rrcCs,ddlm}m}|}||}||dS)zDeletes a vectorstore and all its data from the database Args: descriptor_set: The name of the descriptor set to delete rrN)r"r r!Zremove_descriptorset)class_rr r!dbr0r7r7r8delete_vectorstoreszApertureDB.delete_vectorstorecCsXddlm}|}dddiddddig}||\}}|sHJ||dddS)zReturns a list of all vectorstores in the database Returns: List of descriptor sets with properties r)r!r9r:T)r>r<r=rr?)r"r!rPrQ)rr!rrPresponserbr7r7r8list_vectorstoress  zApertureDB.list_vectorstores)rrrcKs~d|vrd|d}|r.t|t|kr.tdg}t||D]$\}}t|j|j|d}||q.cSsg|] }|jqSr7)rvrFrzr7r7r8ri rLcSs&g|]}t|dddur|jniqS)rwN)getattrrwrr7r7r8riscSs2g|]*}t|dr"|jdur"|jn ttqSr)rrxruuiduuid4rr7r7r8riscSsi|]\}}t||qSr7rKrEr7r7r8rIrLz%ApertureDB.upsert..Z AddDescriptor)r^r)Zdtype)Z batchsize)rfailed)Zaperturedb.ParallelLoaderrrcr+Zembed_documentsrrTryrXrnparrayZfloat32tobytesrr/Zingest BATCHSIZEr )r4rTrrZ ids_to_deleterrrr\datarrprw unique_idrcommandrPZblobsloaderr7r7r8rs@    zApertureDB.upsert)N)rm)rmF)rm)rmrr)rmrr)N)r( __module__ __qualname__rDESCRIPTOR_SETr&WARNr%r3rcrlrqrsrerrrorr classmethodrrrrrr __classcell__r7r7r5r8rs^&MV % 'r)! __future__rr&rrtypingrrrrrrr numpyrZlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.indexing.baser Zlangchain_core.vectorstoresr Ztyping_extensionsrrUrVrrrCryrXrr7r7r7r8s$ $