a bgh@sddlmZddlZddlmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZerlddlmZdd d d d ZeeZGd d d eZdS)) annotationsN) TYPE_CHECKINGAnyDictIterableListOptionalTuple)Document) Embeddings) VectorStore HippoClient localhostZ7788Zadmin)hostportusernamepasswordc@s4eZdZdZdAdddd d d d d d ddZdddddZdBddddddZdCddddddZdddd Zd dd!d"Z ddd#d$Z dDd&dd'd d(d)d*d+d,Z dEdd d d.d'd(d/d0d1d2Z dFdd d d.d'd(d3d0d4d5Z dGd6d d d.d'd(d3d7d8d9Zedddedddfd)ddddd:d;d d d?d@ZdS)HHippoa`Hippo` vector store. You need to install `hippo-api` and run Hippo. Please visit our official website for how to run a Hippo instance: https://www.transwarp.cn/starwarp Args: embedding_function (Embeddings): Function used to embed the text. table_name (str): Which Hippo table to use. Defaults to "test". database_name (str): Which Hippo database to use. Defaults to "default". number_of_shards (int): The number of shards for the Hippo table.Defaults to 1. number_of_replicas (int): The number of replicas for the Hippo table.Defaults to 1. connection_args (Optional[dict[str, any]]): The connection args used for this class comes in the form of a dict. index_params (Optional[dict]): Which index params to use. Defaults to IVF_FLAT. drop_old (Optional[bool]): Whether to drop the current collection. Defaults to False. primary_field (str): Name of the primary key field. Defaults to "pk". text_field (str): Name of the text field. Defaults to "text". vector_field (str): Name of the vector field. Defaults to "vector". The connection args used for this class comes in the form of a dict, here are a few of the options: host (str): The host of Hippo instance. Default at "localhost". port (str/int): The port of Hippo instance. Default at 7788. user (str): Use which user to connect to Hippo instance. If user and password are provided, we will add related header in every RPC call. password (str): Required when user is provided. The password corresponding to the user. Example: .. code-block:: python from langchain_community.vectorstores import Hippo from langchain_community.embeddings import OpenAIEmbeddings embedding = OpenAIEmbeddings() # Connect to a hippo instance on localhost vector_store = Hippo.from_documents( docs, embedding=embeddings, table_name="langchain_test", connection_args=HIPPO_CONNECTION ) Raises: ValueError: If the hippo-api python package is not installed. testdefaultNFr strintzOptional[Dict[str, Any]]zOptional[dict]zOptional[bool])embedding_function table_name database_namenumber_of_shardsnumber_of_replicasconnection_args index_paramsdrop_oldc Cs>||_||_||_||_||_||_d|_d|_d|_g|_ |durHt }| ||_ d|_ z,|j |j|jr|r|j |j|jWn>ty} z&td|jd| WYd} ~ n d} ~ 00z*|j |j|jr|j |j|j|_ Wn@ty0} z&td|jd| WYd} ~ n d} ~ 00|dS)NpktextZvectorz+An error occurred while deleting the table z: z*An error occurred while getting the table )rrembedding_funcrrr _primary_field _text_field _vector_fieldfieldsDEFAULT_HIPPO_CONNECTION_create_connection_aliashccolZcheck_table_existsZ delete_table Exceptionloggingerror get_table_get_env) selfrrrrrrr r!er4t/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/vectorstores/hippo.py__init__PsD  zHippo.__init__dictr)rreturnc s(zddlm}Wn.ty>}ztd|WYd}~n d}~00|dd}|dd|dd}|d d}|durˆdurd |vr|d }d fd d |D}qt|d t}ntdz$t d|d||g||dWSt y"}zt d|WYd}~n d}~00dS)z*Create the connection to the Hippo server.rr zQUnable to import transwarp_hipp_api, please install with `pip install hippo-api`.NrrrZshivar,csg|]}|dqS):r4).0hrr4r5 z2Hippo._create_connection_alias..r:z/Missing standard address type for reuse attemptzcreate HippoClient[])rpwdzFailed to create new connection) transwarp_hippo_api.hippo_clientr ImportErrorgetsplitjoinr ValueErrorloggerinfor-r/) r2rrr3rrrhostsZ given_addressr4r=r5r*s0      zHippo._create_connection_aliaszOptional[list]zOptional[List[dict]]None) embeddings metadatasr8cCs<td|dur(td|||||dS)Nzinit ...zcreate collection)rHrI_create_collection_extract_fields _create_index)r2rLrMr4r4r5r1s    zHippo._get_envlistc CsDddlm}ddlm}t|d}td|g}|||jd|j |||j d|j |||j d|j d|id|r|d D]L\}}t|trt|} |||d|j d| idq|||d|j qtd ||jj|jd||j|j|jd |j|j|j|_td |jd |jd dS)Nr) HippoField) HippoTypez[_create_collection] dim: TF dimension)Z type_paramsz[_create_collection] fields: )nameZauto_idr(rrrz$[_create_collection] : create table z in z successfully)rBrRtranswarp_hippo_api.hippo_typerSlenrHdebugappendr%STRINGr&r'Z FLOAT_VECTORitems isinstancerQr+Z create_tablerrrrr0r,rI) r2rLrMrRrSdimr(keyvalueZ value_dimr4r4r5rNsZ      zHippo._create_collection)r8cCs^ddlm}t|j|rZ|jj}td||D]}|j|j q4td|jdS)z,Grab the existing fields from the Collectionr HippoTablez[_extract_fields] schema:z04 [_extract_fields] fields:N) rBrar\r,schemarHrXr(rYrU)r2rarbxr4r4r5rOs  zHippo._extract_fieldscCsddlm}t|j|r|j|j|j|ji}|dd}|durLdS|j|j|j|jdD]*}t d||d|j krh|SqhdS)z0Return the vector index information if it existsrr`embedding_indexesNz[_get_index] embedding_indexes column) rBrar\r,r+Zget_table_inforrrDrHrXr')r2raZ table_infordrcr4r4r5 _get_indexs&    zHippo._get_indexc Csddlm}ddlm}m}t|j|r|dur|dur|jdurd|j |j dd|_|jj |j |jd|jd |jd |jd d t |j|jdt d nZ|j |j|j|j|jd}|j|j|j |j d}||jd |jd <|jd dkrd||jd |jd <|j |j |jd|jd |jd t |j|jdn|jd dks|jd dkr||jd |jd <|jj |j |jd|jd |jd |jd d|jdddt |j|jdn |jd dkr||jd |jd <|jj |j |jd|jd |jd |jd d|jdd|jdd|jddt |j|jdn|jd dkr ||jd |jd <|jj |j |jd|jd |jd |jd|jd|jddt |j|jdntddS) z Create a index on the collectionrr`) IndexType MetricTypeNZlangchain_auto_create ) index_name metric_type index_typenlistrjrlrkrm)rmzcreate index successfully)IVF_FLATFLATIVF_SQIVF_PQHNSW)ipIPl2L2rornrpnprobe)rmrwrqnbitsm)rmrwrxrzrrMef_construction ef_search)r{r|r}zeIndex name does not match, please enter the correct index name. (FLAT, IVF_FLAT, IVF_PQ,IVF_SQ, HNSW))rBrarVrgrhr\r,rfr rvrnZ create_indexr'rHrXZactivate_indexrIrorprqrrrtrDrG)r2rargrhZ index_dictZ metric_dictr4r4r5rPs                     zHippo._create_indexz Iterable[str]z Optional[int]rz List[str])textsrMtimeout batch_sizekwargsr8c s"ddlm}|r"tdd|Dr0tdgSt|}td|zj|}Wn$ty|fdd|D}Yn0t |dkrtdgStd t |t j |sĈ ||j |j|itd |td j|d ur>|D]8}|D](\} } | jvr| g| qqtj j} t | } d jvrxjd td| td| |D]t|| fddjD} z j | }td|Wn8ty}ztd| |WYd }~n d }~00qdgS)a Add text to the collection. Args: texts: An iterable that contains the text to be added. metadatas: An optional list of dictionaries, each dictionary contains the metadata associated with a text. timeout: Optional timeout, in seconds. batch_size: The number of texts inserted in each batch, defaults to 1000. **kwargs: Other optional parameters. Returns: A list of strings, containing the unique identifiers of the inserted texts. Note: If the collection has not yet been created, this method will create a new collection. rr`css|]}|dkVqdS)Nr4)r;tr4r4r5 r?z"Hippo.add_texts..zNothing to insert, skipping.z[add_texts] texts: csg|]}j|qSr4)r$ embed_queryr;rc)r2r4r5r>r?z#Hippo.add_texts..zdd_texts] len_embeddings:z[add_texts] metadatas:z[add_texts] fields:Nr"z[add_texts] total_count:csg|]}|qSr4r4r)endi insert_dictr4r5r>r?z05 [add_texts] insert z0Failed to insert batch starting at entity: %s/%sr)rBraallrHrXrQr$Zembed_documentsNotImplementedErrorrWr\r,r1r&r'r(r[ setdefaultrYremoverangeminZ insert_rowsrIr-r/)r2rrMrrrrarLdr^r_vectorsZ total_countZ insert_listresr3r4)rrrr2r5 add_textssX             zHippo.add_textsz Optional[str]zList[Document])querykparamexprrrr8cKsB|jdurtdgS|jf|||||d|}dd|DS)a Perform a similarity search on the query string. Args: query (str): The text to search for. k (int, optional): The number of results to return. Default is 4. param (dict, optional): Specifies the search parameters for the index. Defaults to None. expr (str, optional): Filtering expression. Defaults to None. timeout (int, optional): Time to wait before a timeout error. Defaults to None. kwargs: Keyword arguments for Collection.search(). Returns: List[Document]: The document results of the search. N!No existing collection to search.)rrrrrcSsg|] \}}|qSr4r4)r;doc_r4r4r5r>r?z+Hippo.similarity_search..)r,rHrXsimilarity_search_with_score)r2rrrrrrrr4r4r5similarity_searchs   zHippo.similarity_searchzList[Tuple[Document, float]]c KsD|jdurtdgS|j|}|jf|||||d|}|S)a Performs a search on the query string and returns results with scores. Args: query (str): The text being searched. k (int, optional): The number of results to return. Default is 4. param (dict): Specifies the search parameters for the index. Default is None. expr (str, optional): Filtering expression. Default is None. timeout (int, optional): The waiting time before a timeout error. Default is None. kwargs: Keyword arguments for Collection.search(). Returns: List[float], List[Tuple[Document, any, any]]: Nr) embeddingrrrr)r,rHrXr$r&similarity_search_with_score_by_vector) r2rrrrrrrretr4r4r5rs    z"Hippo.similarity_search_with_scorez List[float])rrrrrrr8c sH|jdurtdgS|jdd}||jtd|jtd|gtd|td|td||jj|j|g|||dtd |jd }g} d } tfd d |DD]j} ddt|| D} t | |j| d} tdd |d || }| d7} | | |fq| S)a Performs a search on the query string and returns results with scores. Args: embedding (List[float]): The embedding vector being searched. k (int, optional): The number of results to return. Default is 4. param (dict): Specifies the search parameters for the index. Default is None. expr (str, optional): Filtering expression. Default is None. timeout (int, optional): The waiting time before a timeout error. Default is None. kwargs: Keyword arguments for Collection.search(). Returns: List[Tuple[Document, float]]: Resulting documents and scores. Nrz search_field:zvectors:zoutput_fields:ztopk:zdsl:)Z search_fieldr output_fieldsZtopkZdslz-[similarity_search_with_score_by_vector] res:z%scoresrcsg|]}d|qS)rr4)r;fieldrr4r5r>fr?z@Hippo.similarity_search_with_score_by_vector..cSsi|]\}}||qSr4r4)r;rr_r4r4r5 gr?z@Hippo.similarity_search_with_score_by_vector..)Z page_contentmetadataz;[similarity_search_with_score_by_vector] res[0][score_col]:r) r,rHrXr(rr'rr&zipr poprY)r2rrrrrrrZ score_colrcountr[metarZscorer4rr5r.sB     z,Hippo.similarity_search_with_score_by_vectorzDict[str, Any]zOptional[Dict[Any, Any]]boolz'Hippo') rrrMrrrr search_paramsr!rr8c Ksd|dur i}td|f|||||| d| } td|td|| j||d| S)a Creates an instance of the VST class from the given texts. Args: texts (List[str]): List of texts to be added. embedding (Embeddings): Embedding model for the texts. metadatas (List[dict], optional): List of metadata dictionaries for each text.Defaults to None. table_name (str): Name of the table. Defaults to "test". database_name (str): Name of the database. Defaults to "default". connection_args (dict[str, Any]): Connection parameters. Defaults to DEFAULT_HIPPO_CONNECTION. index_params (dict): Indexing parameters. Defaults to None. search_params (dict): Search parameters. Defaults to an empty dictionary. drop_old (bool): Whether to drop the old collection. Defaults to False. kwargs: Other arguments. Returns: Hippo: An instance of the VST class. Nz'00 [from_texts] init the class of Hippo)rrrrr r!z[from_texts] texts:z[from_texts] metadatas:)rrM)rHrIrXr) clsrrrMrrrr rr!rZ vector_dbr4r4r5 from_textsss"#  zHippo.from_texts)rrrrNNF)NN)N)NNr~)rNNN)rNNN)rNNN)__name__ __module__ __qualname____doc__r6r*r1rNrOrfrPrrrr classmethodr)rr4r4r4r5rsZ:9" Aw[%*Er) __future__rr.typingrrrrrrr Zlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.vectorstoresr rBrr) getLoggerrrHrr4r4r4r5s $