a bgc@sdZddlmZddlZddlZddlZddlZddlmZm Z m Z m Z m Z m Z mZmZddlmZddlmZddlmZddlmZeeZGd d d eZd ZGd d d ZdS)z,Module providing Infinispan as a VectorStore) annotationsN)AnyIterableListOptionalTupleTypeUnioncast)Response)Document) Embeddings) VectorStorec @seZdZdZdQddddddZd d d d d Zd dd ddZd dddddZdddddZddddZ dRdddddZ ddd d!Z ddd"d#Z d$dd%d&Z ddd'd(Zddd)d*ZdSd+d,d-dd.d/d0d1ZdTdddd3d4d5d6ZdUdddd7d4d8d9ZdVd:ddd3d;dd?d@Zd d7dAdBdCZd ddDdEdFdGZdDddHdIZedXdKd.dLd,ddMdMdddN dOdPZdS)Y InfinispanVSa`Infinispan` VectorStore interface. This class exposes the method to present Infinispan as a VectorStore. It relies on the Infinispan class (below) which takes care of the REST interface with the server. Example: ... code-block:: python from langchain_community.vectorstores import InfinispanVS from mymodels import RGBEmbeddings ... vectorDb = InfinispanVS.from_documents(docs, embedding=RGBEmbeddings(), output_fields=["texture", "color"], lambda_key=lambda text,meta: str(meta["_key"]), lambda_content=lambda item: item["color"]) or an empty InfinispanVS instance can be created if preliminary setup is required before populating the store ... code-block:: python from langchain_community.vectorstores import InfinispanVS from mymodels import RGBEmbeddings ... ispnVS = InfinispanVS() # configure Infinispan here # i.e. create cache and schema # then populate the store vectorDb = InfinispanVS.from_documents(docs, embedding=RGBEmbeddings(), output_fields: ["texture", "color"], lambda_key: lambda text,meta: str(meta["_key"]), lambda_content: lambda item: item["color"]) NzOptional[Embeddings]zOptional[List[str]]r) embeddingidskwargsc stfi|_|_tjdd_tjdd_|_jdd_jdkrpjdd_n t dt jd d_ j dkrjd d_ n t d t jd fd d_ jdfdd_jd_|_dS)at Parameters ---------- cache_name: str Embeddings cache name. Default "vector" entity_name: str Protobuf entity name for the embeddings. Default "vector" text_field: str Protobuf field name for text. Default "text" vector_field: str Protobuf field name for vector. Default "vector" lambda_content: lambda Lambda returning the content part of an item. Default returns text_field lambda_metadata: lambda Lambda returning the metadata part of an item. Default returns items fields excepts text_field, vector_field, _type output_fields: List[str] List of fields to be returned from item, if None return all fields. Default None kwargs: Any Rest of arguments passed to Infinispan. See docs cache_namevector entity_nameZ textfieldZ text_fieldtextz9`textfield` is deprecated. Please use `text_field` param.Z vectorfieldZ vector_fieldz=`vectorfield` is deprecated. Please use `vector_field` param.Zlambda_contentcs |SN)_default_contentitemself{/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/vectorstores/infinispanvs.pyiz'InfinispanVS.__init__..Zlambda_metadatacs |Sr)_default_metadatarrrrr lr!Z output_fieldsN) Infinispanispn_configurationstrget _cache_name _entity_name _embedding _textfieldwarningswarnDeprecationWarning _vectorfield _to_content _to_metadata_output_fields_ids)rrrrrrr__init__8s6    zInfinispanVS.__init__dict)rreturncCs4t|}||jd||jd|dd|S)N_type)r5popr/r+)rrmetarrrr"qs  zInfinispanVS._default_metadatazdict[str, Any]cCs ||jSr)r'r+)rrrrrrxszInfinispanVS._default_contentintr&)templ dimensionr6cCsd}||j||jf}d}|D]\}}t|trR|d|dt|d7}nt|trz|d|dt|d7}nt|tr|d|dt|d7}n\t|tr|d|dt|d7}n4t|tr|d |dt|d7}n t d ||d 7}q"|d 7}|S) NzV /** * @Indexed */ message %s { /** * @Vector(dimension=%d) */ repeated float %s = 1; zoptional string z = z; zoptional int64 zoptional double zoptional bytes zoptional bool zEUnable to build proto schema for metadata. Unhandled type for field: z} ) r)r/items isinstancer&r:floatbytesbool Exception)rr;r<Zmetadata_proto_tplZmetadata_protoidxfvrrrschema_builder{s4       zInfinispanVS.schema_builderr )protor6cCs|j|jd|S)zDeploy the schema for the vector db Args: proto(str): protobuf schema Returns: An http Response containing the result of the operation .proto)r$ schema_postr))rrIrrr schema_createszInfinispanVS.schema_create)r6cCs|j|jdS)zDelete the schema for the vector db Returns: An http Response containing the result of the operation rJ)r$ schema_deleter)rrrrrMszInfinispanVS.schema_deleter)configr6cCs&|dkrd|jd}|j|j|S)zCreate the cache for the vector db Args: config(str): configuration of the cache. Returns: An http Response containing the result of the operation raZ { "distributed-cache": { "owners": "2", "mode": "SYNC", "statistics": true, "encoding": { "media-type": "application/x-protostream" }, "indexing": { "enabled": true, "storage": "filesystem", "startup-mode": "AUTO", "indexing-mode": "AUTO", "indexed-entities": [ "z" ] } } } )r)r$ cache_postr()rrNrrr cache_createszInfinispanVS.cache_createcCs|j|jS)zDelete the cache for the vector db Returns: An http Response containing the result of the operation )r$ cache_deleter(rrrrrQszInfinispanVS.cache_deletecCs|j|jS)zClear the cache for the vector db Returns: An http Response containing the result of the operation )r$ cache_clearr(rrrrrRszInfinispanVS.cache_clearrCcCs|j|jS)zOChecks if the cache exists Returns: true if exists )r$ cache_existsr(rrrrrSszInfinispanVS.cache_existscCs|j|jS)zClear the index for the vector db Returns: An http Response containing the result of the operation )r$ index_clearr(rrrrcache_index_clearszInfinispanVS.cache_index_clearcCs|j|jS)z{Rebuild the for the vector db Returns: An http Response containing the result of the operation )r$ index_reindexr(rrrrcache_index_reindexsz InfinispanVS.cache_index_reindexz Iterable[str]zOptional[List[dict]]zOptional[List[float]] List[str])texts metadatas last_vectorrr6cKsg}t|}|r||j|}|r2|||sDdd|D}|jpVdd|D}tt|||} | D]J\} } } d|j|j| i} | | t | }|j | ||j|| ql|S)NcSsg|]}iqSrr.0_rrr r!z*InfinispanVS.add_texts..cSsg|]}ttqSr)r&uuiduuid4r\rrrr_r!r7)listr8r*Zembed_documentsappendr3zipr)r/updatejsondumpsr$putr()rrYrZr[rresultZtexts_lZembedsrZ data_inputmetadataembedkeydataZdata_strrrr add_textss$     zInfinispanVS.add_textszList[Document])querykrr6cKs|j||d}dd|DS)z"Return docs most similar to query.)rprqcSsg|] \}}|qSrrr]docr^rrrr_r!z2InfinispanVS.similarity_search..)similarity_search_with_score)rrprqr documentsrrrsimilarity_searchszInfinispanVS.similarity_searchzList[Tuple[Document, float]]cKs|j|}|j||d}|S)aPerform a search on a query string and return results with score. Args: query (str): The text being searched. k (int, optional): The amount of results to return. Defaults to 4. Returns: List[Tuple[Document, float]] )rrq)r* embed_query&similarity_search_with_score_by_vector)rrprqrrkrurrrrts z)InfinispanVS.similarity_search_with_scorez List[float])rrqrr6cKs|||}dd|DS)NcSsg|] \}}|qSrrrrrrrr_+r!z.)rx)rrrqrresrrrsimilarity_search_by_vector's z(InfinispanVS.similarity_search_by_vector)rrqr6cCs|jdur:d|jd|jdt|dt|}nld}|jddD]}|d|d }qL|d|jd}|d |jd|jdt|dt|}|j||j}t |j }| |S) a$Return docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. Returns: List of pair (Documents, score) most similar to the query vector. Nzselect v, score(v) from z v where v.z <-> ~zselect zv.,z, score(v) from ) r2r)r/rfrgr&r$ req_queryr(loadsr_query_result_to_docs)rrrq query_strZ query_projfieldZ query_resrirrrrx-sT    z3InfinispanVS.similarity_search_with_score_by_vector)rir6cstg}|dD]b}|dpi|jdur0d}nfdd|jD}t||||d}||dfq |S)Nhitshit*csi|]}||qSr)r')r]rlrrr ar!z6InfinispanVS._query_result_to_docs..)Z page_contentrjzscore())r2r r0r1rc)rrirurowentityrsrrrrXs    z"InfinispanVS._query_result_to_docsNone)rjr<r6cCsz|||}||}|j|jjjks.Jdt|jddusFJ| sv| }|j|jjjksnJd| dS)Nz)Unable to create schema. Already exists? errorz(Unable to create cache. Already exists? ) rHrL status_coder$CodesOKrfrrrSrPrU)rrjr<schemaoutputrrr configureis  zInfinispanVS.configurecCs||dSr)rMrQrrrr config_clearzszInfinispanVS.config_clearTzType[InfinispanVS]r zOptional[bool]) clsrYrrZr clear_old auto_configrr6c Ks|f||d|}|rnt|p gdkrn|r4|||t|d} ttt|}||dt| n"|rz|||t|d} |r|j||| d|S)aReturn VectorStore initialized from texts and embeddings. In addition to parameters described by the super method, this implementation provides other configuration params if different configuration from default is needed. Parameters ---------- ids : List[str] Additional list of keys associated to the embedding. If not provided UUIDs will be generated clear_old : bool Whether old data must be deleted. Default True auto_config: bool Whether to do a complete server setup (caches, protobuf definition...). Default True kwargs: Any Rest of arguments passed to InfinispanVS. See docs)rrrr>)r) lenrrwr rr5rrRrn) rrYrrZrrrrZ infinispanvsZvecrrr from_texts~szInfinispanVS.from_texts)NN)r)NN)ro)ro)ro)ro)NNTT)__name__ __module__ __qualname____doc__r4r"rrHrLrMrPrQrRrSrUrWrnrvrtrzrxrrr classmethodrrrrrrsF&9% # +r c @sVeZdZdZddddgdddddf dddd ddd d d d d d dZd9ddd ddddZd:ddd ddddZd;ddd ddddZdddddddZdddddddZ dddddd Z dddd!d"d#Z dddd$d%d&Z ddd'd(d)Z ddd'd*d+Zddd,d-d.Zdd d,d/d0Zdd d1d2d3Zddd,d4d5Zddd,d6d7Zd8S)Helper class for `Infinispan` REST interface. This class exposes the Infinispan operations needed to create and set up a vector db. You need a running Infinispan (15+) server without authentication. You can easily start one, see: https://github.com/rigazilla/infinispan-vector#run-infinispan httprz127.0.0.1:11222z/rest/v2/cachesz/rest/v2/schemasTr&rXrCr) ruserpasswordhosts cache_url schema_urluse_post_for_queryhttp2verifyrc Ksz ddl} Wnty&tdYn0| j|_| |_||_||_||_|d|_|jd|j|_ ||_ ||_ ||_ ||_ |jr|jr|jdkr| j|j|jd} n |j|jf} | j|j |j | | d|_n| j|j |j | d|_dS) aK Parameters ---------- schema: str Schema for HTTP request: "http" or "https". Default "http" user, password: str User and password if auth is required. Default None hosts: List[str] List of server addresses. Default ["127.0.0.1:11222"] cache_url: str URL endpoint for cache API. Default "/rest/v2/caches" schema_url: str URL endpoint for schema API. Default "/rest/v2/schemas" use_post_for_query: bool Whether POST method should be used for query. Default True http2: bool Whether HTTP/2 protocol should be used. `pip install "httpx[http2]"` is needed for HTTP/2. Default True verify: bool Whether TLS certificate must be verified. Default True rNzCould not import httpx python package. Please install it with `pip install httpx`or `pip install "httpx[http2]"` if you need HTTP/2.z://r)usernamer)rhttp1authr)rrr)httpx ImportErrorcodesrr%Z_schema_userZ _passwordZ_host _default_node _cache_url _schema_url_use_post_for_queryZ_http2Z DigestAuthZClient_h2c) rrrrrrrrrrrrrrrrr4sD#        zInfinispan.__init__Fr )rprlocalr6cCs"|jr||||S||||S)aRequest a query Args: query(str): query requested cache_name(str): name of the target cache local(boolean): whether the query is local to clustered Returns: An http Response containing the result set or errors )r _query_post _query_get)rrprrrrrr~s zInfinispan.req_query)rrrr6cCsN|j|jd|dt|}d|i}t|}|jj||dditd}|S)N/z?action=search&local=rp Content-Typeapplication/jsoncontentheaderstimeout)rrr&rfrgrpost REST_TIMEOUT)rrrrapi_urlrmZ data_jsonresponserrrrs* zInfinispan._query_postcCs<|j|jd|d|dt|}|jj|td}|S)Nrz?action=search&query=z&local=r)rrr&rr'r)rrrrrrrrrr's$ zInfinispan._query_get)rlrmrr6cCs8|j|jd|d|}|jj||dditd}|S)aPost an entry Args: key(str): key of the entry data(str): content of the entry in json format cache_name(str): target cache Returns: An http Response containing the result of the operation rrrrrrrrrrrlrmrrrrrrr7s zInfinispan.postcCs8|j|jd|d|}|jj||dditd}|S)aPut an entry Args: key(str): key of the entry data(str): content of the entry in json format cache_name(str): target cache Returns: An http Response containing the result of the operation rrrr)rrrrhrrrrrrhIs zInfinispan.put)rlrr6cCs6|j|jd|d|}|jj|dditd}|S)zGet an entry Args: key(str): key of the entry cache_name(str): target cache Returns: An http Response containing the entry or errors rrr)rr)rrrr'r)rrlrrrrrrr'[s  zInfinispan.get)namerIr6cCs*|j|jd|}|jj||td}|S)zDeploy a schema Args: name(str): name of the schema. Will be used as a key proto(str): protobuf schema Returns: An http Response containing the result of the operation r)rr)rrrrr)rrrIrrrrrrKiszInfinispan.schema_post)rrNr6cCs0|j|jd|}|jj||dditd}|S)zCreate a cache Args: name(str): name of the cache. config(str): configuration of the cache. Returns: An http Response containing the result of the operation rrrrr)rrrNrrrrrrOuszInfinispan.cache_post)rr6cCs(|j|jd|}|jj|td}|S)zDelete a schema Args: name(str): name of the schema. Returns: An http Response containing the result of the operation rr)rrrdeleterrrrrrrrrMszInfinispan.schema_deletecCs(|j|jd|}|jj|td}|S)zDelete a cache Args: name(str): name of the cache. Returns: An http Response containing the result of the operation rr)rrrrrrrrrrQszInfinispan.cache_delete)rr6cCs,|j|jd|d}|jj|td}|S)zClear a cache Args: cache_name(str): name of the cache. Returns: An http Response containing the result of the operation r ?action=clearrr)rrrrrrrrRszInfinispan.cache_clearcCs"|j|jd|d}||S)zCheck if a cache exists Args: cache_name(str): name of the cache. Returns: True if cache exists rr)rrresource_existsrrrrrrrSszInfinispan.cache_exists)rr6cCs|jj|td}|j|jjkS)zCheck if a resource exists Args: api_url(str): url of the resource. Returns: true if resource exists r)rheadrrrr)rrrrrrrszInfinispan.resource_existscCs(|j|jd|d}|jj|tdS)zClear an index on a cache Args: cache_name(str): name of the cache. Returns: An http Response containing the result of the operation rz/search/indexes?action=clearrrrrrrrTszInfinispan.index_clearcCs(|j|jd|d}|jj|tdS)zRebuild index on a cache Args: cache_name(str): name of the cache. Returns: An http Response containing the result of the operation rz/search/indexes?action=reindexrrrrrrrVszInfinispan.index_reindexN)F)F)F)rrrrr4r~rrrrhr'rKrOrMrQrRrSrrTrVrrrrr#s8 "L      r#)r __future__rrfloggingr`r,typingrrrrrrr r rr Zlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.vectorstoresr getLoggerrloggerrrr#rrrrs" (