a `gg@sddlmZddlZddlZddlmZmZmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlZddlmZddlmZmZmZddlmZmZmZmZmZddlm Z e!e"Z#dd d d d d dddZ$GdddeeZ%dS)) annotationsN) AnyDictIterableListLiteralMappingOptionalSequenceSetTupleUnioncast) Embeddings)from_envget_pydantic_field_namessecret_from_env) BaseModel ConfigDictField SecretStrmodel_validator)SelfintzList[Union[List[int], str]]List[List[float]]z List[int]boolzList[Optional[List[float]]]) num_textstokensbatched_embeddingsindices skip_emptyreturnc s&ddt|D}ddt|Dtt|D]J|rJt|dkrJq0||||t|q0g}t|D]|}t|dkr|dqqt|dkr||dqqtfddt|D}tdd|Dd |fd d|Dq|S) NcSsg|]}gqSr".0_r"r"n/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_openai/embeddings/base.py 'z7_process_batched_chunked_embeddings..cSsg|]}gqSr"r"r#r"r"r&r',r(rcs,g|]$}tddt|DqS)css|]\}}||VqdSNr")r$valweightr"r"r& KszA_process_batched_chunked_embeddings...)sumzip)r$ embedding)inum_tokens_in_batch total_weightr"r&r'Js  css|]}|dVqdS)Nr"r$r+r"r"r&r-Ur(z6_process_batched_chunked_embeddings..g?csg|] }|qSr"r"r5) magnituder"r&r'Vr()rangelenappendr.r/) rrrrr results embeddings_resultZaverager")r1r6r2r3r&#_process_batched_chunked_embeddingss.       r=c@seZdZUdZedddZded<edddZded<dZd ed <dZ d ed <eZ d ed<ee dddddZ d ed<ede ddddZ d ed<ee ddddZd ed<ee ddddZd ed<dZded<ed ed!dddZd"ed#<ed$e d%d&gdddZd ed'<dZd(ed)<dZd*ed+<d,Zded-<d.Zded/<edd0d1Zd2ed3<dZded4<dZd5ed6<dZd ed7<d8Zd5ed9<eedZd:ed;<d8Zd5ed<<dZ d=ed><dZ!d?ed@<dAZ"dedB<dCZ#dedD<dZ$dEedF<dZ%dEedG<dZ&d5edH<e'dIddJdKZ(e)dLdMe*d:ddNdOdPZ+e)dQdMdRdSdTdUZ,e-d:dSdVdWZ.dXddYdZd[d\Z/dd]dXd d d^d_d`daZ0dd]dXd d d^d_dbdcZ1dodXddd^dZdedfZ2dpdXddd^dZdgdhZ3d didjdkdlZ4d didjdmdnZ5dS)qOpenAIEmbeddingsu OpenAI embedding model integration. Setup: Install ``langchain_openai`` and set environment variable ``OPENAI_API_KEY``. .. code-block:: bash pip install -U langchain_openai export OPENAI_API_KEY="your-api-key" Key init args — embedding params: model: str Name of OpenAI model to use. dimensions: Optional[int] = None The number of dimensions the resulting output embeddings should have. Only supported in `text-embedding-3` and later models. Key init args — client params: api_key: Optional[SecretStr] = None OpenAI API key. organization: Optional[str] = None OpenAI organization ID. If not passed in will be read from env var OPENAI_ORG_ID. max_retries: int = 2 Maximum number of retries to make when generating. request_timeout: Optional[Union[float, Tuple[float, float], Any]] = None Timeout for requests to OpenAI completion API See full list of supported init args and their descriptions in the params section. Instantiate: .. code-block:: python from langchain_openai import OpenAIEmbeddings embed = OpenAIEmbeddings( model="text-embedding-3-large" # With the `text-embedding-3` class # of models, you can specify the size # of the embeddings you want returned. # dimensions=1024 ) Embed single text: .. code-block:: python input_text = "The meaning of life is 42" vector = embeddings.embed_query("hello") print(vector[:3]) .. code-block:: python [-0.024603435769677162, -0.007543657906353474, 0.0039630369283258915] Embed multiple texts: .. code-block:: python vectors = embeddings.embed_documents(["hello", "goodbye"]) # Showing only the first 3 coordinates print(len(vectors)) print(vectors[0][:3]) .. code-block:: python 2 [-0.024603435769677162, -0.007543657906353474, 0.0039630369283258915] Async: .. code-block:: python await embed.aembed_query(input_text) print(vector[:3]) # multiple: # await embed.aembed_documents(input_texts) .. code-block:: python [-0.009100092574954033, 0.005071679595857859, -0.0029193938244134188] NT)defaultexcluderclient async_clientztext-embedding-ada-002strmodelz Optional[int] dimensionsz Optional[str] deploymentZOPENAI_API_VERSION)r? api_version)default_factoryaliasopenai_api_versionbase_urlZOPENAI_API_BASE)rIrHopenai_api_baseZOPENAI_API_TYPE)rHopenai_api_typeZ OPENAI_PROXY openai_proxyirembedding_ctx_lengthapi_keyZOPENAI_API_KEYzOptional[SecretStr]openai_api_key organizationZ OPENAI_ORG_IDZOPENAI_ORGANIZATIONopenai_organizationz%Union[Literal['all'], Set[str], None]allowed_specialz4Union[Literal['all'], Set[str], Sequence[str], None]disallowed_speciali chunk_sizer4 max_retriestimeout)r?rIz0Optional[Union[float, Tuple[float, float], Any]]request_timeoutheadersrtiktoken_enabledtiktoken_model_nameFshow_progress_barzDict[str, Any] model_kwargsr zUnion[Mapping[str, str], None]default_headersz!Union[Mapping[str, object], None] default_queryretry_min_secondsretry_max_secondszUnion[Any, None] http_clienthttp_async_clientcheck_embedding_ctx_lengthZforbidr")extraZpopulate_by_nameZprotected_namespacesbefore)mode)valuesr!c Cst|}|di}t|D]P}||vr8td|d||vrtd|d|d|d||||<q||}|rtd|d ||d<|S) z>Build extra kwargs from additional params that were passed in.r^zFound z supplied twice.z WARNING! z/ is not default parameter. zJ was transferred to model_kwargs. Please confirm that z is what you intended.z Parameters za should be specified explicitly. Instead they were passed in as part of `model_kwargs` parameter.) rgetlist ValueErrorwarningswarnpop intersectionkeys)clsrkZall_required_field_namesrh field_nameZinvalid_model_kwargsr"r"r& build_extras*   zOpenAIEmbeddings.build_extraafterrr!c Cs|jdvrtd|jr"|jnd|j|j|j|j|j|j d}|j r|j sT|j r|j }|j }|j }td|d|d||j s|j r|j sz ddl}Wn.ty}ztd |WYd}~n d}~00|j|j d |_ d |j i}tjfi||j|_ |js|j rh|j shz ddl}Wn0tyV}ztd |WYd}~n d}~00|j|j d |_ d |j i}tjfi||j|_|S) z?Validate that api key and python package exists in environment.)ZazureZazure_adZazureadzEIf you are using Azure, please use the `AzureOpenAIEmbeddings` class.N)rPrRrKrXrWr_r`zwCannot specify 'openai_proxy' if one of 'http_client'/'http_async_client' is already specified. Received: openai_proxy=z http_client=z http_async_client=rzRCould not import httpx python package. Please install it with `pip install httpx`.)proxyre)rMrnrQZget_secret_valuerSrLrYrWr_r`rNrerfrAhttpx ImportErrorZClientopenaiZOpenAIr;rBZ AsyncClientZ AsyncOpenAI) selfZ client_paramsrNrerfrzeZ sync_specificZasync_specificr"r"r&validate_environment*sp        z%OpenAIEmbeddings.validate_environmentcCs(d|ji|j}|jdur$|j|d<|S)NrDrE)rDr^rE)r}paramsr"r"r&_invocation_paramsds  z#OpenAIEmbeddings._invocation_paramsz List[str]zsz.OpenAIEmbeddings._tokenize..)rTrUZ001  )tqdm)r\rDr[Z transformersrr{rnZfrom_pretrained enumerateencoder7r8rOdecoder9tiktokenZencoding_for_modelKeyErrorZ get_encodingrTrUitemsendswithreplaceZencode_ordinaryr]Z tqdm.autor)r}rrVrrZ model_namerZ tokenizerr1textZ tokenizedjZ token_chunkZ chunk_textencodingZencoder_kwargstokenr_iterr"r"r& _tokenizeks`          zOpenAIEmbeddings._tokenize)rVr)renginerVr!c s|pj}||\}}}g}|D]R} jjfd|| | |ij} t| ts^| } |dd| dDq$t t ||||j } dddfdd fd d | DS) al Generate length-safe embeddings for a list of texts. This method handles tokenization and embedding generation, respecting the set embedding context length and chunk size. It supports both tiktoken and HuggingFace tokenizer based on the tiktoken_enabled flag. Args: texts (List[str]): A list of texts to embed. engine (str): The engine or model to use for embeddings. chunk_size (Optional[int]): The size of chunks for processing embeddings. Returns: List[List[float]]: A list of embeddings for each input text. inputcss|]}|dVqdSr0Nr"r$rr"r"r&r-r(z.dataN List[float]rxcsFdurBjjfddij}t|ts2|}|dddSNrrrr0)rAcreater isinstancedict model_dumpZaverage_embedded_cached_empty_embeddingr}r"r&empty_embeddings  zBOpenAIEmbeddings._get_len_safe_embeddings..empty_embeddingcsg|]}|dur|nqSr*r"r$r~rr"r&r'r(z=OpenAIEmbeddings._get_len_safe_embeddings..) rVrrArrrrrextendr=r8r r}rrrV _chunk_sizerrrrr1responser;r"rrr}r&_get_len_safe_embeddingss$    z)OpenAIEmbeddings._get_len_safe_embeddingsc s|pj}||\}}}g}|p(j}tdt||D]X} jjfd|| | |ijIdH} t| tsz| } | dd| dDq:t t||||j } dddfd d fd d | DIdHS) a Asynchronously generate length-safe embeddings for a list of texts. This method handles tokenization and asynchronous embedding generation, respecting the set embedding context length and chunk size. It supports both `tiktoken` and HuggingFace `tokenizer` based on the tiktoken_enabled flag. Args: texts (List[str]): A list of texts to embed. engine (str): The engine or model to use for embeddings. chunk_size (Optional[int]): The size of chunks for processing embeddings. Returns: List[List[float]]: A list of embeddings for each input text. rrNcss|]}|dVqdSrr"rr"r"r&r-r(z=OpenAIEmbeddings._aget_len_safe_embeddings..rrrxcsLdurHjjfddijIdH}t|ts8|}|dddSr)rBrrrrrrrr"r&r#s   zCOpenAIEmbeddings._aget_len_safe_embeddings..empty_embeddingcs$g|]}|dur|n IdHqSr*r"rrr"r&r'.r(z>OpenAIEmbeddings._aget_len_safe_embeddings..) rVrr7r8rBrrrrrrr=r rr"rr&_aget_len_safe_embeddingss&      z*OpenAIEmbeddings._aget_len_safe_embeddingsz int | NonecCs|p|j}|js~g}tdt||jD]R}|jjfd||||i|j}t|ts`|}| dd|dDq&|St t |j }|j ||dS)aMCall out to OpenAI's embedding endpoint for embedding search docs. Args: texts: The list of texts to embed. chunk_size: The chunk size of embeddings. If None, will use the chunk size specified by the class. Returns: List of embeddings, one for each text. rrcss|]}|dVqdSrr"rr"r"r&r-Fr(z3OpenAIEmbeddings.embed_documents..rr)rVrgr7r8rArrrrrrrCrFrr}rrVZ chunk_size_r;r1rrr"r"r&embed_documents0s    z OpenAIEmbeddings.embed_documentscs|p|j}|jsg}tdt||D]X}|jjfd||||i|jIdH}t|tsd|}| dd|dDq$|St t |j }|j ||dIdHS)aSCall out to OpenAI's embedding endpoint async for embedding search docs. Args: texts: The list of texts to embed. chunk_size: The chunk size of embeddings. If None, will use the chunk size specified by the class. Returns: List of embeddings, one for each text. rrNcss|]}|dVqdSrr"rr"r"r&r-dr(z4OpenAIEmbeddings.aembed_documents..rr)rVrgr7r8rBrrrrrrrCrFrrr"r"r&aembed_documentsNs     z!OpenAIEmbeddings.aembed_documentsr)rr!cCs||gdS)zCall out to OpenAI's embedding endpoint for embedding query text. Args: text: The text to embed. Returns: Embedding for the text. r)r)r}rr"r"r& embed_queryls zOpenAIEmbeddings.embed_querycs||gIdH}|dS)zCall out to OpenAI's embedding endpoint async for embedding query text. Args: text: The text to embed. Returns: Embedding for the text. Nr)r)r}rr;r"r"r& aembed_queryws zOpenAIEmbeddings.aembed_query)N)N)6__name__ __module__ __qualname____doc__rrA__annotations__rBrDrErFrrJrLrMrNrOrrQrSrTrUrVrWrYrZr[r\r]rr^r r_r`rbrdrerfrgrZ model_configr classmethodrvrpropertyrrrrrrrrr"r"r"r&r>[s Q                        9c23 r>)& __future__rloggingrotypingrrrrrrr r r r r rr|rZlangchain_core.embeddingsrZlangchain_core.utilsrrrZpydanticrrrrrZtyping_extensionsr getLoggerrloggerr=r>r"r"r"r&s 8   =