a bg3@s*dZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZddlmZddlmZddlmZddlmZdd lmZdd lmZe d eee efd Ze d eee e ee fd ZGdddZGdddeeefZ Gddde eefZ!eddddGdddeZ"dS)zSagemaker InvokeEndpoint API.N)abstractmethod) AnyDictGenericIteratorListMappingOptionalTypeVarUnion) deprecated)CallbackManagerForLLMRun)LLM)pre_init) ConfigDict)enforce_stop_tokens INPUT_TYPE)bound OUTPUT_TYPEc@s<eZdZdZeddddZddddZedd d ZdS) LineIteratoraParse the byte stream input. The output of the model will be in the following format: b'{"outputs": [" a"]} ' b'{"outputs": [" challenging"]} ' b'{"outputs": [" problem"]} ' ... While usually each PayloadPart event from the event stream will contain a byte array with a full json, this is not guaranteed and some of the json objects may be split acrossPayloadPart events. For example: {'PayloadPart': {'Bytes': b'{"outputs": '}} {'PayloadPart': {'Bytes': b'[" problem"]} '}} This class accounts for this by concatenating bytes written via the 'write' function and then exposing a method which will return lines (ending with a ' ' character) within the buffer via the 'scan_lines' function. It maintains the position of the last read position to ensure that previous bytes are not exposed again. For more details see: https://aws.amazon.com/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/ N)streamreturncCst||_t|_d|_dS)Nr)iter byte_iteratorioBytesIObufferread_pos)selfrry/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/llms/sagemaker_endpoint.py__init__2s  zLineIterator.__init__rcCs|S)Nrrrrr __iter__7szLineIterator.__iter__cCs|j|j|j}|rJ|dtdkrJ|jt|7_|ddSzt|j}Wn*ty|j|j j kr|YqYn0d|vrq|jdt j |j |ddqdS)N Z PayloadPartrBytes)rseekrreadlineordlennextr StopIteration getbuffernbytesrSEEK_ENDwrite)rlinechunkrrr __next__:s   zLineIterator.__next__)__name__ __module__ __qualname____doc__rr!r$r4rrrr rsrc@s\eZdZUdZdZeeed<dZeeed<e e e e dddZ e e edd d Zd S) ContentHandlerBasezHandler class to transform input from LLM to a format that SageMaker endpoint expects. Similarly, the class handles transforming output from the SageMaker endpoint to a format that LLM class expects. z text/plain content_typeaccepts)prompt model_kwargsrcCsdS)zTransforms the input to a format that model can accept as the request Body. Should return bytes or seekable file like object in the format specified in the content_type request header. Nr)rr<r=rrr transform_inputmsz"ContentHandlerBase.transform_input)outputrcCsdS)z[Transforms the output from the model to string that the LLM class expects. Nr)rr?rrr transform_outputusz#ContentHandlerBase.transform_outputN)r5r6r7r8r:r str__annotations__r;rrrbytesr>rr@rrrr r9Ns r9c@seZdZdZdS)LLMContentHandlerzContent handler for LLM class.N)r5r6r7r8rrrr rD|srDz0.3.16z1.0z$langchain_aws.llms.SagemakerEndpoint)ZsinceZremovalZalternative_importc@seZdZUdZdZeed<dZeed<dZ eed<dZ e eed<e ed<d Z eed <dZe eed <dZe eed <ed dZeeedddZeeeefdddZeedddZdee eee eeedddZdS)SagemakerEndpointaSagemaker Inference Endpoint models. To use, you must supply the endpoint name from your deployed Sagemaker model & the region where it is deployed. To authenticate, the AWS client uses the following methods to automatically load credentials: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html If a specific credential profile should be used, you must pass the name of the profile from the ~/.aws/credentials file that is to be used. Make sure the credentials / roles used have the required policies to access the Sagemaker endpoint. See: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html Nclient endpoint_name region_namecredentials_profile_namecontent_handlerF streamingr=endpoint_kwargsZforbid)extra)valuesrc Cs|ddur|Sz|ddl}z@|ddur<|j|dd}n|}|jd|dd|d<Wn.ty}ztd |WYd}~n d}~00Wntytd Yn0|S) z.Dont do anything if client provided externallyrFNrrJ)Z profile_namezsagemaker-runtimerI)rIzCould not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid.zRCould not import boto3 python package. Please install it with `pip install boto3`.)getboto3SessionrF Exception ValueError ImportError)clsrOrQsessionerrr validate_environments0   z&SagemakerEndpoint.validate_environmentr"cCs |jpi}id|jid|iS)zGet the identifying parameters.rHr=)r=rH)r _model_kwargsrrr _identifying_params's  z%SagemakerEndpoint._identifying_paramscCsdS)zReturn type of llm.Zsagemaker_endpointrr#rrr _llm_type0szSagemakerEndpoint._llm_type)r<stop run_managerkwargsrc Ks~|jpi}i||}|jpi}|j||}|jj}|jj} |jr|rz||jjf|j ||jjd|} t | d} d} | D]@} t | } | dd}|durt||}| |7} ||q~| WSty}ztd|WYd}~n d}~00nz"|jjf|j ||| d|}Wn4tyP}ztd |WYd}~n d}~00|j|d}|durvt||}|SdS) afCall out to Sagemaker inference endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. Returns: The string generated by the model. Example: .. code-block:: python response = se("Tell me a joke.") ) EndpointNameBody ContentTyperarGoutputsrNz.Error raised by streaming inference endpoint: )r`rarbAcceptz$Error raised by inference endpoint: )r=rMrKr>r:r;rLrFZ$invoke_endpoint_with_response_streamrHrjsonloadsrPrZon_llm_new_tokenrSrTZinvoke_endpointr@)rr<r]r^r_rZZ_endpoint_kwargsbodyr:r;respiteratorZcurrent_completionr2Z resp_outputrXresponsetextrrr _call5sT        & $  zSagemakerEndpoint._call)NN)r5r6r7r8rFrrBrHrArIrJr rDrLboolr=rrMrZ model_configrrYpropertyrr[r\rr rlrrrr rEs6 A    $ rE)#r8rreabcrtypingrrrrrrr r r Zlangchain_core._api.deprecationr Zlangchain_core.callbacksr Z#langchain_core.language_models.llmsrZlangchain_core.utilsrZpydanticrZlangchain_community.llms.utilsrrArfloatrrr9rDrErrrr s* ,      :.