a bg%@sddlmZddlmZmZmZmZmZmZm Z m Z m Z ddl m Z ddlmZddlmZer|ddlmZmZddlmZGdd d eZd S) ) annotations) TYPE_CHECKINGAnyDict GeneratorIteratorListMappingOptionalUnion)CallbackManagerForLLMRun)LLM)GenerationChunk)RESTfulChatModelHandleRESTfulGenerateModelHandle)LlamaCppGenerateConfigcseZdZUdZdZded<ded<ded<ded <d,ddd d fd d ZeddddZeddddZ d-dddd ddddZ d.ddddddddZ d/dddd d dd!d"Z d0dd#d$d%d&d'Z edd(d)d*d+ZZS)1 Xinferencea `Xinference` large-scale model inference service. To use, you should have the xinference library installed: .. code-block:: bash pip install "xinference[all]" If you're simply using the services provided by Xinference, you can utilize the xinference_client package: .. code-block:: bash pip install xinference_client Check out: https://github.com/xorbitsai/inference To run, you need to start a Xinference supervisor on one server and Xinference workers on the other servers Example: To start a local instance of Xinference, run .. code-block:: bash $ xinference You can also deploy Xinference in a distributed cluster. Here are the steps: Starting the supervisor: .. code-block:: bash $ xinference-supervisor Starting the worker: .. code-block:: bash $ xinference-worker Then, launch a model using command line interface (CLI). Example: .. code-block:: bash $ xinference launch -n orca -s 3 -q q4_0 It will return a model UID. Then, you can use Xinference with LangChain. Example: .. code-block:: python from langchain_community.llms import Xinference llm = Xinference( server_url="http://0.0.0.0:9997", model_uid = {model_uid} # replace model_uid with the model UID return from launching the model ) llm.invoke( prompt="Q: where can we visit in the capital of France? A:", generate_config={"max_tokens": 1024, "stream": True}, ) Example: .. code-block:: python from langchain_community.llms import Xinference from langchain.prompts import PromptTemplate llm = Xinference( server_url="http://0.0.0.0:9997", model_uid={model_uid}, # replace model_uid with the model UID return from launching the model stream=True ) prompt = PromptTemplate( input=['country'], template="Q: where can we visit in the capital of {country}? A:" ) chain = prompt | llm chain.stream(input={'country': 'France'}) To view all the supported builtin models, run: .. code-block:: bash $ xinference list --all Nz Optional[Any]clientz Optional[str] server_url model_uidzDict[str, Any] model_kwargsrrrrc szddlm}WnRtybzddlm}Wn.ty\}ztd|WYd}~n d}~00Yn0|pji}tjfi|||d|jdurtd|jdurtd|||_ dS)Nr) RESTfulClientzCould not import RESTfulClient from xinference. Please install it with `pip install xinference` or `pip install xinference_client`.rzPlease provide server URLzPlease provide the model UID) xinference.clientr ImportErrorZxinference_clientsuper__init__r ValueErrorrr)selfrrrre __class__q/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/llms/xinference.pyr}s.    zXinference.__init__str)returncCsdS)zReturn type of llm.Z xinferencer"rr"r"r# _llm_typeszXinference._llm_typezMapping[str, Any]cCs"id|jid|jid|jiS)zGet the identifying parameters.rrrrr&r"r"r#_identifying_paramsszXinference._identifying_paramszOptional[List[str]]z"Optional[CallbackManagerForLLMRun])promptstop run_managerkwargsr%c Ks|jdurtd|j|j}|di}i|j|}|rF||d<|r||dr|d}|j||||dD] }||7}qj|S|j||d} | d d d SdS) aqCall the xinference model and return the output. Args: prompt: The prompt to use for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Returns: The generated string by the model. NClient is not initialized!generate_configr*stream)modelr)r+r.r)r.choicesrtext)rr get_modelrgetr_stream_generategenerate) rr)r*r+r,r1r.Zcombined_text_outputtoken completionr"r"r#_calls&    zXinference._callz=Union['RESTfulGenerateModelHandle', 'RESTfulChatModelHandle']z"Optional['LlamaCppGenerateConfig']zGenerator[str, None, None])r1r)r+r.r%c csz|j||d}|D]b}t|tr|dg}|r|d}t|tr|dd} |d} |rn|j| |j| d| VqdS) a^ Args: prompt: The prompt to use for generation. model: The model used for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Yields: A string token. r2r3rr4r0logprobs)r9verbose log_probsN)r8 isinstancedictr6on_llm_new_tokenr=) rr1r)r+r.Zstreaming_responsechunkr3choicer9r>r"r"r#r7s      zXinference._stream_generatezIterator[GenerationChunk]cksf|di}i|j|}|r&||d<|||D].}|r2||}|rZ|j|j|jd|Vq2dS)Nr.r*)r=)r6r_create_generate_stream$_stream_response_to_generation_chunkrAr4r=)rr)r*r+r,r.Z stream_resprBr"r"r#_streams  zXinference._streamzOptional[Dict[str, List[str]]]z Iterator[str])r)r.r%ccs8|jdurtd|j|j}|j||dEdHdS)Nr-r2)rrr5rr8)rr)r.r1r"r"r#rDs z"Xinference._create_generate_streamr)stream_responser%cCsd}t|trv|dg}|rj|d}t|tr`|dd}t|t|dd|dddd Std q~t|d Sntd dS) z0Convert a stream response to a generation chunk.r0r3rr4 finish_reasonNr<)rHr<)r4Zgeneration_infozchoice type error!)r4zstream_response type error!)r?r@r6r TypeError)rGr9r3rCr"r"r#rEs"        z/Xinference._stream_response_to_generation_chunk)NN)NN)NN)NN)N)__name__ __module__ __qualname____doc__r__annotations__rpropertyr'r(r;r7rFrD staticmethodrE __classcell__r"r"r r#rs2 \ # 0&rN) __future__rtypingrrrrrrr r r Zlangchain_core.callbacksr Z#langchain_core.language_models.llmsr Zlangchain_core.outputsrrrrZxinference.model.llm.corerrr"r"r"r#s ,