a bg"@spddlZddlmZmZmZmZddlmZddlm Z ddl m Z ddl m Z dZdZd ZGd d d e ZdS) N)AnyListMappingOptional)CallbackManagerForLLMRun)LLM) ConfigDict)enforce_stop_tokenszgoogle/flan-t5-largetext2text-generation)r text-generation summarizationc@seZdZUdZdZeed<eZe ed<dZ e e ed<dZ e e ed<eddZede e e ee e e e e e e ee ee eeed d d Zeee efdddZee dddZde e ee e eee dddZdS)WeightOnlyQuantPipelineaWeight only quantized model. To use, you should have the `intel-extension-for-transformers` packabge and `transformers` package installed. intel-extension-for-transformers: https://github.com/intel/intel-extension-for-transformers Example using from_model_id: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline from intel_extension_for_transformers.transformers import ( WeightOnlyQuantConfig ) config = WeightOnlyQuantConfig hf = WeightOnlyQuantPipeline.from_model_id( model_id="google/flan-t5-large", task="text2text-generation" pipeline_kwargs={"max_new_tokens": 10}, quantization_config=config, ) Example passing pipeline in directly: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline from intel_extension_for_transformers.transformers import ( AutoModelForSeq2SeqLM ) from intel_extension_for_transformers.transformers import ( WeightOnlyQuantConfig ) from transformers import AutoTokenizer, pipeline model_id = "google/flan-t5-large" tokenizer = AutoTokenizer.from_pretrained(model_id) config = WeightOnlyQuantConfig model = AutoModelForSeq2SeqLM.from_pretrained( model_id, quantization_config=config, ) pipe = pipeline( "text-generation", model=model, tokenizer=tokenizer, max_new_tokens=10, ) hf = WeightOnlyQuantPipeline(pipeline=pipe) Npipelinemodel_id model_kwargspipeline_kwargsZallow)extraF) rtaskdevice device_maprr load_in_4bit load_in_8bitquantization_configkwargsreturnc  Ks$|dur"t|tr"|dkr"tdtjddur:tdz8ddlm} m} ddl m } dd l m }dd l m }Wntytd Yn0t|tr|dkr| std d t|}nt|tr|dkrd}|dur|durd}|pi}|j|fi|}zn|dkr,| j|f||| d|d|}n@|dvrV| j|f||| d|d|}ntd|dtdWn8ty}ztd|d|WYd}~n d}~00d|vrdd|D}|pi}|f|||||d|}|jtvr td|jdtd|f||||d| S)z5Construct the pipeline object from model_id and task.Nrz7`Device` and `device_map` cannot be set simultaneously!Ztorchz;Weight only quantization pipeline only support PyTorch now!r)AutoModelForCausalLMAutoModelForSeq2SeqLM)is_ipex_available) AutoTokenizer)rzCould not import transformers python package. Please install it with `pip install transformers` and `pip install intel-extension-for-transformers`.z)Don't find out Intel GPU on this machine!zxpu:cpur F)rrrZuse_llm_runtimer)r r Got invalid task , currently only  are supportedzCould not load the z# model due to missing dependencies.trust_remote_codecSsi|]\}}|dkr||qS)r$).0kvr%r%/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/llms/weight_only_quantization.py sz9WeightOnlyQuantPipeline.from_model_id..)rmodel tokenizerrr)rrrr) isinstanceint ValueError importlibutil find_specZ-intel_extension_for_transformers.transformersrrZ,intel_extension_for_transformers.utils.utilsrZ transformersrr ImportErrorstrZfrom_pretrained VALID_TASKSitemsr)clsrrrrrrrrrrrrrrZ hf_pipelineZ _model_kwargsr,r+eZ_pipeline_kwargsrr%r%r) from_model_idOs            z%WeightOnlyQuantPipeline.from_model_id)rcCs|j|j|jdS)zGet the identifying parameters.rrrr:selfr%r%r)_identifying_paramssz+WeightOnlyQuantPipeline._identifying_paramscCsdS)zReturn type of llm.Zweight_only_quantizationr%r;r%r%r) _llm_typesz!WeightOnlyQuantPipeline._llm_type)promptstop run_managerrrcKs||}|jjdkr0|ddt|d}nN|jjdkrJ|dd}n4|jjdkrd|dd}ntd|jjd td |rt||}|S) abCall the HuggingFace model and return the output. Args: prompt: The prompt to use for generation. stop: A list of strings to stop generation when encountered. Returns: The generated text. Example: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline llm = WeightOnlyQuantPipeline.from_model_id( model_id="google/flan-t5-large", task="text2text-generation", ) llm.invoke("This is a prompt.") r rZgenerated_textNr r Z summary_textr!r"r#)rrlenr/r5r )r<r?r@rArresponsetextr%r%r)_calls      zWeightOnlyQuantPipeline._call)rNNNFFN)NN)__name__ __module__ __qualname____doc__rr__annotations__DEFAULT_MODEL_IDrr4rrdictrrZ model_config classmethodr.boolrr9propertyrr=r>rrrEr%r%r%r)r sT 1  j r )r0typingrrrrZ langchain_core.callbacks.managerrZ#langchain_core.language_models.llmsrZpydanticrZlangchain_community.llms.utilsr rKZ DEFAULT_TASKr5r r%r%r%r)s