a `g{/@sddlmZddlZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZddlmZmZeeZedd d ZGd d d eeZGd d d eZ Gddde!e Z"eddGdddZ#ddddddZ$dS)) annotationsN)ABCabstractmethod) dataclass)Enum) AbstractSetAnyCallable CollectionIterableListLiteralOptionalSequenceTypeTypeVarUnion)BaseDocumentTransformerDocumentTS TextSplitter)boundc @seZdZdZddedddfddddd d d d d d ZedddddZd6ddddddZdddddZ dddddd Z d!ddd"d#d$Z e d%d%dd&d'd(Z e d)ded*fd+ddd,d-d%d.d/d0d1Zd2d%d2d3d4d5ZdS)7rz)Interface for splitting text into chunks.iFTintzCallable[[str], int]z$Union[bool, Literal['start', 'end']]boolNone) chunk_size chunk_overlaplength_functionkeep_separatoradd_start_indexstrip_whitespacereturncCsF||krtd|d|d||_||_||_||_||_||_dS)adCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_functionZ_keep_separator_add_start_index_strip_whitespace)selfrrrrr r!r*k/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_text_splitters/base.py__init__!szTextSplitter.__init__str List[str]textr"cCsdS)z$Split text into multiple components.Nr*)r)r0r*r*r+ split_textBszTextSplitter.split_textNzOptional[List[dict]]zList[Document])texts metadatasr"c Cs|pigt|}g}t|D]z\}}d}d}||D]^} t||} |jr|||j} || td| }|| d<t| }t | | d} | | q8q|S)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater1copydeepcopyr'r%findmaxrappend) r)r2r3Z _metadatas documentsir0indexZprevious_chunk_lenchunkr6offsetZnew_docr*r*r+create_documentsFs zTextSplitter.create_documentszIterable[Document])r>r"cCs:gg}}|D]}||j||jq|j||dS)zSplit documents.)r3)r=r5r6rC)r)r>r2r3docr*r*r+split_documentsZs   zTextSplitter.split_documents Optional[str])docs separatorr"cCs,||}|jr|}|dkr$dS|SdS)N)joinr(strip)r)rGrHr0r*r*r+ _join_docsbs  zTextSplitter._join_docsz Iterable[str])splitsrHr"c CsV||}g}g}d}|D]}||}||t|dkr@|nd|jkr||jkrptd|d|jt|dkr|||} | dur|| ||jks||t|dkr|nd|jkr|dkr|||dt|dkr|nd8}|dd}q||||t|dkr(|nd7}q|||} | durR|| |S)NrzCreated a chunk of size z%, which is longer than the specified )r&r7r$loggerwarningrLr=r%) r)rMrHZ separator_lenrGZ current_doctotald_lenrDr*r*r+ _merge_splitsksJ            zTextSplitter._merge_splitsr) tokenizerkwargsr"c sdz6ddlm}t|s tddddfdd }WntyPtd Yn0|fd |i|S) z>Text splitter that uses HuggingFace tokenizer to count length.r)PreTrainedTokenizerBasezATokenizer received was not an instance of PreTrainedTokenizerBaser-rr/cst|S)Nr7encoder0rUr*r+_huggingface_tokenizer_lengthszNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengthz`Could not import transformers python package. Please install it with `pip install transformers`.r)Z transformersrW isinstancer# ImportError)clsrUrVrWr\r*r[r+from_huggingface_tokenizers    z'TextSplitter.from_huggingface_tokenizergpt2allzType[TS]'Union[Literal['all'], AbstractSet[str]]&Union[Literal['all'], Collection[str]]r)r_ encoding_name model_nameallowed_specialdisallowed_specialrVr"c  sz ddl}Wnty&tdYn0|dur<||n ||dddfdd }t|tr||d }i||}|fd |i|S) z9Text splitter that uses tiktoken encoder to count length.rNzCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.r-rr/cstj|dSN)rgrhrXrZrgrhencr*r+_tiktoken_encodersz=TextSplitter.from_tiktoken_encoder.._tiktoken_encoder)rerfrgrhr)tiktokenr^encoding_for_model get_encoding issubclassTokenTextSplitter) r_rerfrgrhrVrmrl extra_kwargsr*rjr+from_tiktoken_encoders$        z"TextSplitter.from_tiktoken_encoderzSequence[Document])r>rVr"cKs|t|S)z2Transform sequence of documents by splitting them.)rElist)r)r>rVr*r*r+transform_documentssz TextSplitter.transform_documents)N)__name__ __module__ __qualname____doc__r7r,rr1rCrErLrT classmethodr`setrsrur*r*r*r+rs0! *+csLeZdZdZddedfddddd d d fd d ZdddddZZS)rqz/Splitting text to tokens using model tokenizer.raNrbr-rFrcrdrr)rerfrgrhrVr"c sntjfi|z ddl}Wnty8tdYn0|durN||}n ||}||_||_||_dS)zCreate a new TextSplitter.rNzCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.) superr,rmr^rnro _tokenizer_allowed_special_disallowed_special)r)rerfrgrhrVrmrk __class__r*r+r,s      zTokenTextSplitter.__init__r.r/cs8dddfdd }tjjjj|d}t||dS)aSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text (str): The input text to be split into smaller chunks. Returns: List[str]: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. r-z List[int])_textr"csjj|jjdSri)r}rYr~r)rr)r*r+_encode s z-TokenTextSplitter.split_text.._encode)rtokens_per_chunkdecoderY)r0rU) Tokenizerr%r$r}rsplit_text_on_tokens)r)r0rrUr*rr+r1szTokenTextSplitter.split_text)rvrwrxryr{r,r1 __classcell__r*r*rr+rqsrqc@sxeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdS)Languagez"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlZsolcsharpcobolcluaperlhaskellelixir powershellN)rvrwrxryCPPZGOZJAVAZKOTLINZJSrPHPPROTOPYTHONZRSTZRUBYZRUSTZSCALAZSWIFTMARKDOWNZLATEXHTMLZSOLZCSHARPCOBOLCZLUAZPERLZHASKELLZELIXIRZ POWERSHELLr*r*r*r+rs6rT)frozenc@s2eZdZUdZded<ded<ded<ded<d S) rzTokenizer data class.rrrzCallable[[List[int]], str]rzCallable[[str], List[int]]rYN)rvrwrxry__annotations__r*r*r*r+r;s rr-r.)r0rUr"cCsg}||}d}t||jt|}|||}|t|kr||||t|kr\q||j|j7}t||jt|}|||}q2|S)z6Split incoming text and return chunks using tokenizer.r)rYminrr7r=rr)r0rUrMZ input_idsZ start_idxZcur_idxZ chunk_idsr*r*r+rIs    r)% __future__rr9loggingabcrr dataclassesrenumrtypingrrr r r r r rrrrrZlangchain_core.documentsrr getLoggerrvrOrrrqr-rrrr*r*r*r+s   8  @?