a `g@sLddlmZddlmZmZmZmZddlmZm Z m Z GdddeZ dS)) annotations)AnyListOptionalcast) TextSplitter Tokenizersplit_text_on_tokenscs~eZdZUdZdddddd d fd d Zdd d ddZdddddZdddddZdZde d<dddddZ Z S)%SentenceTransformersTokenTextSplitterz8Splitting text to tokens using sentence model tokenizer.2'sentence-transformers/all-mpnet-base-v2Nintstrz Optional[int]rNone) chunk_overlap model_nametokens_per_chunkkwargsreturnc srtjfi|d|izddlm}WntyDtdYn0||_||j|_|jj|_|j|ddS)zCreate a new TextSplitter.rr)SentenceTransformerzCould not import sentence_transformers python package. This is needed in order to for SentenceTransformersTokenTextSplitter. Please install it with `pip install sentence-transformers`.)rN) super__init__Zsentence_transformersr ImportErrorr_model tokenizer_initialize_chunk_configuration)selfrrrrr __class__|/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_text_splitters/sentence_transformers.pyr s    z.SentenceTransformersTokenTextSplitter.__init__)rrcCsZtt|jj|_|dur"|j|_n||_|j|jkrVtd|jd|jd|jddS)NzThe token limit of the models 'z' is: z. Argument tokens_per_chunk=z > maximum token limit.)rr rZmax_seq_lengthZmaximum_tokens_per_chunkr ValueErrorr)rrrrr r#s   zESentenceTransformersTokenTextSplitter._initialize_chunk_configurationz List[str]textrcs8dddfdd }tjjjj|d}t||dS)aSplits the input text into smaller components by splitting text on tokens. This method encodes the input text using a private `_encode` method, then strips the start and stop token IDs from the encoded result. It returns the processed segments as a list of strings. Args: text (str): The input text to be split. Returns: List[str]: A list of string components derived from the input text after encoding and processing. r List[int]r"cs|ddS)N)_encode)r#rrr %encode_strip_start_and_stop_token_idsDsz_SentenceTransformersTokenTextSplitter.split_text..encode_strip_start_and_stop_token_ids)rrdecodeencode)r#r)rZ_chunk_overlaprrr*r )rr#r)rrr(r split_text5sz0SentenceTransformersTokenTextSplitter.split_textcCst||S)ayCounts the number of tokens in the given text. This method encodes the input text using a private `_encode` method and calculates the total number of tokens in the encoded result. Args: text (str): The input text for which the token count is calculated. Returns: int: The number of tokens in the encoded text. )lenr')rr#rrr count_tokensPs z2SentenceTransformersTokenTextSplitter.count_tokensl _max_length_equal_32_bit_integerr$cCs|jj||jdd}|S)NZdo_not_truncate) max_lengthZ truncation)rr+r/)rr#Z&token_ids_with_start_and_end_token_idsrrr r'`s z-SentenceTransformersTokenTextSplitter._encode)r r N) __name__ __module__ __qualname____doc__rrr,r.r/__annotations__r' __classcell__rrrr r s  r N) __future__rtypingrrrrZlangchain_text_splitters.baserrr r rrrr s