a ~g\@sxddlmZddlZddlmZmZmZmZmZddl m Z m Z Gddde Z dddd d d d Z Gd dde ZdS)) annotationsN)AnyListLiteralOptionalUnion)Language TextSplittercs>eZdZdZddddddfd d Zdd d d dZZS)CharacterTextSplitterz(Splitting text that looks at characters. FstrboolrNone) separatoris_separator_regexkwargsreturnc s"tjfi|||_||_dS)Create a new TextSplitter.N)super__init__ _separator_is_separator_regex)selfrrr __class__g/var/www/html/emsaiapi.evdpl.com/venv/lib/python3.9/site-packages/langchain_text_splitters/character.pyr szCharacterTextSplitter.__init__ List[str]textrcCsB|jr |jn t|j}t|||j}|jr0dn|j}|||S)&Split incoming text and return chunks.)rrreescape_split_text_with_regex_keep_separator _merge_splits)rrrsplitsrrrr split_texts z CharacterTextSplitter.split_text)r F)__name__ __module__ __qualname____doc__rr( __classcell__rrrrr sr r $Union[bool, Literal['start', 'end']]r)rrkeep_separatorrcs|r|rtd|d||dkrFfddtdtddDnfd dtdtdD}tddkr|d d7}|dkr|d gn dg|}qt||}nt|}d d|DS) N()endcs g|]}||dqSr.0iZ_splitsrr (z*_split_text_with_regex..rr4cs g|]}||dqSr3rr5r8rrr9*r:cSsg|]}|dkr|qS)r!r)r6srrrr97r:)r"splitrangelenlist)rrr/r'rr8rr$s $ r$cs|eZdZdZdddddd d fd d Zd dddddZd ddddZeddddddZe dddddZ Z S)RecursiveCharacterTextSplitterzSplitting text by recursively look at characters. Recursively tries to split by different characters to find one that works. NTFzOptional[List[str]]r.r rr) separatorsr/rrrc s.tjfd|i||p gd|_||_dS)rr/)r   r!N)rr _separatorsr)rrCr/rrrrrrAsz'RecursiveCharacterTextSplitter.__init__r r)rrCrcCs(g}|d}g}t|D]P\}}|jr*|nt|}|dkrD|}qjt||r|}||dd}qjq|jrt|nt|}t|||j} g} |jrdn|}| D]d} || |jkr| | q| r| | |} | | g} |s| | q| | |} | | q| r$| | |} | | |S)r r<r!r4N) enumeraterr"r#searchr$r%_length_function _chunk_sizeappendr&extend _split_text)rrrCZ final_chunksrZnew_separatorsr7Z_srr'Z _good_splitsr=Z merged_textZ other_inforrrrMMs>         z*RecursiveCharacterTextSplitter._split_textrcCs|||jS)zSplit the input text into smaller chunks based on predefined separators. Args: text (str): The input text to be split. Returns: List[str]: A list of text chunks obtained after splitting. )rMrF)rrrrrr(us z)RecursiveCharacterTextSplitter.split_textr)languagerrcKs||}|f|dd|S)aReturn an instance of this class based on a specific language. This method initializes the text splitter with language-specific separators. Args: language (Language): The language to configure the text splitter for. **kwargs (Any): Additional keyword arguments to customize the splitter. Returns: RecursiveCharacterTextSplitter: An instance of the text splitter configured for the specified language. T)rCr)get_separators_for_language)clsrNrrCrrr from_languages z,RecursiveCharacterTextSplitter.from_language)rNrcCs|tjks|tjkrgdS|tjkr.gdS|tjkr@gdS|tjkrRgdS|tjkrdgdS|tjkrvgdS|tjkrgdS|tj krgdS|tj krgd S|tj krgd S|tj krgd S|tj krgd S|tjkrgd S|tjkr gdS|tjkr gdS|tjkr4gdS|tjkrHgdS|tjkr\gdS|tjkrpgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjvrtd|dntd|dttdS)a Retrieve a list of separators specific to the given language. Args: language (Language): The language for which to get the separators. Returns: List[str]: A list of separators appropriate for the specified language. ) class z void z int z float z double  if  for  while  switch  case r rDrEr!)  func  var  const  type rSrTrVrWr rDrEr!)rR public protected private  static rSrTrUrVrWr rDrEr!)rRr\r]r^z internal z companion z fun  val rYrSrTrUz when rW else r rDrEr!) function rZ let rYrRrSrTrUrVrW default r rDrEr!) enum interface z namespace r[rRrbrZrcrYrSrTrUrVrWrdr rDrEr!) rbrRrS foreach rU do rVrWr rDrEr!) z message z service rez option  import z syntax r rDrEr!)rR def z def r rDrEr!)z =+ z -+ z \*+ z .. * r rDrEr!) rjrRrS unless rUrTrhz begin z rescue r rDrEr!)rjz defp z defmodule z defprotocol z defmacro z defmacrop rSrkrUrWz cond z with rTrhr rDrEr!) z fn rZrcrSrUrTz loop  match rZr rDrEr!)rRz object rjr`rYrSrTrUrlrWr rDrEr!)rXrR struct rerSrTrUrhrVrWr rDrEr!) z #{1,6} z``` z \*\*\*+ z ---+ z ___+ r rDrEr!)z \\chapter{z \\section{z \\subsection{z \\subsubsection{z \\begin{enumerate}z \\begin{itemize}z \\begin{description}z \\begin{list}z \\begin{quote}z \\begin{quotation}z \\begin{verse}z \\begin{verbatim}z \\begin{align}z$$$rEr!)zs