a `gB@sddlmZddlZddlmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZGdddeZGd d d ZGd d d eZGd ddeZGdddZdS)) annotationsN)AnyDictListTuple TypedDictUnionDocument)Language)RecursiveCharacterTextSplittercs(eZdZdZdddfdd ZZS)MarkdownTextSplitterz=Attempts to split the text along Markdown-formatted headings.rNone)kwargsreturnc s&|tj}tjfd|i|dS)z"Initialize a MarkdownTextSplitter. separatorsN)Zget_separators_for_languager MARKDOWNsuper__init__)selfrr __class__o/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_text_splitters/markdown.pyrs zMarkdownTextSplitter.__init__)__name__ __module__ __qualname____doc__r __classcell__rrrrr sr c@sDeZdZdZdddddddZd d d d d Zdd dddZdS)MarkdownHeaderTextSplitterz4Splitting markdown files based on specified headers.FTzList[Tuple[str, str]]boolheaders_to_split_onreturn_each_line strip_headerscCs$||_t|dddd|_||_dS)a Create a new MarkdownHeaderTextSplitter. Args: headers_to_split_on: Headers we want to track return_each_line: Return each line w/ associated headers strip_headers: Strip split headers from the content of the chunk cSs t|dS)Nr)len)splitrrr*z5MarkdownHeaderTextSplitter.__init__..T)keyreverseN)r#sortedr"r$rr"r#r$rrrrs  z#MarkdownHeaderTextSplitter.__init__zList[LineType]List[Document])linesrcCsg}|D]}|rB|dd|dkrB|ddd|d7<q|r|dd|dkrt|ddt|dkr|ddddddkr|js|ddd|d7<|d|dd<q||qdd |DS) zCombine lines with common metadata into chunks. Args: lines: Line of text / associated header metadata metadatacontentz  r#cSs g|]}t|d|ddqSr1r0 page_contentr0r .0chunkrrr UszHMarkdownHeaderTextSplitter.aggregate_lines_to_chunks..)r%r&r$append)rr.Zaggregated_chunkslinerrraggregate_lines_to_chunks/s. z4MarkdownHeaderTextSplitter.aggregate_lines_to_chunksstrtextrcCs&|d}g}g}i}g}i}d}d} |D]} | } dttj| } |s| drp| ddkrpd}d} q| drd}d} n| | rd}d} |r|| q*|j D]\} } | | rt | t | ks| t | dkr| d ur`| d }|r.|d d |kr.| }|d |vr| |d q|| | t | d d}|||d|| <|r|d|| d| |js|| qq| r|| n(|r|d|| d| | }q*|r|d||d|js||Sdd|DSd S)zLSplit markdown file. Args: text: Markdown file r2Fz```Tz~~~ Nr3r/levelname)rDrEdatarF)r1r0cSs g|]}t|d|ddqSr4r r7rrrr:sz9MarkdownHeaderTextSplitter.split_text..)r&stripjoinfilterr> isprintable startswithcountr;r"r%popcopyclearr$r#r=)rr@r.Zlines_with_metadataZcurrent_contentZcurrent_metadataZ header_stackZinitial_metadataZ in_code_blockZ opening_fencer<Z stripped_lineseprEZcurrent_header_levelZ popped_headerheaderrrr split_textZs                z%MarkdownHeaderTextSplitter.split_textN)FT)rrrrrr=rRrrrrrs +rc@s"eZdZUdZded<ded<dS)LineTypezLine type as typed dict.zDict[str, str]r0r>r1Nrrrr__annotations__rrrrrSs rSc@s*eZdZUdZded<ded<ded<dS) HeaderTypezHeader type as typed dict.intrDr>rErFNrTrrrrrVs rVc@seZdZdZdddddddZd*d d d dddZdddddZddddddZddddddZddd d!Z dd"d#d$d%Z dd"d#d&d'Z dd"d#d(d)Z d S)+&ExperimentalMarkdownSyntaxTextSplitteraAn experimental text splitter for handling Markdown syntax. This splitter aims to retain the exact whitespace of the original text while extracting structured metadata, such as headers. It is a re-implementation of the MarkdownHeaderTextSplitter with notable changes to the approach and additional features. Key Features: - Retains the original whitespace and formatting of the Markdown text. - Extracts headers, code blocks, and horizontal rules as metadata. - Splits out code blocks and includes the language in the "Code" metadata key. - Splits text on horizontal rules (`---`) as well. - Defaults to sensible splitting behavior, which can be overridden using the `headers_to_split_on` parameter. Parameters: ---------- headers_to_split_on : List[Tuple[str, str]], optional Headers to split on, defaulting to common Markdown headers if not specified. return_each_line : bool, optional When set to True, returns each line as a separate chunk. Default is False. Usage example: -------------- >>> headers_to_split_on = [ >>> ("#", "Header 1"), >>> ("##", "Header 2"), >>> ] >>> splitter = ExperimentalMarkdownSyntaxTextSplitter( >>> headers_to_split_on=headers_to_split_on >>> ) >>> chunks = splitter.split(text) >>> for chunk in chunks: >>> print(chunk) This class is currently experimental and subject to change based on feedback and further development. zHeader 1zHeader 2zHeader 3zHeader 4zHeader 5zHeader 6)r3z##z###z####z#####z######NFTz"Union[List[Tuple[str, str]], None]r r!cCs@g|_tdd|_g|_||_|r.t||_n|j|_||_dS)a1Initialize the text splitter with header splitting and formatting options. This constructor sets up the required configuration for splitting text into chunks based on specified headers and formatting preferences. Args: headers_to_split_on (Union[List[Tuple[str, str]], None]): A list of tuples, where each tuple contains a header tag (e.g., "h1") and its corresponding metadata key. If None, default headers are used. return_each_line (bool): Whether to return each line as an individual chunk. Defaults to False, which aggregates lines into larger chunks. strip_headers (bool): Whether to exclude headers from the resulting chunks. Defaults to True. rAr6N) chunksr current_chunkcurrent_header_stackr$dictsplittable_headersDEFAULT_HEADER_KEYSr#r,rrrrs  z/ExperimentalMarkdownSyntaxTextSplitter.__init__r>r-r?c Cs|jtdd|_|j|jdd}|r|d}||}||}| |}|r| |j sz|jj |7_ t |d}|d}|||q,|r| ||||j_ |d|jjd<| q,|r| q,|jj |7_ q,| |jrd d |jDS|jS) aySplit the input text into structured chunks. This method processes the input text line by line, identifying and handling specific patterns such as headers, code blocks, and horizontal rules to split it into structured chunks based on headers, code blocks, and horizontal rules. Args: text (str): The input text to be split into chunks. Returns: List[Document]: A list of `Document` objects representing the structured chunks of the input text. If `return_each_line` is enabled, each line is returned as a separate `Document`. rArYT)keependsrrBZCodecSs6g|].}|jD]}|r|st||jdqqS)r5)r6 splitlinesisspacer r0)r8r9r<rrrr:qs zEExperimentalMarkdownSyntaxTextSplitter.split_text..)rZrOr r[r\rbrM _match_header _match_code _match_horz_complete_chunk_docr$r6r%group_resolve_header_stack_resolve_code_chunkr0r#) rr@ raw_linesraw_lineZ header_matchZ code_matchZ horz_match header_depth header_textrrrrR:s@           z1ExperimentalMarkdownSyntaxTextSplitter.split_textrWr)rmrnrcCs\t|jD]<\}\}}||kr ||f|j|<|jd|d|_dSq |j||fdS)NrB) enumerater\r;)rrmrnidepth_rrrriys z.)z^```(.*)z^~~~(.*)css|]}|r|VqdSNrr8rwrrr r(zEExperimentalMarkdownSyntaxTextSplitter._match_code..nextrr<matchesrr{rresz2ExperimentalMarkdownSyntaxTextSplitter._match_codecs&fdddD}tdd|DdS)Ncsg|]}t|qSrrxryr{rrr:szFExperimentalMarkdownSyntaxTextSplitter._match_horz..)z ^\*\*\*+\nz^---+\nz^___+\ncss|]}|r|VqdSr|rr}rrrr~r(zEExperimentalMarkdownSyntaxTextSplitter._match_horz..rrrr{rrfs z2ExperimentalMarkdownSyntaxTextSplitter._match_horz)NFT) rrrrr_rrRrirjrgrdrerfrrrrrXs&( !?  rX) __future__rrvtypingrrrrrrZlangchain_core.documentsr Zlangchain_text_splitters.baser Z"langchain_text_splitters.characterr r rrSrVrXrrrrs      E