a bgQ@sddlZddlZddlZddlmZmZddlmZmZm Z m Z m Z ddl m Z ddlmZGdddeeZGdd d eZGd d d eZdS) N)ABCabstractmethod)DictIteratorOptionalTupleUnion)Document) BaseLoaderc@seZdZdZdddddddd d dddd d d eeeeeefeeeeeeeeefeeefeeefeeeed d d dZ e e dddZ e edddZeedddZeee e dddZd eeeee e dddZeeeefdddZd S)!DedocBaseLoadera Base Loader that uses `dedoc` (https://dedoc.readthedocs.io). Loader enables extracting text, tables and attached files from the given file: * `Text` can be split by pages, `dedoc` tree nodes, textual lines (according to the `split` parameter). * `Attached files` (when with_attachments=True) are split according to the `split` parameter. For attachments, langchain Document object has an additional metadata field `type`="attachment". * `Tables` (when with_tables=True) are not split - each table corresponds to one langchain Document object. For tables, Document object has additional metadata fields `type`="table" and `text_as_html` with table HTML representation. documentTF auto_tabbyrus+eng:autoN)split with_tableswith_attachmentsrecursion_deep_attachmentspdf_with_text_layerlanguagepagesis_one_column_documentdocument_orientationneed_header_footer_analysisneed_binarizationneed_pdf_table_analysis delimiterencoding) file_pathrrrrrrrrrrrrrrreturncCs~ddtD|_hd|_||jvrBtd|d|jd||_||_||_|jdkrbdnd }||jd <||jd <d S) a Initialize with file path and parsing parameters. Args: file_path: path to the file for processing split: type of document splitting into parts (each part is returned separately), default value "document" "document": document text is returned as a single langchain Document object (don't split) "page": split document text into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document text into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document text into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV cSsi|]\}}|dvr||qS)>selfrrr ).0keyvaluer#r#x/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/dedoc.py dsz,DedocBaseLoader.__init__..>pagenodeliner Got $ for `split`, but should be one of ``r*treeZlinearstructure_typeZneed_content_analysisN)localsitemsparsing_parametersvalid_split_values ValueErrorrrr )r"r rrrrrrrrrrrrrrr0r#r#r'__init__#s A   zDedocBaseLoader.__init__r!ccsddl}zddlm}Wnty2tdYn0||d}d|jd_|.}|j|j i|j d|id }Wdn1s0Y|j | |jd EdHdS) Lazily load documents.rN) DedocManagerzE`dedoc` package not found, please install it with `pip install dedoc`)Zmanager_configTloggerZattachments_dir)r parameters document_treer)tempfileZdedocr9 ImportError _make_configconfigdisabledTemporaryDirectoryparser r3_split_documentZ to_api_schemadictr)r"r>r9Z dedoc_managerZtmpdirr=r#r#r' lazy_loadws"    $zDedocBaseLoader.lazy_loadcCsdS)zu Make configuration for DedocManager according to the file extension and parsing parameters. Nr#r"r#r#r'r@szDedocBaseLoader._make_config) paragraphr!cs>dfdd|dD}|r2|dd|n|d}|S)z1Get text (recursively) of the document tree node. csg|]}|qSr#) _json2txt)r$ subparagraphrHr#r' sz-DedocBaseLoader._json2txt.. subparagraphstext)join)r"rIZsubparagraphs_textrOr#rHr'rKs zDedocBaseLoader._json2txt)r=document_metadatar!ccsVt|ddkr4|dD]}|j||dEdHqnt|di||ddVdS)z4Parse recursively document tree obtained by `dedoc`.rNrr=rQNrOmetadataZ page_contentrS)len_parse_subparagraphsr )r"r=rQrLr#r#r'rVs z$DedocBaseLoader._parse_subparagraphs)r=radditional_metadatar!ccs|d}|ri||}|dkrF|j|ddd}t||dVn*|dkr|ddd}|d dd }d }|D]T} | dd |kr||| 7}qvt|i|d |idV| dd }|| }qvt|i|d |idVn|d kr0|dddD]*} | d} t|| i|| dVqn@|d krX|j|dd|dEdHntd|d|jd|jr|ddD]4} || \} } t| i| dd| ddVq|dD]"}|j||jddidEdHqdS)z=Split document into parts according to the `split` parameter.rSr contentZ structure)rIrTr)rNrpage_idr+r*rRNr,r-r.Ztablestable)typeZ text_as_html attachmentsr\ attachment)r=rrW) rKr rVr5r4r _get_tablerEr)r"r=rrWrQrOnodesrYZ page_textr*Z line_metadatar[ table_text table_htmlr^r#r#r'rEst         zDedocBaseLoader._split_document)r[r!c Csd}|dD]:}|D](}|ddd|dD7}|d7}q|d7}q d }|dD]|}|d 7}|D]b}dd d|dD}t|}|d 7}|d r|d7}|d|dd|dd|d7}qd|d7}qT|d7}||fS)z.Get text and HTML representation of the table.rZcells css|]}|dVqdSrONr#r$r+r#r#r' z-DedocBaseLoader._get_table..lines rJzK z css|]}|dVqdSrer#rfr#r#r'rgrhzz z z
)rPhtmlescape)r"r[rarowcellrbZ cell_textr#r#r'r_s4       zDedocBaseLoader._get_table)N)__name__ __module__ __qualname____doc__strboolrintrr6rr rGrrFr@rKrVrErr_r#r#r#r'r s`     T  Kr c@seZdZdZedddZdS)DedocFileLoaderaw DedocFileLoader document loader integration to load files using `dedoc`. The file loader automatically detects the file type (with the correct extension). The list of supported file types is gives at https://dedoc.readthedocs.io/en/latest/index.html#id1. Please see the documentation of DedocBaseLoader to get more details. Setup: Install ``dedoc`` package. .. code-block:: bash pip install -U dedoc Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocFileLoader loader = DedocFileLoader( file_path="example.pdf", # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } r7cCs ddlm}||j|j|jdS)Nr)make_manager_config)r Zparsing_paramsr)Zdedoc.utils.langchainrwr r3r)r"rwr#r#r'r@`s  zDedocFileLoader._make_configN)rorprqrrrFr@r#r#r#r'rvsBrvcseZdZdZdddddddd d d dddd d d eeeeeeefeeeeeeeeefeeefeeefeeeed d fddZ e e dddZ e dddZeee eeeee effdddZZS)DedocAPIFileLoaderaU Load files using `dedoc` API. The file loader automatically detects the file type (even with the wrong extension). By default, the loader makes a call to the locally hosted `dedoc` API. More information about `dedoc` API can be found in `dedoc` documentation: https://dedoc.readthedocs.io/en/latest/dedoc_api_usage/api.html Please see the documentation of DedocBaseLoader to get more details. Setup: You don't need to install `dedoc` library for using this loader. Instead, the `dedoc` API needs to be run. You may use Docker container for this purpose. Please see `dedoc` documentation for more details: https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker .. code-block:: bash docker pull dedocproject/dedoc docker run -p 1231:1231 Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocAPIFileLoader loader = DedocAPIFileLoader( file_path="example.pdf", # url=..., # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } zhttp://0.0.0.0:1231r TFr rrrrN)urlrrrrrrrrrrrrrr)r ryrrrrrrrrrrrrrrr!cs>tj|||||||| | | | | |||d||_d|jd<dS)a Initialize with file path, API url and parsing parameters. Args: file_path: path to the file for processing url: URL to call `dedoc` API split: type of document splitting into parts (each part is returned separately), default value "document" "document": document is returned as a single langchain Document object (don't split) "page": split document into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV )r rrrrrrrrrrrrrrjsonZ return_formatN)superr6ryr3)r"r ryrrrrrrrrrrrrrr __class__r#r'r6s&AzDedocAPIFileLoader.__init__r7ccs0|j|j|j|jd}|j||jdEdHdS)r8)ryr r;r<N) _send_fileryr r3rEr)r"Zdoc_treer#r#r'rG s zDedocAPIFileLoader.lazy_loadcCsiS)Nr#rHr#r#r'r@szDedocAPIFileLoader._make_config)ryr r;r!c Csddl}tj|}t|d2}d||fi}|j|d||d}Wdn1sV0Y|jdkr~td|j t |j } | S) z7Send POST-request to `dedoc` API and return the resultsrNrbfilez/upload)filesdatazError during file handling: ) requestsospathbasenameopenpost status_coder5rXdecoderzloads) r"ryr r;r file_namerrrresultr#r#r'r~s   4 zDedocAPIFileLoader._send_file)rorprqrrrsrtrrurr6rr rGrFr@rlistr~ __classcell__r#r#r|r'rxjsPM    Urx)rkrzrabcrrtypingrrrrrZlangchain_core.documentsr Z)langchain_community.document_loaders.baser r rvrxr#r#r#r's   M