a bg[@sdZddlmZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZmZmZmZddlmZddlZddlZdd lmZdd lm Z dd l!m"Z"dd l#m$Z$m%Z%erddl&Z&ddl'Z'ddl(Z(ddl)Z)dd l*m+Z+gdZ,gdZ-dddddZ.e/e0Z1dZ2dZ3dZ4dZ5hdZ6dddddddZ7ddddd Z8dddd!d"Z9d#d$gZ:d%ddd&d'd(Z;Gd)d*d*e ZGd/d0d0e Z?Gd1d2d2e Z@Gd3d4d4e ZAGd5d6d6e ZBdS)7z(Module contains common parsers for PDFs.) annotationsN)datetime)Path)TemporaryDirectory) TYPE_CHECKINGAnyBinaryIOIterableIteratorLiteralMappingOptionalSequenceUnioncast)urlparse)Document)BaseBlobParser)Blob)BaseImageBlobParserRapidOCRBlobParser)TextLinearizationConfig)Z DCTDecodeZDCTZ JPXDecode) Z LZWDecodeZLZWZ FlateDecodeZFlZ ASCII85DecodeZA85ZASCIIHexDecodeZAHxZRunLengthDecodeZRLZCCITTFaxDecodeZCCFZ JBIG2Decodez,Sequence[Union[Iterable[np.ndarray], bytes]]str)imagesreturncCspzddlm}Wnty*tdYn0|}d}|D]0}||\}}|r:dd|D}|d|7}q:|S)zExtract text from images with RapidOCR. Args: images: Images to extract text from. Returns: Text extracted from images. Raises: ImportError: If `rapidocr-onnxruntime` package is not installed. r)RapidOCRzc`rapidocr-onnxruntime` package not found, please install it with `pip install rapidocr-onnxruntime`cSsg|] }|dqS)).0textrr~/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/parsers/pdf.py Zz5extract_from_images_with_rapidocr.. )Zrapidocr_onnxruntimer ImportErrorjoin)rrZocrr imgresult_rrr!!extract_from_images_with_rapidocr@s   r*z {image_text} r$z >producer total_pagescreatorsource creationdater)blobcontentformatrcCs^|rZ|jp d}|dkr6|dd}d|d|d}n$|dkrZd tj|d d d |d }|S)aFormat the content of the image with the source of the blob. blob: The blob containing the image. format:: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) #z markdown-img]z\\]z![z]()zhtml-imgz T)quotez src=)r.replacehtmlescape)r0r1r2r.rrr!_format_inner_imageis   r:dict[str, Any])metadatarcCs4t|stdt|ddts0td|S)zValidate that the metadata has all the standard keys and the page is an integer. The standard keys are: - source - total_page - creationdate - creator - producer Validate that page is an integer if it is present. z3The PDF parser must valorize the standard metadata.pagerz(The PDF metadata page must be a integer.)_STD_METADATA_KEYSissubsetkeys ValueError isinstancegetint)r<rrr!_validate_metadatas rEc Csi}ddd}|D]\}}t|ttfvr6t|}|drL|dd}|}|dvrz"t|dd d  d ||<Wqt y|||<Yq0q||vr||||<|||<qt |tr| ||<qt |tr|||<q|S) zPurge metadata from unwanted keys and normalize key names. Args: metadata: The original metadata dictionary. Returns: The cleaned and normalized the key format of metadata dictionary. r,r.)Z page_count file_path/rN)r/Zmoddate'rzD:%Y%m%d%H%M%S%zT) itemstyperrD startswithlowerrstrptimer7 isoformatrArBstrip)r<Z new_metadataZmap_keykvrrr!_purge_metadatas6           rSz  list[str])extrastext_from_pagercs\dddddfdd ||d}|sXd }d td d |}|rPtd |}||}|S)a5Insert extras such as image/table in a text between two paragraphs if possible, else at the end of the text. Args: extras: List of extra content (images/tables) to insert. text_from_page: The text content from the page. Returns: The merged text with extras inserted. rUrbool Optional[str])rVrWrecursrc s|rtD]}||}|dkrd}|r:||d|d}|rP|||d}n@d}dtdd|}|rt||}|d||||d}qqd}n|}|S)NFrrTcSs|SNrxrrr!r#zO_merge_text_and_extras.._recurs_merge_text_and_extras..)_PARAGRAPH_DELIMITERrfindr&filter) rVrWrZdelimposZ previous_textall_text all_extras str_extras_recurs_merge_text_and_extrasrr!ris* z=_merge_text_and_extras.._recurs_merge_text_and_extrasTrrTcSs|Sr\rr]rrr!r_r#z(_merge_text_and_extras..r[)r&rbr`)rVrWrerfrgrrhr!_merge_text_and_extrass   rjc sfeZdZdZddeddddddd d d d d dddfddZdddddZdd dddZZS) PyPDFParseraParse a blob from a PDF using `pypdf` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images. It integrates the 'pypdf' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdf Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PyPDFParser parser = PyPDFParser( # password = None, mode = "single", pages_delimiter = " ", # images_parser = TesseractBlobParser(), ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) NFr=r plain)modepages_delimiter images_parserimages_inner_formatextraction_modeextraction_kwargszOptional[Union[str, bytes]]rXLiteral['single', 'page']rOptional[BaseImageBlobParser]+Literal['text', 'markdown-img', 'html-img']zLiteral['plain', 'layout']Optional[dict[str, Any]])passwordextract_imagesrmrnrorprqrrc s`t|dvrtd||_|r.|s.t}||_||_||_||_||_ ||_ |pXi|_ dS)uInitialize a parser based on PyPDF. Args: password: Optional password for opening encrypted PDFs. extract_images: Whether to extract images from the PDF. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extraction_mode: “plain” for legacy functionality, “layout” extract text in a fixed width format that closely adheres to the rendered layout in the source pdf. extraction_kwargs: Optional additional parameters for the extraction process. Raises: ValueError: If the `mode` is not "single" or "page". singler=mode must be single or pageN) super__init__rArxrrorprwrmrnrqrr) selfrwrxrmrnrorprqrr __class__rr!r}&s% zPyPDFParser.__init__rIterator[Document]r0rc #sHz ddlWnty&tdYn0dddfdd }|}j|jd }td d d d tt|jppiB|j t |j d B}g}t |j D]d\}}||d}  |} t| g| } jdkrt| t|||j|dBdVq|| qjdkr$tj|t|dVWdn1s:0YdS)am Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. Raises: ImportError: If the `pypdf` package is not found. Yield: An iterator over the parsed documents. rNzCpypdf package not found, please install it with `pip install pypdf`zpypdf.PageObjectrr=rcs0jdr|S|jfdjijSdS)z Extract text from image given the version of pypdf. Args: page: The page object to extract text from. Returns: str: The extracted text. 3rqN) __version__rL extract_textrqrrr=pypdfr~rr!_extract_text_from_pageos z7PyPDFParser.lazy_parse.._extract_text_from_pagerwZPyPDFr)r+r-r/)r.r,rr=)r=Z page_label page_contentr<rz)rr% as_bytes_ioZ PdfReaderrwrSrdictr<r.lenpages enumerateextract_images_from_pagerjrPrmrrEZ page_labelsappendrnr&) r~r0r pdf_file_obj pdf_reader doc_metadataZ single_texts page_numberr=rWimages_from_pagererrr! lazy_parseYsT           zPyPDFParser.lazy_parsezpypdf._page.PageObjectrc Csp|js dSddlm}dtt|dvr0dS|dd}g}|D] }d}||ddkrH||d d dtvr||d ||d }}tj || tj d  ||d}nB||d d dt vrt|t|| }n td|durHt} ||j| ddtj| dd} t|j| j} |t| | |jqHtj t!"t#d|dS)zExtract images from a PDF page and get the text using images_to_text. Args: page: The page object from which to extract images. Returns: str: The extracted text from the images on the page. rrImagez/XObjectz /ResourcesNz/Subtypez/Imagez/Filterrz/Heightz/WidthZdtyper[Unknown PDF Filter!ZPNG)r2z image/pngZ mime_type image_text)$roPILrrrr@Z get_object_PDF_FILTER_WITHOUT_LOSSnp frombufferget_datauint8reshape_PDF_FILTER_WITH_LOSSarrayopenioBytesIOloggerwarningZ fromarraysaver from_datagetvaluenextrrrr:rp_FORMAT_IMAGE_STRr2 _JOIN_IMAGESr&rb) r~r=rZxObjectrobjZnp_imageheightwidth image_bytesr0rrrr!rs>      z$PyPDFParser.extract_images_from_page)NF) __name__ __module__ __qualname____doc___DEFAULT_PAGES_DELIMITERr}rr __classcell__rrrr!rks2$3Mrkc seZdZdZdZd%ddeddddddd d d d d dfddZedd dddZedddddZ d&dd dddddZ d d!d"d#d$Z Z S)'PDFMinerParseraParse a blob from a PDF using `pdfminer.six` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images, and defining extraction mode. It integrates the 'pdfminer.six' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pdfminer.six pillow Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PDFMinerParser parser = PDFMinerParser( # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_to_text = convert_images_to_text_with_tesseract(), ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) FNrzr )rwrmrnrorpconcatenate_pagesrXrYrsrrtruzOptional[bool])rxrwrmrnrorprcs|t|dvrtd|r(|s(t}||_||_||_||_||_||_ |durxt j sjdt _ t d|rrdnd|_dS)aHInitialize a parser based on PDFMiner. Args: password: Optional password for opening encrypted PDFs. mode: Extraction mode to use. Either "single" or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from PDF. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) concatenate_pages: Deprecated. If True, concatenate all PDF pages into one a single document. Otherwise, return one document per page. Returns: This method does not directly return data. Use the `parse` or `lazy_parse` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the `mode` is not "single" or "page". Warnings: `concatenate_pages` parameter is deprecated. Use `mode='single' or 'page' instead. ryr{NTzS`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'` instead.rzr=)r|r}rArrxrorprwrmrnr_warn_concatenate_pagesrr)r~rxrwrmrnrorprrrr!r}s$( zPDFMinerParser.__init__zUnion[bytes, str])srcs|ddlmt|tr4|dr4t|ddddSz(dd |D}d fd d |DWStyvt|YS0dS) z Decodes a PDFDocEncoding string to Unicode. Adds py3 compatibility to pdfminer's version. Args: s: The string to decode. Returns: str: The decoded Unicode string. rPDFDocEncodingsNzutf-16beignorecss$|]}t|trt|n|VqdSr\)rBrord)rcrrr! Ur#z-PDFMinerParser.decode_text..rc3s|]}|VqdSr\r)rorrr!rVr#)Zpdfminer.utilsrrBbytesrLrr& IndexError)rZordsrrr! decode_textDs  zPDFMinerParser.decode_textr)rrcCsddlm}t|dr|}t|tr8tttj|St||rNt |j St|t t frft |St|t r|D]\}}t|||<qx|S|S)z Recursively resolve the metadata values. Args: obj: The object to resolve and decode. It can be of any type. Returns: The resolved and decoded object. r) PSLiteralresolve)Zpdfminer.psparserrhasattrrrBlistmaprresolve_and_decodernamerrrrJ)rrrQrRrrr!rZs       z!PDFMinerParser.resolve_and_decoderTrr;)fprwcachingrc Csddlm}m}m}||}||||d}i} |jD]} | | q4| D]R\} } zt| | | <WqLt y} zt d| t | WYd} ~ qLd} ~ 00qLt t||| d<| S)ag Extract metadata from a PDF file. Args: fp: The file pointer to the PDF file. password: The password for the PDF file, if encrypted. Defaults to an empty string. caching: Whether to cache the PDF structure. Defaults to True. Returns: Metadata of the PDF file. r) PDFDocumentPDFPage PDFParser)rwrzD[WARNING] Metadata key "%s" could not be parsed due to exception: %sNr,)pdfminer.pdfpagerrrinfoupdaterJrr ExceptionrrrrrZ create_pages)r~rrwrrrrparserdocr<rrQrRerrr! _get_metadatavs"  zPDFMinerParser._get_metadatarrrc#s2znddl}ddlm}ddlmmmmmm m ddl m }m ddlm}t|jdkrltdWntytd Yn0|}t^ |j|jpd d }}tj|jpd d } |j| d <G f d dd|} t ||| |d} g} t|D]\} } d d| | }|!}j"dkr d dt#|t$| d| iBdVn"|%dr|dd}| &|q,j"dkrj'(| }t#|t$| dVWdn1s0YWdn1s$0YdS)a Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. Raises: ImportError: If the `pdfminer.six` or `pillow` package is not found. Yield: An iterator over the parsed documents. rN)PDFLayoutAnalyzer)LAParams LTContainerLTImageLTItemLTPageLTText LTTextBox)PDFPageInterpreterPDFResourceManager)ri:>4zThis parser is tested with pdfminer.six version 20201018 or later. Remove pdfminer, and install pdfminer.six with `pip uninstall pdfminer && pip install pdfminer.six`.zMpdfminer package not found, please install it with `pip install pdfminer.six`rrr.c sLeZdZddddddfdd Zd dd  fd d ZZS)z*PDFMinerParser.lazy_parse..VisitorrNrrDzOptional[LAParams]None)rsrcmgrpagenolaparamsrcstj|||ddS)N)rr)r|r})r~rrrrrr!r}sz3PDFMinerParser.lazy_parse..Visitor.__init__r)ltpagerc s.dddfdd |dS)Nrr)itemrcst|r|D] }|qnt|r6|t|rLdnrt|rjrddlm}|}||}tt |}d|j d<t j |j }t||jndS)Nr$r) ImageWriterr3r.)rBwriteget_textroZpdfminer.imagerZ export_imager from_pathrr<rrrr:rp)rchildrZ image_writerfilenamer0r)rrrrrenderr~tempdirtext_iorr!rs.          zIPDFMinerParser.lazy_parse..Visitor.receive_layout..renderr)mer)rrrrrr~rr)rr!receive_layouts"z9PDFMinerParser.lazy_parse..Visitor.receive_layout)rN)rrrr}rrr rrrrrrrrr~rrrr!Visitorsr)rr=r r[rz))pdfminerZpdfminer.converterrZpdfminer.layoutrrrrrrrZpdfminer.pdfinterprrrrrDrr%rrZ get_pagesrwrSrr.rStringIOrtruncateseekZ process_pagerrPrmrrEendswithrrnr&)r~r0rrrrrrrrrZvisitor_for_allZ all_contentir=reZdocument_contentrrr!rsb $     ((           zPDFMinerParser.lazy_parse)F)rT) rrrrrrr} staticmethodrrrrrrrrr!rs&2"<.rc seZdZdZeZd+ddedddddddd d d d d dddd fddZdddddZ d,ddddddZ dddd ddd Z ddd!d"d#d$Z ddd d%d&d'Z dd d(d)d*ZZS)- PyMuPDFParseraParse a blob from a PDF using `PyMuPDF` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images, and defining extraction mode. It integrates the 'PyMuPDF' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pymupdf Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PyMuPDFParser parser = PyMuPDFParser( # password = None, mode = "single", pages_delimiter = " ", # images_parser = TesseractBlobParser(), # extract_tables="markdown", # extract_tables_settings=None, # text_kwargs=None, ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) NFr=r )rwrmrnrorpextract_tablesextract_tables_settingsrvrXrYrsrrtruz/Union[Literal['csv', 'markdown', 'html'], None]r) text_kwargsrxrwrmrnrorpr r rc szt|dvrtd|r.|dvr.td||_||_||_|pFi|_|rX|sXt}||_||_ ||_ ||_ | |_ dS)aInitialize a parser based on PyMuPDF. Args: password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extract_tables: Whether to extract tables in a specific format, such as "csv", "markdown", or "html". extract_tables_settings: Optional dictionary of settings for customizing table extraction. Returns: This method does not directly return data. Use the `parse` or `lazy_parse` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the mode is not "single" or "page". ValueError: If the extract_tables format is not "markdown", "html", or "csv". ryr{)markdownr8csvzmode must be markdownN) r|r}rArmrnrwr rrxrpror r ) r~r rxrwrmrnrorpr r rrr!r}Xs +   zPyMuPDFParser.__init__rrrcCs ||Sr\) _lazy_parse)r~r0rrr!rszPyMuPDFParser.lazy_parse)r0r rccszhddl}|p|j}|jsfddlm}m}m}m}ddddd|dd|ddd||ddddddddd|_WntytdYn0t j | }|j dur| |} n|j |dd } | jr| |j|| |} g} | D]J} || | |} |jd kr t| t| d | jiBd Vq| | q|jd krRt|j| t| d VWdn1sh0YWdn1s0YdS) a Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. text_kwargs: Optional keyword arguments to pass to the `get_text` method. If provided at run time, it will override the default text_kwargs. Raises: ImportError: If the `pypdf` package is not found. Yield: An iterator over the parsed documents. rN)DEFAULT_JOIN_TOLERANCEDEFAULT_MIN_WORDS_HORIZONTALDEFAULT_MIN_WORDS_VERTICALDEFAULT_SNAP_TOLERANCElines)ZclipZvertical_strategyZhorizontal_strategyZvertical_linesZhorizontal_linesZsnap_toleranceZsnap_x_toleranceZsnap_y_toleranceZjoin_toleranceZjoin_x_toleranceZjoin_y_toleranceZedge_min_lengthZmin_words_verticalZmin_words_horizontalZintersection_toleranceZintersection_x_toleranceZintersection_y_toleranceZtext_toleranceZtext_x_toleranceZtext_y_toleranceZstrategyZ add_lineszGpymupdf package not found, please install it with `pip install pymupdf`Zpdf)streamfiletyper=rrz)pymupdfr r Z pymupdf.tablerrrrr%r_lockrdatar is_encryptedZ authenticaterw_extract_metadata_get_page_contentrPrmrrEnumberrrnr&)r~r0r rrrrrrFrrZ full_contentr=rerrr!rsp                zPyMuPDFParser._lazy_parsezpymupdf.Documentz pymupdf.Pager;)rr=r rc Cs^|jfii|j|}|||}||}g}|rB|||rP||t||}|S)a:Get the text of the page using PyMuPDF and RapidOCR and issue a warning if it is empty. Args: doc: The PyMuPDF document object. page: The PyMuPDF page object. blob: The blob being parsed. Returns: str: The text content of the page. )rr _extract_images_from_page_extract_tables_from_pagerrj) r~rr=r rWrZtables_from_pagerVrerrr!rs     zPyMuPDFParser._get_page_contentr)rr0rc s\tiddd|j|jtdfddjD}dD]}|jvr:j|||<q:|S)zExtract metadata from the document and page. Args: doc: The PyMuPDF document object. blob: The blob being parsed. Returns: dict: The extracted metadata. ZPyMuPDFr)r+r-r/r.rFr,cs,i|]$}tj|ttfr|j|qSr)rBr<rrDrrQrrr! (sz3PyMuPDFParser._extract_metadata..)ZmodDateZ creationDate)rSr.rr<)r~rr0r<rQrr!r!rs&   zPyMuPDFParser._extract_metadata)rr=rc Cs|js dSddl}|}g}|D]}|jr"|d}|||}tj|jtjd|j |j d} t } t | | tj| dd} t|j| j} |t| | |jq"tjttd|dS) a Extract images from a PDF page and get the text using images_to_text. Args: doc: The PyMuPDF document object. page: The PyMuPDF page object. Returns: str: The extracted text from the images on the page. rrNrr[zapplication/x-npyrr)rorZ get_imagesZPixmaprrZsamplesrrrrrrnumpyrrrrrrrrr:rprr2rr&rb) r~rr=rZimg_listrr'ZxrefZpiximagerr0rrrr!r4s0     z'PyMuPDFParser._extract_images_from_pagercCs|jdurdSddl}t|jj|fi|j}|r|jdkrRtdd|DS|jdkrptdd|DS|jd krtd d|DStd |jd dS) zExtract tables from a PDF page. Args: page: The PyMuPDF page object. Returns: str: The extracted tables in the specified format. Nrrr cSsg|] }|qSr)Z to_markdownrtablerrr!r"mr#z;PyMuPDFParser._extract_tables_from_page..r8cSs g|]}|jddddqS)F)headerindexZ bold_rows) to_pandasZto_htmlr%rrr!r"ps r cSsg|]}|jdddqS)F)r'r()r)Zto_csvr%rrr!r"{s zextract_tables z not implemented) r rrr&Z find_tablesr  _JOIN_TABLESr&rA)r~r=rZ tables_listrrr!r[s2      z'PyMuPDFParser._extract_tables_from_page)NF)N)rrrr threadingLockrrr}rrrrrrrrrrr!rs(6(= ] 'rc@sBeZdZdZddddddZdd d d d Zd ddddZdS)PyPDFium2ParserzParse `PDF` with `PyPDFium2`.FrXr)rxrcCs2z ddl}Wnty&tdYn0||_dS)zInitialize the parser.rNzKpypdfium2 package not found, please install it with `pip install pypdfium2`) pypdfium2r%rx)r~rxr.rrr!r}s   zPyPDFium2Parser.__init__rrrc csddl}|}|j|dd}zjt|D]T\}}|}|}||d||7}||j|d} t || dVq*W|n |0Wdn1s0YdS)Lazily parse the blob.rNT)Z autocloser$r.r=r) r.rZ PdfDocumentrZ get_textpageZget_text_rangecloserr.r) r~r0r.rFrrr=Z text_pager1r<rrr!rs  zPyPDFium2Parser.lazy_parsezpypdfium2._helpers.page.PdfPagerrcCsD|js dSddlm}t|j|jfd}ttdd|}t|S)8Extract images from page and get the text with RapidOCR.rrN)rbcSs |Sr\)Z get_bitmapZto_numpyr]rrr!r_r#z;PyPDFium2Parser._extract_images_from_page..)rxZ pypdfium2.rawrawrZ get_objectsZFPDF_PAGEOBJ_IMAGErr*)r~r=Zpdfium_crrrr!rs  z)PyPDFium2Parser._extract_images_from_pageN)F)rrrrr}rrrrrr!r-s r-c@sVeZdZdZdddddddd Zd d d d dZdddddZdddddZdS)PDFPlumberParserzParse `PDF` with `PDFPlumber`.NFzOptional[Mapping[str, Any]]rXr)r deduperxrcCsBz ddl}Wnty&tdYn0|p.i|_||_||_dS)zInitialize the parser. Args: text_kwargs: Keyword arguments to pass to ``pdfplumber.Page.extract_text()`` dedupe: Avoiding the error of duplicate characters if `dedupe=True`. rNzEpillow package not found, please install it with `pip install pillow`)rr%r r5rx)r~r r5rxrrrr!r}s     zPDFPlumberParser.__init__rrrc#s\ddl}8}||fddjDEdHWdn1sN0YdS)r/rNc sbg|]Z}t|d|tjj|jdtjdfifddjDdqS)r$r)r.rFr=r,cs.i|]&}tj|ttfvr|j|qSr)rKr<rrDr r!rr!r"sz:PDFPlumberParser.lazy_parse...r) r_process_page_contentrrr.rrrr<)rr=r0rr~rr!r"s& z/PDFPlumberParser.lazy_parse..) pdfplumberrrr)r~r0r8rFrr7r!rs   zPDFPlumberParser.lazy_parsezpdfplumber.page.PagerrcCs.|jr|jfi|jS|jfi|jS)z)Process the page content based on dedupe.)r5Z dedupe_charsrr )r~r=rrr!r6sz&PDFPlumberParser._process_page_contentc Csddlm}|jsdSg}|jD]}|ddjtvr|dddkr|t| d|dd |dd f|d  d q|tj |d tj d |dd |dd d q |ddjtvr||d q tdq t|S)r2rrrrFilterZBitsPerComponentr1ZWidthZHeightLrr[r)rrrxrrrrrr frombytesrconvertrrrrwarningswarnr*)r~r=rrr'rrr!rs6     z*PDFPlumberParser._extract_images_from_page)NFF)rrrrr}rr6rrrrr!r4sr4c@s<eZdZdZddddddddd d Zd d d ddZdS)AmazonTextractPDFParsera{Send `PDF` files to `Amazon Textract` and parse them. For parsing multi-page PDFs, they have to reside on S3. The AmazonTextractPDFLoader calls the [Amazon Textract Service](https://aws.amazon.com/textract/) to convert PDFs into a Document structure. Single and multi-page documents are supported with up to 3000 pages and 512 MB of size. For the call to be successful an AWS account is required, similar to the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) requirements. Besides the AWS configuration, it is very similar to the other PDF loaders, while also supporting JPEG, PNG and TIFF and non-native PDF formats. ```python from langchain_community.document_loaders import AmazonTextractPDFLoader loader=AmazonTextractPDFLoader("example_data/alejandro_rosalez_sample-small.jpeg") documents = loader.load() ``` One feature is the linearization of the output. When using the features LAYOUT, FORMS or TABLES together with Textract ```python from langchain_community.document_loaders import AmazonTextractPDFLoader # you can mix and match each of the features loader=AmazonTextractPDFLoader( "example_data/alejandro_rosalez_sample-small.jpeg", textract_features=["TABLES", "LAYOUT"]) documents = loader.load() ``` it will generate output that formats the text in reading order and try to output the information in a tabular structure or output the key/value pairs with a colon (key: value). This helps most LLMs to achieve better accuracy when processing these texts. N)linearization_configzOptional[Sequence[int]]z Optional[Any]z!Optional[TextLinearizationConfig]r)textract_featuresclientrArcsztddlddlmm}|_||_|durFfdd|D|_ng|_|dur\||_n|jjddddd |_Wnt yt d Yn0|szddl }| d |_ Wqt yt d Yq0n||_ dS) a5Initializes the parser. Args: textract_features: Features to be used for extraction, each feature should be passed as an int that conforms to the enum `Textract_Features`, see `amazon-textract-caller` pkg client: boto3 textract client linearization_config: Config to be used for linearization of the output should be an instance of TextLinearizationConfig from the `textractor` pkg rNcsg|]}|qSr)ZTextract_Features)rftcrr!r"asz4AmazonTextractPDFParser.__init__..Tz# z## *)Zhide_figure_layoutZ title_prefixZsection_header_prefixZlist_element_prefixzCould not import amazon-textract-caller or amazon-textract-textractor python package. Please install it with `pip install amazon-textract-caller` & `pip install amazon-textract-textractor`.ZtextractzRCould not import boto3 python package. Please install it with `pip install boto3`.) ZtextractcallerZtextractor.entities.documententitiesdocumentrF textractorrBrArr%boto3rCboto3_textract_client)r~rBrCrArJrKrrEr!r}Fs>       z AmazonTextractPDFParser.__init__rrrccs|jrtt|jnd}|rJ|jdkrJ|jrJ|jjt|j|j|jd}n"|jj| |j|jj j |jd}|j j |}t|jD],\}}t |j|jd|j|dddVqdS) zIterates over the Blob pages and returns an Iterator with a Document for each page, like the other parsers If multi-page document, blob.path has to be set to the S3 URI and for single page docs the blob.data is taken NZs3)input_documentfeaturesrL)rMrNZ call_moderL)configrr0r)pathrrschemenetlocrFZ call_textractrBrLas_bytesZTextract_Call_ModeZ FORCE_SYNCrJrrrrrrAr.)r~r0Zurl_parse_resultZtextract_response_jsonrIidxr=rrr!rs0 z"AmazonTextractPDFParser.lazy_parse)NN)rrrrr}rrrrr!r@s/?r@c@sBeZdZdZdddddZdddd d d Zddd d dZdS)DocumentIntelligenceParserzjLoads a PDF with Azure Document Intelligence (formerly Form Recognizer) and chunks at character level.rr)rCmodelcCstd||_||_dS)Na<langchain_community.document_loaders.parsers.pdf.DocumentIntelligenceParserand langchain_community.document_loaders.pdf.DocumentIntelligenceLoader are deprecated. Please upgrade to langchain_community.document_loaders.DocumentIntelligenceLoader for any file parsing purpose using Azure Document Intelligence service.)r>r?rCrV)r~rCrVrrr!r}s z#DocumentIntelligenceParser.__init__rr)r0r(rccsB|jD]6}ddd|jD}t||j|jdd}|VqdS)N cSsg|] }|jqSr)r1)rlinerrr!r"r#z=DocumentIntelligenceParser._generate_docs..r0r)rr&rrr.r)r~r0r(pr1drrr!_generate_docss z)DocumentIntelligenceParser._generate_docsrccsZ|>}|j|j|}|}|||}|EdHWdn1sL0YdS)r/N)rrCZbegin_analyze_documentrVr(r[)r~r0Zfile_objZpollerr(docsrrr!rs   z%DocumentIntelligenceParser.lazy_parseN)rrrrr}r[rrrrr!rUs  rU)Cr __future__rr8rloggingr+r>rpathlibrtempfilertypingrrrr r r r r rrr urllib.parserr#rZlangchain_core.documentsrZ)langchain_community.document_loaders.baserZ1langchain_community.document_loaders.blob_loadersrZ3langchain_community.document_loaders.parsers.imagesrrr8rrr.Z)textractor.data.text_linearization_configrrrr* getLoggerrrrrr*rr>r:rErSr`rjrkrrr-r4r@rUrrrr!sh    4      '5_Nm/_