a bgؾ@s.ddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl mZmZmZmZmZmZmZmZmZmZddlmZddlZddlmZddlmZdd lm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1er*ddl2m3Z3e4e5Z6Gddde1Z7Gddde eZ8Gddde8Z9Gddde8Z:Gddde8Z;Gddde ZGd d!d!e8Z?Gd"d#d#e8Z@Gd$d%d%e8ZAGd&d'd'e8ZBGd(d)d)e$ZCGd*d+d+e8ZDGd,d-d-e8ZEe:ZFdS).N)ABC)StringIO)PathPurePath) TYPE_CHECKINGAnyBinaryIOIteratorLiteralMappingOptionalSequenceUnioncast)urlparse)Document)get_from_dict_or_env) BaseLoader)Blob)DedocBaseLoader)BaseImageBlobParser)_DEFAULT_PAGES_DELIMITERAmazonTextractPDFParserDocumentIntelligenceParserPDFMinerParserPDFPlumberParser PyMuPDFParserPyPDFium2Parser PyPDFParser)UnstructuredFileLoader)TextLinearizationConfigcsBeZdZdZd eeefeedfdd Ze dddZ Z S) UnstructuredPDFLoaderaLoad `PDF` files using `Unstructured`. You can run the loader in one of two modes: "single" and "elements". If you use "single" mode, the document will be returned as a single langchain Document object. If you use "elements" mode, the unstructured library will split the document into elements such as Title and NarrativeText. You can pass in additional unstructured kwargs after mode to apply different unstructured settings. Examples -------- from langchain_community.document_loaders import UnstructuredPDFLoader loader = UnstructuredPDFLoader( "example.pdf", mode="elements", strategy="fast", ) docs = loader.load() References ---------- https://unstructured-io.github.io/unstructured/bricks.html#partition-pdf single) file_pathmodeunstructured_kwargsc s$t|}tjf||d|dS)a% Args: file_path: The path to the PDF file to load. mode: The mode to use when loading the file. Can be one of "single", "multi", or "all". Default is "single". **unstructured_kwargs: Any kwargs to pass to the unstructured. )r#r$N)strsuper__init__)selfr#r$r% __class__v/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/pdf.pyr(JszUnstructuredPDFLoader.__init__returncCs"ddlm}|fd|ji|jS)Nr) partition_pdffilename)Zunstructured.partition.pdfr0r#r%)r)r0r,r,r- _get_elements[s z#UnstructuredPDFLoader._get_elements)r") __name__ __module__ __qualname____doc__rr&rrr(listr2 __classcell__r,r,r*r-r!2s r!c@seZdZdZddeeefeedddZ dddd Z e ee d d d Z e ee d d dZe ee d ddZeedddZdS) BasePDFLoaderzBase Loader class for `PDF` files. If the file is a web path, it will download it to a temporary file, use it, then clean up the temporary file after completion. Nheadersr#r;cCsLt||_d|_||_d|jvr0tj|j|_tj|js*||jr*t |_ tj |j\}}| |jrt|jjdd}tj|j jd|}|j|_||jsHtj|j|jd}|jdkrtd|jt|d d }||jWdn1s0Yt||_ntj|jsHtd |jdS) zInitialize with a file path. Args: file_path: Either a local, S3 or web path to a PDF file. headers: Headers to use for GET request to download a file from a web path. N~/tmpr:z3Check the url of your file; returned status code %swb)r$z'File path %s is not a valid file or url)r&r#web_pathr;ospath expanduserisfile _is_valid_urltempfileTemporaryDirectorytemp_dirsplitext_is_s3_presigned_urlrsplitjoinname _is_s3_urlrequestsget status_code ValueErroropenwritecontent)r)r#r;_suffixZtemp_pdfrfr,r,r-r(hs2     , zBasePDFLoader.__init__r.cCst|dr|jdS)NrK)hasattrrKcleanupr)r,r,r-__del__s zBasePDFLoader.__del__)urlr/cCst|}t|jot|jS)zCheck if the url is valid.)rboolnetlocscheme)raparsedr,r,r-rHszBasePDFLoader._is_valid_urlcCs>z$t|}|jdkr |jr WdSWdSty8YdS0dS)zcheck if the url is S3Zs3TFN)rrdrcrUraresultr,r,r-rQs zBasePDFLoader._is_s3_urlcCs6zt|}ttd|jWSty0YdS0dS)z'Check if the url is a presigned S3 url.z\.s3\.amazonaws\.com$FN)rrbresearchrcrUrfr,r,r-rMs  z"BasePDFLoader._is_s3_presigned_urlcCs|jdur|jS|jSN)rCr#r_r,r,r-sourceszBasePDFLoader.source)r3r4r5r6rr&rr dictr(r` staticmethodrbrHrQrMpropertyrkr,r,r,r-r9as & r9c@s"eZdZdZeedddZdS)OnlinePDFLoaderzLoad online `PDF`.r.cCstt|j}|S)zLoad documents.)r!r&r#load)r)loaderr,r,r-rpszOnlinePDFLoader.loadN)r3r4r5r6r7rrpr,r,r,r-rosrocseZdZdZddddedddeeefeeee fee e e dee e d ee d ee dd fd d ZeedddZZS) PyPDFLoaderaLoad and parse a PDF file using 'pypdf' library. This class provides methods to load and parse PDF documents, supporting various configurations such as handling password-protected files, extracting images, and defining extraction mode. It integrates the `pypdf` library for PDF processing and offers both synchronous and asynchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdf Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PyPDFLoader loader = PyPDFLoader( file_path = "./example_data/layout-parser-paper.pdf", # headers = None # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_parser = RapidOCRBlobParser(), ) Lazy load documents: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) NFpagetextplain)r$ images_parserimages_inner_formatpages_delimiterextraction_modeextraction_kwargsr"rsrtz markdown-imgzhtml-imgruZlayout) r#passwordr;extract_imagesr$rvrwrxryrzr/c s.tj||dt||||||| | d|_dS)u Initialize with a file path. Args: file_path: The path to the PDF file to be loaded. headers: Optional headers to use for GET request to download a file from a web path. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extraction_mode: “plain” for legacy functionality, “layout” extract text in a fixed width format that closely adheres to the rendered layout in the source pdf extraction_kwargs: Optional additional parameters for the extraction process. Returns: This method does not directly return data. Use the `load`, `lazy_load` or `aload` methods to retrieve parsed documents with content and metadata. r:)r~r$rrvrwrxryrzN)r'r(rparser) r)r#r~r;rr$rvrwrxryrzr*r,r-r(s+zPyPDFLoader.__init__r.ccsF|jr$tjt|jd|jd}n t|j}|j|EdHdSz Lazy load given path as pages. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. rbrEN rCr from_datarVr#read from_pathrZ lazy_parser)blobr,r,r- lazy_load%s  zPyPDFLoader.lazy_load)NNF)r3r4r5r6rrr&rr bytesrlrbr rr(r rrr8r,r,r*r-rrs46 9rrcsPeZdZdZdddeeefeee dfddZ e e dd d Z ZS) PyPDFium2Loaderz;Load `PDF` using `pypdfium2` and chunks at character level.NF)r;r)r#r;rcs tj||dt|d|_dS)Initialize with a file path.r:)rN)r'r(rr)r)r#r;rr*r,r-r(9szPyPDFium2Loader.__init__r.ccsF|jr$tjt|jd|jd}n t|j}|j|EdHdS)Lazy load given path as pages.rrN) rCrrrVr#rrrparserr,r,r-rDs zPyPDFium2Loader.lazy_load)r3r4r5r6rr&rr rlrbr(r rrr8r,r,r*r-r6s  rc@seZdZdZddddddddeeefeeeeeeee dee ee e d ee d d d Z e eed ddZeedddZdS)PyPDFDirectoryLoaderaLoad and parse a directory of PDF files using 'pypdf' library. This class provides methods to load and parse multiple PDF documents in a directory, supporting options for recursive search, handling password-protected files, extracting images, and defining extraction modes. It integrates the `pypdf` library for PDF processing and offers synchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdf Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PyPDFDirectoryLoader loader = PyPDFDirectoryLoader( path = "./example_data/", glob = "**/[!.]*.pdf", silent_errors = False, load_hidden = False, recursive = False, extract_images = False, password = None, mode = "page", images_to_text = None, headers = None, extraction_mode = "plain", # extraction_kwargs = None, ) Load documents: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) **/[!.]*.pdfFNrsru)r~r$rvr;ryrzr{r}) rEglob silent_errors load_hidden recursiverr~r$rvr;ryrzc CsL||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ dS)uInitialize with a directory path. Args: path: The path to the directory containing PDF files to be loaded. glob: The glob pattern to match files in the directory. silent_errors: Whether to log errors instead of raising them. load_hidden: Whether to include hidden files in the search. recursive: Whether to search subdirectories recursively. extract_images: Whether to extract images from PDFs. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for extracting the entire document or "page" for page-wise extraction. images_parser: Optional image blob parser.. headers: Optional headers to use for GET request to download a file from a web path. extraction_mode: “plain” for legacy functionality, “layout” for experimental layout mode functionality extraction_kwargs: Optional additional parameters for the extraction process. Returns: This method does not directly return data. Use the `load` method to retrieve parsed documents with content and metadata. N) r~r$rErrrrrrvr;ryrz) r)rErrrrrr~r$rvr;ryrzr,r,r-r(s(zPyPDFDirectoryLoader.__init__)rEr/cCstdd|jD S)Ncss|]}|dVqdS).N startswith).0partr,r,r- z3PyPDFDirectoryLoader._is_visible..)anypartsrr,r,r- _is_visiblesz PyPDFDirectoryLoader._is_visibler.c Cst|j}g}|jr ||jn ||j}|D]}|r0|||sR|jr0zXt t ||j |j |j |j|j|j|jd}|}|D]}t ||jd<q||Wq0ty}z"|jrt|n|WYd}~q0d}~00q0|S)N)r~r$rrvr;ryrzrk)rrErrglobris_filer relative_torrrr&r~r$rrvr;ryrzrpmetadataextend Exceptionrloggerwarning) r)pdocsitemsirqZsub_docsdocer,r,r-rps4   zPyPDFDirectoryLoader.load)rFFFF)r3r4r5r6rr&rrbr r rrlr(rmrr7rrpr,r,r,r-rOs<7   5rc s~eZdZdZddeddddddeeefeee dee ee e dee ee dd fd d Z eed d dZZS)PDFMinerLoaderaLoad and parse a PDF file using 'pdfminer.six' library. This class provides methods to load and parse PDF documents, supporting various configurations such as handling password-protected files, extracting images, and defining extraction mode. It integrates the `pdfminer.six` library for PDF processing and offers both synchronous and asynchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pdfminer.six Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PDFMinerLoader loader = PDFMinerLoader( file_path = "./example_data/layout-parser-paper.pdf", # headers = None # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_to_text = convert_images_to_text_with_tesseract(), ) Lazy load documents: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) Nr"Frt)r~r$rxrrvrwr;concatenate_pagesr{r|) r#r~r$rxrrvrwr;rr/c s,tj||dt|||| |||d|_dS)arInitialize with a file path. Args: file_path: The path to the PDF file to be loaded. headers: Optional headers to use for GET request to download a file from a web path. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) concatenate_pages: Deprecated. If True, concatenate all PDF pages into one a single document. Otherwise, return one document per page. Returns: This method does not directly return data. Use the `load`, `lazy_load` or `aload` methods to retrieve parsed documents with content and metadata. r:)r~rrvrr$rxrwN)r'r(rr) r)r#r~r$rxrrvrwr;rr*r,r-r(s'zPDFMinerLoader.__init__r.ccsF|jr$tjt|jd|jd}n t|j}|j|EdHdSrrrr,r,r-rAs  zPDFMinerLoader.lazy_load)r3r4r5r6rrr&rr r rbrrlr(r rrr8r,r,r*r-rs.7 4rcsLeZdZdZddeeefeedfddZ e e ddd Z Z S) PDFMinerPDFasHTMLLoaderz2Load `PDF` files as HTML content using `PDFMiner`.Nr:r<cs@zddlm}Wnty*tdYn0tj||ddS)rrextract_text_to_fpzO`pdfminer` package not found, please install it with `pip install pdfminer.six`r:N)pdfminer.high_levelr ImportErrorr'r()r)r#r;rr*r,r-r(Us  z PDFMinerPDFasHTMLLoader.__init__r.ccsddlm}ddlm}ddlm}t}||jd*}|tt ||d|ddWd n1sf0Yd |j d urt |jn|j i}t | |d Vd S) Load file.rr)LAParams) open_filenamerhtml)codecZlaparams output_typeNrkZ page_contentr)rrZpdfminer.layoutrZpdfminer.utilsrrr#rrrCr&rgetvalue)r)rrrZ output_stringfprr,r,r-rcs   $z!PDFMinerPDFasHTMLLoader.lazy_load)r3r4r5r6rr&rr rlr(r rrr8r,r,r*r-rRs rcseZdZdZddeddddddd eeefeee dee ee e dee d dfee ee ee fe dd fd d Ze eed ddZe eed ddZeedddZZS) PyMuPDFLoadera3Load and parse a PDF file using 'PyMuPDF' library. This class provides methods to load and parse PDF documents, supporting various configurations such as handling password-protected files, extracting tables, extracting images, and defining extraction mode. It integrates the `PyMuPDF` library for PDF processing and offers both synchronous and asynchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pymupdf Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PyMuPDFLoader loader = PyMuPDFLoader( file_path = "./example_data/layout-parser-paper.pdf", # headers = None # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_parser = TesseractBlobParser(), # extract_tables = "markdown", # extract_tables_settings = None, ) Lazy load documents: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) NrsFrt) r~r$rxrrvrwextract_tablesr;extract_tables_settingsr{r|)csvmarkdownr) r#r~r$rxrrvrwrr;rkwargsr/c s@|dvrtdtj|| dt|||| ||||| d |_dS)aInitialize with a file path. Args: file_path: The path to the PDF file to be loaded. headers: Optional headers to use for GET request to download a file from a web path. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extract_tables: Whether to extract tables in a specific format, such as "csv", "markdown", or "html". extract_tables_settings: Optional dictionary of settings for customizing table extraction. **kwargs: Additional keyword arguments for customizing text extraction behavior. Returns: This method does not directly return data. Use the `load`, `lazy_load`, or `aload` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the `mode` argument is not one of "single" or "page". r{zmode must be single or pager:) r~r$rx text_kwargsrrvrwrrN)rUr'r(rr) r)r#r~r$rxrrvrwrr;rrr*r,r-r(s0zPyMuPDFLoader.__init__)rr/cksd|rtd|d|j}|jr@tjt|jd|jd}n t |j}|j ||dEdHdS)zLazy load given path as pages or single document (see `mode`). Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. zReceived runtime arguments zd. Passing runtime args to `load` is deprecated. Please pass arguments during initialization instead.rr)rN) rrrrCrrrVr#rrZ _lazy_parse)r)rrrr,r,r- _lazy_loads  zPyMuPDFLoader._lazy_loadcKst|jfi|Srj)r7r)r)rr,r,r-rpszPyMuPDFLoader.loadr.ccs|EdHdSrj)rr_r,r,r-rszPyMuPDFLoader.lazy_load)r3r4r5r6rrr&rr r rbrrlrr(r rrr7rprr8r,r,r*r-rxs6: ?rc seZdZdZdeeefeeee e ee fe ddfdd Z e e eefd d d Ze ed d d Ze e d ddZed ddZeddddZeedddZeedddZeed ddZZS)MathpixPDFLoaderz)Load `PDF` files using `Mathpix` service.mdFN)r#processed_file_formatmax_wait_time_secondsshould_clean_pdfextra_request_datarr/c spt|dd|_t|dd|_|dd|ddtj|fi|||_|durZ|ni|_||_||_ dS)aInitialize with a file path. Args: file_path: a file for loading. processed_file_format: a format of the processed file. Default is "md". max_wait_time_seconds: a maximum time to wait for the response from the server. Default is 500. should_clean_pdf: a flag to clean the PDF file. Default is False. extra_request_data: Additional request data. **kwargs: additional keyword arguments. mathpix_api_keyZMATHPIX_API_KEYmathpix_api_idZMATHPIX_API_IDN) rrrpopr'r(rrrr)r)r#rrrrrr*r,r-r( s  zMathpixPDFLoader.__init__r.cCs|j|jdS)N)Zapp_idZapp_key)rrr_r,r,r-_mathpix_headers2sz!MathpixPDFLoader._mathpix_headerscCsdS)Nzhttps://api.mathpix.com/v3/pdfr,r_r,r,r-ra6szMathpixPDFLoader.urlcCs"d|jdii|j}dt|iS)NZconversion_formatsTZ options_json)rrjsondumps)r)optionsr,r,r-data:s  zMathpixPDFLoader.datacCstt|jd0}d|i}tj|j|j||jd}Wdn1sF0Y|}d|vrrt d|dd|vr|d}|St ddS)Nrfile)r;filesrerrorzMathpix request failed: pdf_idzUnable to send PDF to Mathpix.) rVr&r#rRpostrarrrrU)r)r\rresponse response_datarr,r,r-send_pdfBs$zMathpixPDFLoader.send_pdf)rr/c Cs|jd|}td|jdD]}tj||jd}|}|dd}|dd}|durd|}|dur~|d |d d 7}t||d d} | d krdS| dkrtdqt d| t dqt dS)zmWait for processing to complete. Args: pdf_id: a PDF id. Returns: None r>rr:rN error_infoz%Unable to retrieve PDF from Mathpix: z (id)status completedz#Unable to retrieve PDF from Mathpixz.Status: %s, waiting for processing to complete) rarangerrRrSrrrUrinfotimesleep TimeoutError) r)rrarYrrrr error_msgrr,r,r-wait_for_processingQs&       z$MathpixPDFLoader.wait_for_processingcCs>|||jd|d|j}tj||jd}|jdS)Nr>rr:zutf-8)rrarrRrSrrXdecode)r)rrarr,r,r-get_processed_pdfvs z"MathpixPDFLoader.get_processed_pdf)contentsr/cCsVddd|dD}|dddd}|dd d d d d dd}|S)ziClean the PDF file. Args: contents: a PDF file contents. Returns:  cSsg|]}|ds|qS)z![]r)rliner,r,r- rz.MathpixPDFLoader.clean_pdf..z \section{z# }rz\$$z\%%z\((z\)r)rOrNreplace)r)rr,r,r- clean_pdf|s  zMathpixPDFLoader.clean_pdfcCs@|}||}|jr"||}|j|j|d}t||dgS)N)rkr#rr)rrrrrkr)r)rrrr,r,r-rps   zMathpixPDFLoader.load)rrFN)r3r4r5r6rr&rintrbr rlrr(rnrrarrrrrr7rrpr8r,r,r*r-rs2 '%rcs\eZdZdZd eeefeeee fe ee e ddfdd Z e eddd ZZS) PDFPlumberLoaderz$Load `PDF` files using `pdfplumber`.NF)r#rdeduper;rr/csRz ddl}Wnty&tdYn0tj||d|p>i|_||_||_dS)rrNzMpdfplumber package not found, please install it with `pip install pdfplumber`r:) pdfplumberrr'r(rrr)r)r#rrr;rrr*r,r-r(s     zPDFPlumberLoader.__init__r.cCsNt|j|j|jd}|jr8tjt|jd |jd}n t |j}| |S)r)rrrrr) rrrrrCrrrVr#rrr)r)rrr,r,r-rps zPDFPlumberLoader.load)NFNF)r3r4r5r6rr&rr r rrbrlr(r7rrpr8r,r,r*r-rs rc seZdZdZdddeeefeeeee eeeeeeee eddd fddZ e e dd d Zee dd d Zeeed ddZZS)AmazonTextractPDFLoaderaLoad `PDF` files from a local file system, HTTP or S3. To authenticate, the AWS client uses the following methods to automatically load credentials: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html If a specific credential profile should be used, you must pass the name of the profile from the ~/.aws/credentials file that is to be used. Make sure the credentials / roles used have the required policies to access the Amazon Textract service. Example: .. code-block:: python from langchain_community.document_loaders import AmazonTextractPDFLoader loader = AmazonTextractPDFLoader( file_path="s3://pdfs/myfile.pdf" ) document = loader.load() N)linearization_configr ) r#textract_featuresclientcredentials_profile_name region_name endpoint_urlr;rr/c stj||dz ddlWnty6tdYn0|rPfdd|D} ng} |sb|sb|rzVddl} |dur| j|d} n| } i} |r|| d<|r|| d <| jdi| }WnNtytd Yn6ty} ztd | | WYd} ~ n d} ~ 00t | ||d |_ dS)aUInitialize the loader. Args: file_path: A file, url or s3 path for input file textract_features: Features to be used for extraction, each feature should be passed as a str that conforms to the enum `Textract_Features`, see `amazon-textract-caller` pkg client: boto3 textract client (Optional) credentials_profile_name: AWS profile name, if not default (Optional) region_name: AWS region, eg us-east-1 (Optional) endpoint_url: endpoint url for the textract service (Optional) linearization_config: Config to be used for linearization of the output should be an instance of TextLinearizationConfig from the `textractor` pkg r:rNztCould not import amazon-textract-caller python package. Please install it with `pip install amazon-textract-caller`.csg|]}j|qSr,)ZTextract_Features)rxZtcr,r-rrz4AmazonTextractPDFLoader.__init__..)Z profile_namerrtextractzRCould not import boto3 python package. Please install it with `pip install boto3`.zCould not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid. )rrr)r) r'r(Ztextractcallerrboto3SessionrrrUrr)r)r#rrrrrr;rfeaturesrsessionZ client_paramsrr*rr-r(sN    z AmazonTextractPDFLoader.__init__r.cCs t|SzLoad given path as pages.r7rr_r,r,r-rp&szAmazonTextractPDFLoader.loadccsb|jr ||jr t|jd}n,t|j}t|dkrLtd|jd|j |EdHdS)zLazy load documentsrz the file z is a multi-page document, but not stored on S3. Textract requires multi-page documents to be on S3.N) rCrQrrr#r_get_number_of_pagesrUrErrrr,r,r-r*s  z!AmazonTextractPDFLoader.lazy_load)rr/c Cszddl}ddlm}m}Wnty6tdYn0|jdkr|$}||}t|j WdS1st0Yn^|jdkrd}| | }t | |D]\}}|d7}q|S|jdvrdStd|jdS) Nr)Image ImageSequencezcCould not import pypdf or Pilloe python package. Please install it with `pip install pypdf Pillow`.zapplication/pdfz image/tiffr )z image/pngz image/jpegzunsupported mime type: )pypdfZPILr rrmimetypeZ as_bytes_ioZ PdfReaderlenpagesrVas_bytes enumerater rU) rrr rZinput_pdf_fileZ pdf_reader num_pagesZimgrYr,r,r-r ?s,     *    z,AmazonTextractPDFLoader._get_number_of_pages)NNNNNN)r3r4r5r6rr&rr r rrlr(r7rrpr rrmrrr r8r,r,r*r-rs2   L rc@seZdZdZedddZdS)DedocPDFLoadera DedocPDFLoader document loader integration to load PDF files using `dedoc`. The file loader can automatically detect the correctness of a textual layer in the PDF document. Note that `__init__` method supports parameters that differ from ones of DedocBaseLoader. Setup: Install ``dedoc`` package. .. code-block:: bash pip install -U dedoc Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocPDFLoader loader = DedocPDFLoader( file_path="example.pdf", # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/pdf_handling.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing is_one_column_document: detect number of columns for PDF without a textual layer, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result need_binarization: clean pages background (binarize) for PDF without a textual layer need_pdf_table_analysis: parse tables for PDF without a textual layer r.cCs$ddlm}|t|j|j|jdS)Nr)make_manager_pdf_config)r#Zparsing_paramsrN)Zdedoc.utils.langchainrr&r#Zparsing_parametersrN)r)rr,r,r- _make_configs  zDedocPDFLoader._make_configN)r3r4r5r6rlrr,r,r,r-r\sVrcs`eZdZdZd eeefeeee ddfdd Z e e ddd Z ee dd d ZZS) DocumentIntelligenceLoaderz+Load a PDF with Azure Document Intelligenceprebuilt-documentN)r#rmodelr;r/cs"tj||dt||d|_dS)aInitialize the object for file processing with Azure Document Intelligence (formerly Form Recognizer). This constructor initializes a DocumentIntelligenceParser object to be used for parsing files using the Azure Document Intelligence API. The load method generates a Document node including metadata (source blob and page number) for each page. Parameters: ----------- file_path : str The path to the file that needs to be parsed. client: Any A DocumentAnalysisClient to perform the analysis of the blob model : str The model name or ID to be used for form recognition in Azure. Examples: --------- >>> obj = DocumentIntelligenceLoader( ... file_path="path/to/file", ... client=client, ... model="prebuilt-document" ... ) r:)rrN)r'r(rr)r)r#rrr;r*r,r-r(s!z#DocumentIntelligenceLoader.__init__r.cCs t|Sr r r_r,r,r-rpszDocumentIntelligenceLoader.loadccs"t|j}|j|EdHdS)rN)rrr#rrrr,r,r-rs z$DocumentIntelligenceLoader.lazy_load)rN)r3r4r5r6rr&rrr rlr(r7rrpr rr8r,r,r*r-rs $rcsHeZdZdZd eeefeeddfdd Ze e ddd Z Z S) ZeroxPDFLoaderaDocument loader utilizing Zerox library: https://github.com/getomni-ai/zerox Zerox converts PDF document to series of images (page-wise) and uses vision-capable LLM model to generate Markdown representation. Zerox utilizes anyc operations. Therefore when using this loader inside Jupyter Notebook (or any environment running async) you will need to: ```python import nest_asyncio nest_asyncio.apply() ``` gpt-4o-miniN)r#r zerox_kwargsr/c stj|d||_||_dS)N)r#)r'r(rr)r)r#rrr*r,r-r(szZeroxPDFLoader.__init__r.ccs~ddl}ddlm}||ft|j|jd|j}t|j dkrz|j dj }|j D] }t |j |j |j |ddVqXdS)zLazily load pages.rN)zerox)r#rr?)rkrsrr)asyncioZpyzeroxrrunr&r#rrrrrsrrXrk)r)r rZ zerox_outputrrsr,r,r-rs   zZeroxPDFLoader.lazy_load)r) r3r4r5r6rr&rrr(r rrr8r,r,r*r-rs r)GrloggingrDrhrIrabcriorpathlibrrtypingrrrr r r r r rr urllib.parserrRZlangchain_core.documentsrZlangchain_core.utilsrZ)langchain_community.document_loaders.baserZ1langchain_community.document_loaders.blob_loadersrZ*langchain_community.document_loaders.dedocrZ3langchain_community.document_loaders.parsers.imagesrZ0langchain_community.document_loaders.parsers.pdfrrrrrrrrZ1langchain_community.document_loaders.unstructuredrZ)textractor.data.text_linearization_configr getLogger__file__rr!r9rorrrrrrrrrrrrrZPagedPDFSplitterr,r,r,r-sT  0       (   /P | w&(a3F