a bgX @sddlmZddlZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZddlmZdd lmZe eefZed ZGd d d eZdS) ) annotationsN)Path)IteratorLiteralOptionalSequenceUnion)Document)BaseBlobParser) BlobLoaderFileSystemBlobLoader) GenericLoader) get_parserdefaultc speZdZdZddddddfdd Zd d d d Zedddddddddddddddddd ddZZS)ConcurrentLoaderz%Load and pars Documents concurrently.r r intNone) blob_loader blob_parser num_workersreturncst||||_dS)N)super__init__r)selfrrr __class__}/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/concurrent.pyrszConcurrentLoader.__init__zIterator[Document])rc#sntjjjdJfddjD}tj|D]}|EdHq8Wdn1s`0YdS)z.Load documents lazily with concurrent parsing.) max_workerscsh|]}jj|qSr)submitrZ lazy_parse).0Zblobexecutorrrr )sz-ConcurrentLoader.lazy_load..N) concurrentfuturesThreadPoolExecutorrrZ yield_blobs as_completedresult)rr&futurerr"r lazy_load"s zConcurrentLoader.lazy_loadz**/[!.]*rNFr)globexcludesuffixes show_progressparserr parser_kwargs _PathLikestrz Sequence[str]zOptional[Sequence[str]]boolzUnion[DEFAULT, BaseBlobParser]zOptional[dict]) pathr,r-r.r/r0rr1rc Csbt|||||d} t|trP|dkrF|jtjkrF|jfi|p>i} qTt|} n|} || | |dS)aCreate a concurrent generic document loader using a filesystem blob loader. Args: path: The path to the directory to load documents from. glob: The glob pattern to use to find documents. suffixes: The suffixes to use to filter documents. If None, all files matching the glob will be loaded. exclude: A list of patterns to exclude from the loader. show_progress: Whether to show a progress bar or not (requires tqdm). Proxies to the file system loader. parser: A blob parser which knows how to parse blobs into documents num_workers: Max number of concurrent workers to use. parser_kwargs: Keyword arguments to pass to the parser. )r,r-r.r/r)r)r isinstancer3rr ) clsr5r,r-r.r/r0rr1rrrrrfrom_filesystem0s  z ConcurrentLoader.from_filesystem)r) __name__ __module__ __qualname____doc__rr+ classmethodr8 __classcell__rrrrrs r) __future__rconcurrent.futuresr%pathlibrtypingrrrrrZlangchain_core.documentsr Z)langchain_community.document_loaders.baser Z1langchain_community.document_loaders.blob_loadersr r Z,langchain_community.document_loaders.genericr Z5langchain_community.document_loaders.parsers.registryrr3r2DEFAULTrrrrrs