a bgQ,@sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZeeZ Gdd d eZ!Gd d d eZ"dS) z:Pebblo's safe dataloader is a wrapper for document loadersN)version)AnyDictIterableIteratorListOptional)Document) BaseLoader) BATCH_SIZE_BYTESPLUGIN_VERSIONApp FrameworkIndexedDocumentPebbloLoaderAPIWrappergenerate_size_based_batches get_full_pathget_loader_full_pathget_loader_type get_runtimeget_source_sizec @seZdZUdZdZeed<d"dddeeeee eee eeed d d Z e e d d d Z dd ddZee d ddZedd ddZed ddZe ed ddZee e dddZe e d ddZe ee dddZeddd d!ZdS)#PebbloSafeLoaderzkPebblo Safe Loader class is a wrapper around document loaders enabling the data to be scrutinized. F_discover_sentNlocal)classifier_locationanonymize_snippets) langchain_loadernameowner descriptionapi_key load_semanticclassifier_urlrrc Cs|rt|tstd||_tt|_||_tj dp>||_ ||_ ||_ t|j|_g|_g|_tt|jdddd} t| |_t|j|_t|_| |j|jd|jdkrdt|jini|_||_t|||| d |_|j|jdS) NzMust specify a valid name.ZPEBBLO_LOAD_SEMANTIC.'r)loader source_path source_typesource_path_size)r!rr#r) isinstancestr NameErrorapp_nameuuiduuid4load_idr'osenvirongetr"rr rr(docs docs_with_idtypesplitrr)rr*r batch_sizeloader_details_get_app_detailsappr pb_clientZsend_loader_discover) selfrrrr r!r"r#rrZ loader_namer?y/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/pebblo.py__init__%s>  "   zPebbloSafeLoader.__init__returncCs|j|_||jS)zxLoad Documents. Returns: list: Documents fetched from load method of the wrapped `loader`. )r'loadr5classify_in_batches)r>r?r?r@rDVs zPebbloSafeLoader.loadc Cst|j|j}g}t|}t|D]l\}}||dk}||_||_|jj|j|j |j |d}| ||j r|| |}n|}||q"||_dS)z Classify documents in batches. This is to avoid API timeouts when sending large number of documents. Batches are generated based on the page_content size. )Z loading_endN)rr5r9len enumerate _index_docsr6r=classify_documentsr<r:_add_pebblo_specific_metadatar"_add_semantic_to_docs _unindex_docsextend) r>ZbatchesZprocessed_docsZ total_batchesibatchZ is_last_batchclassified_docsZbatch_processed_docsr?r?r@rEas*     z$PebbloSafeLoader.classify_in_batchesc csz|j}WnHtyV}z0|jjjd}t|t||WYd}~n d}~00z t|}Wntyg|_ YqYn0t |f|_ | |_ |j |j |j|j}|||jr|||_ n ||_ |j dVqXdS)zLoad documents in lazy fashion. Raises: NotImplementedError: raised when lazy_load id not implemented within wrapped loader. Yields: list: Documents from loader's lazy loading. z does not implement lazy_load()Nr)r' lazy_loadNotImplementedError __class____name__loggererrornext StopIterationr5listrIr6r=rJr<r:rKr"rLrM)r>Z doc_iteratorexcZerr_strdocclassified_docr?r?r@rRs*           zPebbloSafeLoader.lazy_loadcCs d|_dS)NT)r)clsr?r?r@set_discover_sentsz"PebbloSafeLoader.set_discover_sentc Cs:t\}}t|j|j|j|j||ttdtddd}|S)z\Fetch app details. Internal method. Returns: App: App details. Zlangchain_community)rr)rrr r1runtime frameworkZplugin_versionclient_version) rr r.rr r1r rr)r>rar`r<r?r?r@r;s  z!PebbloSafeLoader._get_app_detailscCsddt|jD}|S)z Indexes the documents and returns a list of IndexedDocument objects. Returns: List[IndexedDocument]: A list of IndexedDocument objects with unique IDs. cSs*g|]"\}}tfdt|i|qS)pb_id)rr,dict.0rOr\r?r?r@ sz0PebbloSafeLoader._index_docs..)rHr5)r>r6r?r?r@rIszPebbloSafeLoader._index_docs)rQrCcCsVdd|jD}|D]&}|d}||vr||||qdd|D}|S)aF Adds semantic metadata to the given list of documents. Args: classified_docs (Dict): A dictionary of dictionaries containing the classified documents with pb_id as key. Returns: List[Document]: A list of Document objects with added semantic metadata. cSs i|]}|jt|j|jdqS) page_contentmetadata)rcr rirjrfr\r?r?r@ sz:PebbloSafeLoader._add_semantic_to_docs..rccSsg|]}|qSr?r?rkr?r?r@rgz:PebbloSafeLoader._add_semantic_to_docs..)r6valuesr4_add_semantic_to_doc)r>rQZ indexed_docsr]Zdoc_idZsemantic_metadata_docsr?r?r@rLs   z&PebbloSafeLoader._add_semantic_to_docscCsddt|jD}|S)z Converts a list of IndexedDocument objects to a list of Document objects. Returns: List[Document]: A list of Document objects. cSs g|]\}}t|j|jdqSrh)r rirjrer?r?r@rgsz2PebbloSafeLoader._unindex_docs..)rHr6)r>r5r?r?r@rMszPebbloSafeLoader._unindex_docs)r\r]rCcCs8t|di|jd<t|di|jd<|S)a4 Adds semantic metadata to the given document in-place. Args: doc (Document): A Document object. classified_doc (dict): A dictionary containing the classified document. Returns: Document: The Document object with added semantic metadata. entitiesZpebblo_semantic_entitiesZtopicsZpebblo_semantic_topics)rZr4keysrj)r>r\r]r?r?r@ros   z%PebbloSafeLoader._add_semantic_to_docc Cst|jD]h}|j}|jjjdkr6t|d|j|d<nt|d|d|j|d<||jidd|d<qdS)z*Add Pebblo specific metadata to documents.ZSharePointLoadersource full_pathZ pb_checksumN) r6rjr'rTrUrr4r(rc)r>rQr\Z doc_metadatar?r?r@rKs  z.PebbloSafeLoader._add_pebblo_specific_metadata)rrNFN)rU __module__ __qualname____doc__rbool__annotations__r r,rrArr rDrErrR classmethodr_r r;rrIrrLrMrdrorKr?r?r?r@rs@    1  "  rc @seZdZdZdddddeeeeeeeeeee feeeee fddddZ e e ddd Z ee dd d ZdS) PebbloTextLoaderz Loader for text data. Since PebbloSafeLoader is a wrapper around document loaders, this loader is used to load text data directly into Documents. N)rridsrj metadatas)textsrrr{rjr|rCcCs"||_||_||_||_||_dS)a Args: texts: Iterable of text data. source: Source of the text data. Optional. Defaults to None. ids: List of unique identifiers for each text. Optional. Defaults to None. metadata: Metadata for all texts. Optional. Defaults to None. metadatas: List of metadata for each text. Optional. Defaults to None. N)r}rrr{rjr|)r>r}rrr{rjr|r?r?r@rAs zPebbloTextLoader.__init__rBccst|jD]r\}}d}|jpi}|jrN|t|jkrN|j|rN||j||jrl|t|jkrl|j|}t|||dVq dS)zi Lazy load text data into Documents. Returns: Iterator of Documents N)idrirj)rHr}rjr|rGupdater{r )r>rOtextZ_idrjr?r?r@rR9s  zPebbloTextLoader.lazy_loadcCs g}|D]}||q |S)z` Load text data into Documents. Returns: List of Documents )rRappend)r>Z documentsr\r?r?r@rDIs  zPebbloTextLoader.load)rUrtrurvrr,rrrrrArr rRrDr?r?r?r@rzs   rz)#rvloggingr2r/importlib.metadatartypingrrrrrrZlangchain_core.documentsr Z)langchain_community.document_loaders.baser Z$langchain_community.utilities.pebblor r r rrrrrrrrr getLoggerrUrVrrzr?r?r?r@s    8 y