a bgL!@sdZddlZddlmZmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZerddlmZddlmZdd lmZdd lmZmZmZeeZGd d d eZGd ddeZGdddeZdS)zQLoader that uses Playwright to load a page, then uses unstructured to parse html.N)ABCabstractmethod) TYPE_CHECKING AsyncIteratorDictIteratorListOptional)Document) BaseLoader)Browser)Page)Response)r r rc@s@eZdZdZedddedddZedd d edd d Zd S)PlaywrightEvaluatorzAbstract base class for all evaluators. Each evaluator should take a page, a browser instance, and a response object, process the page as necessary, and return the resulting text. r r rpagebrowserresponsereturncCsdS)aSynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nselfrrrrr/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/url_playwright.pyevaluates zPlaywrightEvaluator.evaluate AsyncPage AsyncBrowser AsyncResponsecsdS)aAsynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nrrrrrevaluate_async*sz"PlaywrightEvaluator.evaluate_asyncN)__name__ __module__ __qualname____doc__rstrrrrrrrrs rc@sPeZdZdZdeeedddZddded d d Zd d ded ddZ dS)UnstructuredHtmlEvaluatorz@Evaluate the page HTML content using the `unstructured` library.N)remove_selectorscCs2z ddl}Wnty&tdYn0||_dS)z%Initialize UnstructuredHtmlEvaluator.rNzQunstructured package not found, please install it with `pip install unstructured`) unstructured ImportErrorr$)rr$r%rrr__init__>s   z"UnstructuredHtmlEvaluator.__init__r r rrc Cslddlm}|jpgD].}||}|D]}|r,|dq,q|}||d}ddd|DS)z3Synchronously process the HTML content of the page.rpartition_htmlelement => element.remove()text cSsg|] }t|qSrr".0elrrr Vz6UnstructuredHtmlEvaluator.evaluate.. Zunstructured.partition.htmlr)r$locatorallZ is_visiblercontentjoin rrrrr)selectorelementselementZ page_sourcerrrrJs  z"UnstructuredHtmlEvaluator.evaluaterrrc sddlm}|jpgD]@}||IdH}|D]"}|IdHr2|dIdHq2q|IdH}||d}ddd|DS) z4Asynchronously process the HTML content of the page.rr(Nr*r+r-cSsg|] }t|qSrr.r/rrrr2fr3z.r4r9rrrrXs  z(UnstructuredHtmlEvaluator.evaluate_async)N) rrr r!r rr"r'rrrrrrr#;s  r#c @s|eZdZdZdeeeeeeeeeee eefdddZ e e ddd Z ee dd d Zee dd d ZdS)PlaywrightURLLoaderaLoad `HTML` pages with `Playwright` and parse with `Unstructured`. This is useful for loading pages that require javascript to render. Attributes: urls (List[str]): List of URLs to load. continue_on_failure (bool): If True, continue loading other URLs on failure. headless (bool): If True, the browser will run in headless mode. proxy (Optional[Dict[str, str]]): If set, the browser will access URLs through the specified proxy. Example: .. code-block:: python from langchain_community.document_loaders import PlaywrightURLLoader urls = ["https://api.ipify.org/?format=json",] proxy={ "server": "https://xx.xx.xx:15818", # https://: "username": "username", "password": "password" } loader = PlaywrightURLLoader(urls, proxy=proxy) data = loader.load() TN)urlscontinue_on_failureheadlessr$ evaluatorproxycCsbz ddl}Wnty&tdYn0||_||_||_||_|rP|rPtd|pZt||_dS)z%Load a list of URLs using Playwright.rNzMplaywright package not found, please install it with `pip install playwright`z:`remove_selectors` and `evaluator` cannot be both not None) playwrightr&r>r?r@rB ValueErrorr#rA)rr>r?r@r$rArBrCrrrr's    zPlaywrightURLLoader.__init__)rc csddlm}|}|jj|j|jd}|jD]}zR|}||}|dur\t d||j |||}d|i}t ||dVWq.t y} z.|jrtd|d | n| WYd} ~ q.d} ~ 00q.|Wdn1s0YdS) zLoad the specified URLs using Playwright and create Document instances. Returns: A list of Document instances with loaded content. r)sync_playwrightr@rBN"page.goto() returned None for url sourceZ page_contentmetadataError fetching or processing , exception: )playwright.sync_apirEchromiumlaunchr@rBr>new_pagegotorDrArr Exceptionr?loggererrorclose) rrEprurlrrr,rJerrr lazy_loads&   zPlaywrightURLLoader.lazy_loadcsdd|2IdHS)Load the specified URLs with Playwright and create Documents asynchronously. Use this function when in a jupyter notebook environment. Returns: A list of Document instances with loaded content. csg|z3dHW}|q6S)Nr)r0docrrrr2r3z-PlaywrightURLLoader.aload..N) alazy_load)rrrraloadszPlaywrightURLLoader.aloadc Cs.ddlm}|4IdH}|jj|j|jdIdH}|jD]}zd|IdH}||IdH}|durvt d||j |||IdH}d|i}t ||dVWq<t y} z.|jrtd|d | n| WYd} ~ qrPrQrDrArr rRr?rSrTrU) rr^rVrrWrrr,rJrXrrrr\s&  zPlaywrightURLLoader.alazy_load)TTNNN)rrr r!rr"boolr rrr'rr rYr]rr\rrrrr=is"   r=)r!loggingabcrrtypingrrrrrr Zlangchain_core.documentsr Z)langchain_community.document_loaders.baser r_r rr rrrrM getLoggerrrSrr#r=rrrrs       &.