a bg)<@sdZddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlZddlmZddlmZddlmZddlmZeeZedd d d d d d ZeeedddZGdddeZdS)zWeb base loader class.N)Any AsyncIteratorDictIteratorListOptionalSequenceUnion) deprecated)Document) BaseLoader)get_user_agentzJtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8zen-US,en;q=0.5zhttps://www.google.com/1z keep-alive) User-AgentAcceptzAccept-LanguageZRefererZDNT ConnectionzUpgrade-Insecure-Requests)soupurlreturncCsjd|i}|d}r"||d<|jdddid}rH|dd|d<|d }rf|d d |d <|S) z)Build metadata from BeautifulSoup output.sourcetitlemetaname description)attrscontentzNo description found.htmllangzNo language found.language)findget_textget)rrmetadatarrrr#{/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/web_base.py_build_metadatas r%c@seZdZdZd5ddd eeeefeee eee e eeeee eee ee fe ee ee fee ee fe e e dd d d Z eed ddZd6ee e eedddZeejedddZeee dddZeeddddZd7e eeeedfee ddd Zd8eeeedfee d!d"d#Zd9eeeedfee d!d$d%Zd:eeedfeee d&d'd(Zd;eedfe dd)d*Zeed d+d,Z e!ed d-d.Z"e#d/d0d1d2eed d3d4Z$dS)< WebBaseLoaderaQ WebBaseLoader document loader integration Setup: Install ``langchain_community``. .. code-block:: bash pip install -U langchain_community Instantiate: .. code-block:: python from langchain_community.document_loaders import WebBaseLoader loader = WebBaseLoader( web_path = "https://www.espn.com/" # header_template = None, # verify_ssl = True, # proxies = None, # continue_on_failure = False, # autoset_encoding = True, # encoding = None, # web_paths = (), # requests_per_second = 2, # default_parser = "html.parser", # requests_kwargs = None, # raise_for_status = False, # bs_get_text_kwargs = None, # bs_kwargs = None, # session = None, # show_progress = True, # trust_env = False, ) Lazy load: .. code-block:: python docs = [] for doc in loader.lazy_load(): docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python ESPN - Serving Sports Fans. Anytime. Anywhere. {'source': 'https://www.espn.com/', 'title': 'ESPN - Serving Sports Fans. Anytime. Anywhere.', 'description': 'Visit ESPN for live scores, highlights and sports news. Stream exclusive games on ESPN+ and play fantasy sports.', 'language': 'en'} Async load: .. code-block:: python docs = [] async for doc in loader.alazy_load(): docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python ESPN - Serving Sports Fans. Anytime. Anywhere. {'source': 'https://www.espn.com/', 'title': 'ESPN - Serving Sports Fans. Anytime. Anywhere.', 'description': 'Visit ESPN for live scores, highlights and sports news. Stream exclusive games on ESPN+ and play fantasy sports.', 'language': 'en'} .. versionchanged:: 0.3.14 Deprecated ``aload`` (which was not async) and implemented a native async ``alazy_load``. Expand below for more details. .. dropdown:: How to update ``aload`` Instead of using ``aload``, you can use ``load`` for synchronous loading or ``alazy_load`` for asynchronous lazy loading. Example using ``load`` (synchronous): .. code-block:: python docs: List[Document] = loader.load() Example using ``alazy_load`` (asynchronous): .. code-block:: python docs: List[Document] = [] async for doc in loader.alazy_load(): docs.append(doc) This is in preparation for accommodating an asynchronous ``aload`` in the future: .. code-block:: python docs: List[Document] = await loader.aload() NTFr# html.parser) show_progress trust_env)web_pathheader_template verify_sslproxiescontinue_on_failureautoset_encodingencoding web_pathsrequests_per_seconddefault_parserrequests_kwargsraise_for_statusbs_get_text_kwargs bs_kwargssessionr*r+rcCsJ|r|rtd|r t||_nHt|tr4|g|_n4t|trJt||_ntdt|dt|d| |_| |_ | pzi|_ | |_ ||_ | pi|_ |pi|_|r||_nt}|pt}|dszddlm}|j|d<WntytdYn0t||_||_|r(|j|||_||_ ||_!||_"||_#d S) aInitialize loader. Args: web_paths: Web paths to load from. requests_per_second: Max number of concurrent requests to make. default_parser: Default parser to use for BeautifulSoup. requests_kwargs: kwargs for requests raise_for_status: Raise an exception if http status code denotes an error. bs_get_text_kwargs: kwargs for beatifulsoup4 get_text bs_kwargs: kwargs for beatifulsoup4 web page parsing show_progress: Show progress bar when loading pages. trust_env: set to True if using proxy to make web requests, for example using http(s)_proxy environment variables. Defaults to False. zmReceived web_path and web_paths. Only one can be specified. web_path is deprecated, web_paths should be used.z+web_path must be str or Sequence[str] got (z*) or web_paths must be Sequence[str] got ()rr) UserAgentzxfake_useragent not found, using default user agent.To get a realistic header for requests, `pip install fake_useragent`.N)$ ValueErrorlistr3 isinstancestrr TypeErrortyper4r5r6r7r*r8r9r:requestsSessiondefault_header_templatecopyr!Zfake_useragentr<random ImportErrorloggerinfodictheadersverifyr/updater0r1r2r+)selfr,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r*r+r<r#r#r$__init__sV$               zWebBaseLoader.__init__)rcCs t|jdkrtd|jdS)NzMultiple webpaths found.r)lenr3r=)rOr#r#r$r,szWebBaseLoader.web_path?)rretriescooldownbackoffrc stj|jd4IdHR}t|D].}zt|jj|jjd}|jj sRd|d<|j |fi|j |B4IdHL}|j r| | IdHWdIdHWWdIdHS1IdHs0YWq"tjyP} z^||dkrn@td|d|dd|d | d t|||IdHWYd} ~ q"d} ~ 00q"WdIdHq1IdHsx0Ytd dS) N)r+)rLcookiesFsslrQError fetching z with attempt /z: z . Retrying...zretry count exceeded)aiohttpZ ClientSessionr+rangerKr:rLrXget_dictrMr!r6r7textZClientConnectionErrorrIwarningasynciosleepr=) rOrrUrVrWr:ikwargsresponseer#r#r$_fetchs@ V^zWebBaseLoader._fetch)r semaphorerc s|4IdHz&||IdHWWdIdHSty}zZ|jr|td|dWYd}~WdIdHdStd|d|WYd}~n d}~00WdIdHq1IdHs0YdS)NrZz*, skipping due to continue_on_failure=Truer'za and aborting, use continue_on_failure=True to continue loading urls after encountering an error.)rg Exceptionr0rIr` exception)rOrrhrfr#r#r$_fetch_with_rate_limits&   z$WebBaseLoader._fetch_with_rate_limit)urlsrcst|j}g}|D] }t|||}||qzD|jrfddlm}|j |ddddIdHWStj |IdHWSWn,t yt dtj |IdHYS0dS) z/Fetch all urls concurrently with rate limiting.r) tqdm_asynciozFetching pagesTrQ)descasciiZ minintervalNz2For better logging of progress, `pip install tqdm`) ra Semaphorer4 ensure_futurerkappendr*Z tqdm.asynciormgatherrHwarningswarn)rOrlrhtasksrtaskrmr#r#r$ fetch_alls     zWebBaseLoader.fetch_all)parserrcCs*gd}||vr&tdd|ddS)z#Check that parser is valid for bs4.)r)Zlxmlxmlzlxml-xmlZhtml5libz`parser` must be one of z, .N)r=join)ryZ valid_parsersr#r#r$ _check_parser.s zWebBaseLoader._check_parser)resultsrlryrc Cspddlm}g}t|D]R\}}||}|durP|dr@d}n|j}||||||fi|jq|S)z0Unpack fetch results into BeautifulSoup objects.r BeautifulSoupN.xmlrz)bs4r enumerateendswithr5r}rrr9) rOr~rlryrZ final_resultsrcresultrr#r#r$_unpack_fetch_results7s   z#WebBaseLoader._unpack_fetch_results)rlryrcCs t||}|j|||dS)z2Fetch all urls, then return soups for all results.ry)rarunrxrrOrlryr~r#r#r$ scrape_allIszWebBaseLoader.scrape_allcs ||IdH}|j|||dS)z8Async fetch all urls, then return soups for all results.Nr)rxrrr#r#r$ ascrape_allNszWebBaseLoader.ascrape_all)rryr9rcCsddlm}|dur*|dr$d}n|j}|||jj|fi|j}|jrX||j durl|j |_ n|j rz|j |_ ||j |fi|piS)Nrrrrz) rrrr5r}r:r!r6r7r2r1apparent_encodingr_)rOrryr9rZhtml_docr#r#r$_scrapeUs     zWebBaseLoader._scrapecCs"|dur|j}|j|j||jdS)z?Scrape data from webpage and return it in BeautifulSoup format.N)ryr9)r5rr,r9)rOryr#r#r$scrapeoszWebBaseLoader.scrapeccsJ|jD]>}|j||jd}|jfi|j}t||}t||dVqdS)z+Lazy load text from the url(s) in web_path.)r9Z page_contentr"N)r3rr9r r8r%r )rOpathrr_r"r#r#r$ lazy_loadws   zWebBaseLoader.lazy_loadcCsV||jIdH}t|j|D]2\}}|jfi|j}t||}t||dVqdS)z1Async lazy load text from the url(s) in web_path.Nr)rr3zipr r8r%r )rOr~rrr_r"r#r#r$ alazy_loads  zWebBaseLoader.alazy_loadz0.3.14z1.0zSee API reference for updated usage: https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html)ZsinceZremovalmessagecCsX||j}g}t|j|D]6\}}|jfi|j}t||}|t||dq|S)z9Load text from the urls in web_path async into Documents.r)rr3rr r8r%rrr )rOr~docsrrr_r"r#r#r$aloads  zWebBaseLoader.aload)r'NTNFTNr#r(r)NFNNN)rSr(rT)N)N)N)NN)N)%__name__ __module__ __qualname____doc__r r@rrrKboolintrrrPpropertyr,floatrgrarprkrrx staticmethodr}rrrrrrr rrrr rr#r#r#r$r&*se U    $   r&)rraloggingrttypingrrrrrrrr r\rCZlangchain_core._apir Zlangchain_core.documentsr Z)langchain_community.document_loaders.baser Z$langchain_community.utils.user_agentr getLoggerrrIrEr@rKr%r&r#r#r#r$s*(