a bg) @sLddlmZmZmZddlmZddlmZddlm Z GdddeZ dS))IteratorLiteralOptional) BaseLoader)Document) get_from_envc@sLeZdZdZddddeeeedeedddZe e d d d Z dS) SpiderLoaderzLoad web pages as Documents using Spider AI. Must have the Python package `spider-client` installed and a Spider API key. See https://spider.cloud for more. Nscrape)api_keymodeparamsr crawl)urlr r r cCs|durddd}zddlm}Wnty<tdYn0|dvrVtd |d |pbtd d }||d |_||_||_||_dS)aInitialize with API key and URL. Args: url: The URL to be processed. api_key: The Spider API key. If not specified, will be read from env var `SPIDER_API_KEY`. mode: The mode to run the loader in. Default is "scrape". Options include "scrape" (single page) and "crawl" (with deeper crawling following subpages). params: Additional parameters for the Spider API. NmarkdownT)Z return_formatmetadatar)SpiderzB`spider` package not found, please run `pip install spider-client`r zUnrecognized mode 'z%'. Expected one of 'scrape', 'crawl'.r ZSPIDER_API_KEY)r )spiderr ImportError ValueErrorrrr r )selfrr r r rry/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/spider.py__init__s&    zSpiderLoader.__init__)returnccsg}|jdkr2|jj|j|jd}|r^||n,|jdkr^|jj|j|jd}|r^|||D]|}|jdkr|ddd}|ddi}|durt ||d V|jdkrb|dd}|di}|durbt ||d VqbdS) z+Load documents based on the specified mode.r )r rrcontentrN) page_contentr) r rZ scrape_urlrr appendZ crawl_urlextendgetr)rZ spider_docsresponsedocrrrrr lazy_load:s.        zSpiderLoader.lazy_load) __name__ __module__ __qualname____doc__strrrdictrrrr#rrrrrs  +rN) typingrrrZlangchain_core.document_loadersrZlangchain_core.documentsrZlangchain_core.utilsrrrrrrs