a bgP0@sTddlZddlmZmZmZddlmZddlmZddl m Z GdddeZ dS)N)IteratorLiteralOptional) BaseLoader)Document) get_from_envc@steZdZdZeedddZeedddZddddd eeeeee d eed d d Z e e dddZ dS)FireCrawlLoadera! FireCrawlLoader document loader integration Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``. .. code-block:: bash pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key" Instantiate: .. code-block:: python from langchain_community.document_loaders import FireCrawlLoader loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} )paramsreturncCsd}gd}|D]}||rd}q(q|rtdtd|vr^|ddurX|d|d<|d=d|vr|ddur~|d|d<|d=d |vr|d dur|d |d <|d =d |vr|d dur|d |d <|d =d |vrt|d tr||d |d<|d =|S)NF)includesexcludesallowBackwardCrawlingallowExternalContentLinks pageOptionsTBDeprecated parameters detected. See Firecrawl v1 docs for updates.r Z includePathsr Z excludePathsr ZallowBackwardLinksrZallowExternalLinksrZ scrapeOptions)getwarningswarnDeprecationWarning isinstancedictlegacy_scrape_options_adapter)selfr use_legacy_optionsZ legacy_keyskeyr|/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/firecrawl.pylegacy_crawler_options_adapterCsD         z.FireCrawlLoader.legacy_crawler_options_adaptercCsd}dg}d|vrd|dvr|dddksN|dddksN|dddkrd}d |dvr|dd r||dd |d <n|dd d |d <d |dvr|dd r|dd |d <d|dvr|ddr|dd|d <|d=gd}|D]}||rd}q q|rtdtd|vrF|ddur@|d|d=d|vrn|ddurh|d|d=d|vr|ddur|d|d=d|vr|ddur|d|d=d|vr|ddur|d|d=d|vr|ddur|d|d=d|vr6|ddur0|d|d=d|vr`|ddurZ|d|d<|d=d|vr|ddur|d|d <|d=d!|vr||d!<|S)"NFmarkdownZextractorOptionsmodezllm-extractionzllm-extraction-from-raw-htmlzllm-extraction-from-markdownTZextractionPromptpromptz-Extract page information based on the schema.ZextractionSchemaZschemaZ userPrompt) includeMarkdown includeHtmlincludeRawHtmlincludeExtract includeLinks screenshotfullPageScreenshotonlyIncludeTags removeTagsrr!r"htmlr#rawHtmlr$extractr%linksr&r'zscreenshot@fullPager(Z includeTagsr)Z excludeTagsformats)rrrrremoveappend)rr rr.Z scrape_keysrrrrrss                               z-FireCrawlLoader.legacy_scrape_options_adapterNcrawl)api_keyapi_urlrr )r1scrapemap)urlr2r3rr cCszddlm}Wnty*tdYn0|dvrDtd|d|sPtd|p\tdd }|||d |_||_||_|p~i|_d S) aRInitialize with API key and url. Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py r) FirecrawlAppzD`firecrawl` package not found, please run `pip install firecrawl-py`)r1r4searchr5Invalid mode 'z/'. Allowed: 'crawl', 'scrape', 'search', 'map'.zUrl must be providedr2ZFIRECRAWL_API_KEY)r2r3N) firecrawlr7 ImportError ValueErrorrr6rr )rr6r2r3rr r7rrr__init__s"   zFireCrawlLoader.__init__)r ccs |jdkr(|jj|j||jdg}n|jdkrh|js@td|jj|j||jd}| dg}nT|jdkr|jstd|jj |j|jd}n&|jdkrtd ntd |jd |D]Z}|jdkr|}i}n,| d p| d p| dd}| di}|s qt ||dVqdS)Nr4)r r1zURL is required for crawl modedatar5zURL is required for map moder8z?Search mode is not supported in this version, please downgrade.r9z%'. Allowed: 'crawl', 'scrape', 'map'.rr*r+metadata) page_contentr@) rr:Z scrape_urlr6rr r<Z crawl_urlrrZmap_urlr)rZfirecrawl_docsZcrawl_responsedocrAr@rrr lazy_load sH       zFireCrawlLoader.lazy_load)__name__ __module__ __qualname____doc__rrrstrrrr=rrrCrrrrr s90p .r) rtypingrrrZlangchain_core.document_loadersrZlangchain_core.documentsrZlangchain_core.utilsrrrrrrs