a bg@sfddlZddlZddlmZddlmZmZmZddl m Z ddl m Z e eZGddde ZdS)N)Path)DictIteratorUnion)Document) BaseLoaderc@sReZdZdZd eeefeedfeedfeddddZe e ddd Z dS) BSHTMLLoaderaS __ModuleName__ document loader integration Setup: Install ``langchain-community`` and ``bs4``. .. code-block:: bash pip install -U langchain-community bs4 Instantiate: .. code-block:: python from langchain_community.document_loaders import BSHTMLLoader loader = BSHTMLLoader( file_path="./example_data/fake-content.html", ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Test Title My First Heading My first paragraph. {'source': './example_data/fake-content.html', 'title': 'Test Title'} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Test Title My First Heading My first paragraph. {'source': './example_data/fake-content.html', 'title': 'Test Title'} N) file_path open_encoding bs_kwargsget_text_separatorreturncCshz ddl}Wnty&tdYn0||_||_|durXtjdsPtdddi}||_||_dS)ainitialize with path, and optionally, file encoding to use, and any kwargs to pass to the BeautifulSoup object. Args: file_path: The path to the file to load. open_encoding: The encoding to use when opening the file. bs_kwargs: Any kwargs to pass to the BeautifulSoup object. get_text_separator: The separator to use when calling get_text on the soup. rNzUbeautifulsoup4 package not found, please install it with `pip install beautifulsoup4`ZlxmlzBy default BSHTMLLoader uses the 'lxml' package. Please either install it with `pip install -U lxml` or pass in init arg `bs_kwargs={'features': '...'}` to overwrite the default BeautifulSoup kwargs.features) bs4 ImportErrorr r importlibutil find_specr r )selfr r r r rrz/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/html_bs.py__init__Ss     zBSHTMLLoader.__init__)rccsddlm}t|jd|jd"}||fi|j}Wdn1sF0Y||j}|jrpt |jj }nd}t |j|d}t ||dVdS) z)Load HTML document into document objects.r) BeautifulSoupr)encodingNr )sourcetitle)Z page_contentmetadata) rropenr r r Zget_textr rstrstringr)rrfZsouptextrrrrr lazy_loadys 0 zBSHTMLLoader.lazy_load)NNr ) __name__ __module__ __qualname____doc__rr rdictrrrr$rrrrr sH    &r)importlib.utilrloggingpathlibrtypingrrrZlangchain_core.documentsrZ)langchain_community.document_loaders.baser getLoggerr%loggerrrrrrs