a bg@s<ddlmZmZddlmZddlmZGdddeZdS))AnyList)Document) WebBaseLoaderc@sJeZdZdZeedddZeeedddZeeedd d Z d S) HNLoaderz_Load `Hacker News` data. It loads data from either main page results or the comments page.)returncCs*|}d|jvr||S||SdS)aGet important HN webpage information. HN webpage components are: - title - content - source url, - time of post - author of the post - number of comments - rank of the post itemN)Zscrapeweb_path load_comments load_results)self soup_inforu/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/hn.pyload s   z HNLoader.load)r rcs.|d}|ddfdd|DS)zLoad comments from a HN post.ztr[class='athing comtr']ztr[id='pagespace']titlecs&g|]}t|jjddqS))sourcer) page_contentmetadata)rtextstripr ).0commentr rrr #s  z*HNLoader.load_comments..)select select_oneget)r r commentsrrrr s   zHNLoader.load_comments)souprc Cs|d}g}|D]h}|dj}|dddidd}|dddij}|j|||d}|t||||d q|S) zLoad items from an HN page.ztr[class='athing']zspan[class='rank']spanclassZ titlelineahref)rrlinkranking)rr$r%r) rrrfindrrr appendr) r ritemsZ documentsZlineItemr%r$rrrrrr +s"  zHNLoader.load_resultsN) __name__ __module__ __qualname____doc__rrrrr r rrrrrs rN)typingrrZlangchain_core.documentsrZ-langchain_community.document_loaders.web_baserrrrrrs