a bgw#@sddlmZddlZddlZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZertddlZddlmZdd d d d ZGd ddeZdddddZGdddeZeZdS)) annotationsN)Path) TYPE_CHECKINGDictListOptionalUnionDocument) BaseLoader) EntityLikedictstr)rowreturncCs.|d}|d}|d}|d|d|dS)zBCombine message information in a readable format ready to be used.datefromtextz on z:  )rrZsenderrrr{/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/telegram.pyconcatenate_rowssrc@s,eZdZdZddddZdddd Zd S) TelegramChatFileLoaderzLoad from `Telegram chat` dump.zUnion[str, Path])pathcCs ||_dS)zInitialize with a path.N) file_path)selfrrrr__init__szTelegramChatFileLoader.__init__List[Document]rcCsrt|j}t|dd}t|}Wdn1s60Yddd|dD}dt|i}t||d gS) Load documents.utf8encodingNcss0|](}|ddkrt|dtrt|VqdS)typemessagerN) isinstancerr).0r%rrr 'sz.TelegramChatFileLoader.load..messagessource page_contentmetadata)rropenjsonloadjoinrr )rpfdrr-rrrr0 s (  zTelegramChatFileLoader.loadN)__name__ __module__ __qualname____doc__rr0rrrrrsrzUnion[str, List[str]]r)rrc Csddlm}|dgddd}t|tr.|g}dd|D}t|D]\}}|d |jd <qDg}|D]d}||j}t|D]J\}}t||jd |d d }|jd d |jd|jd<| |q|qd|S)zIConvert a string or list of strings to a list of Documents with metadata.r)RecursiveCharacterTextSplitteri )r .!?, r#) chunk_size separatorsZ chunk_overlapcSsg|]}t|dqS))r,r )r'pagerrr >z text_to_docs..rC)rCchunkr+-rGr*) Zlangchain_text_splittersr9r&r enumerater-Z split_textr,r append) rr9Z text_splitterZ page_docsidocZ doc_chunkschunksrGrrr text_to_docs1s*    rNc@sfeZdZdZdddddddd d Zd d d dZdddddZddddddZdd ddZdS)TelegramChatApiLoaderz)Load `Telegram` chat json directory dump.Ntelegram_data.jsonzOptional[EntityLike]z Optional[int]z Optional[str]r chat_entityapi_idapi_hashusernamercCs"||_||_||_||_||_dS)aIInitialize with API parameters. Args: chat_entity: The chat entity to fetch data from. api_id: The API ID. api_hash: The API hash. username: The username. file_path: The file path to save the data to. Defaults to "telegram_data.json". NrQ)rrRrSrTrUrrrrrVs zTelegramChatApiLoader.__init__Nonerc sddlm}g}||j|j|j4IdHp}||j2zJ3dHW}|jdu}|rZ|jjnd}| |j |j |j |j||dq86WdIdHq1IdHs0Yt|jddd"}tj||dd d Wdn1s0YdS) z8Fetch data from Telegram API and save it as a JSON file.r)TelegramClientN) sender_idrr message.idis_reply reply_to_idwzutf-8r!F) ensure_asciiindent)Z telethon.syncrWrUrSrTZ iter_messagesrRZreply_toZreply_to_msg_idrJrXrr isoformatidr.rr/dump)rrWdataclientr%rZr[r3rrrfetch_data_from_telegramns"  2 z.TelegramChatApiLoader.fetch_data_from_telegram pd.DataFramer )rcrcshddddfdd ||d}||djdgd dtd<fd d |d D}|S) a Create a dictionary of message threads from the given data. Args: data (pd.DataFrame): A DataFrame containing the conversation data with columns: - message.sender_id - text - date - message.id - is_reply - reply_to_id Returns: dict: A dictionary where the key is the parent message ID and the value is a list of message IDs in ascending order. intrfz List[int]) parent_id reply_datarcs>||d|kd}g}|D]}||g||7}q |S)a^ Recursively find all replies to a given parent message ID. Args: parent_id (int): The parent message ID. reply_data (pd.DataFrame): A DataFrame containing reply messages. Returns: list: A list of message IDs that are replies to the parent message ID. r[rY)tolist)rhriZdirect_repliesZ all_repliesZreply_id) find_repliesrrrks z@TelegramChatApiLoader._get_message_threads..find_repliesrZr[)Zsubsetcsi|]}||g|qSrr)r'rhrkZreply_messagesrr sz>TelegramChatApiLoader._get_message_threads..rY)ZdropnaZastyperg)rrcZparent_messagesmessage_threadsrrlr_get_message_threadss z*TelegramChatApiLoader._get_message_threadszDict[int, List[int]])rnrcrcCs`d}|D]J\}}||d|jddd}dd|D}|d|d 7}q |S) aw Combine the message texts for each parent message ID based on the list of message threads. Args: message_threads (dict): A dictionary where the key is the parent message ID and the value is a list of message IDs in ascending order. data (pd.DataFrame): A DataFrame containing the conversation data: - message.sender_id - text - date - message.id - is_reply - reply_to_id Returns: str: A combined string of message texts sorted by date. r#rYr)ZbyrcSsg|] }t|qSr)r)r'elemrrrrDrEz@TelegramChatApiLoader._combine_message_texts..r?z. )itemsisinZ sort_valuesrjr1strip)rrnrcZ combined_textrhZ message_idsZ message_textsrrr_combine_message_textssz,TelegramChatApiLoader._combine_message_textsrc Cs|jdurHz"ddl}|t|WntyFtdYn0t|j}t |dd}t |}Wdn1s~0Yz ddl }WntytdYn0| |}||}||}|||} t| S)rNrzy`nest_asyncio` package not found. please install with `pip install nest_asyncio` r r!zf`pandas` package not found. please install with `pip install pandas` )rR nest_asyncioapplyasynciorunre ImportErrorrrr.r/r0pandasZjson_normalizeZ DataFramerortrN) rrur2r3r4pdZnormalized_messagesZdfrnZcombined_textsrrrr0s.    (       zTelegramChatApiLoader.load)NNNNrP) r5r6r7r8rrerortr0rrrrrOSs;&rO) __future__rrwr/pathlibrtypingrrrrrZlangchain_core.documentsr Z)langchain_community.document_loaders.baser rzr{Ztelethon.hintsr rrrNrOZTelegramChatLoaderrrrrs     ":