a bg?H@sdZddlmZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZeeZd gZ eGdddZ!ddhZ"hdZ#dddddZ$GdddeZ%GdddeZ&eGdddeZ'dS)zLoads YouTube transcript.) annotationsN)Enum)Path)AnyDict GeneratorListOptionalSequenceUnion)parse_qsurlparse) ParseError)Document)model_validator) dataclass) BaseLoaderz0https://www.googleapis.com/auth/youtube.readonlyc@seZdZUdZeddZded<eddZded<eddZ ded<d d d d Z e d de dddddZ dd ddZdS)GoogleApiClientaGeneric Google API Client. To use, you should have the ``google_auth_oauthlib,youtube_transcript_api,google`` python package installed. As the google api expects credentials you need to set up a google account and register your Service. "https://developers.google.com/docs/api/quickstart/python" *Security Note*: Note that parsing of the transcripts relies on the standard xml library but the input is viewed as trusted in this case. Example: .. code-block:: python from langchain_community.document_loaders import GoogleApiClient google_api_client = GoogleApiClient( service_account_path=Path("path_to_your_sec_file.json") ) z .credentialszcredentials.jsonrcredentials_pathservice_account_pathz token.json token_pathNonereturncCs||_dSN)_load_credentialscredsselfrz/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/youtube.py __post_init__2szGoogleApiClient.__post_init__beforemodervaluesrcCs&|jds |jds td|jS)DValidate that either folder_id or document_ids is set, but not both.rr-Must specify either channel_name or video_ids)kwargsget ValueErrorclsr&rrr #validate_channel_or_videoIds_is_set5s z3GoogleApiClient.validate_channel_or_videoIds_is_setc Cs z@ddlm}ddlm}ddlm}ddlm}ddlm }Wnt yZt dYn0d}|j r||j t|j S|j r|t|jt}|r|js|r|jr|jr||n|t|jt}|jdd }t|jd }||Wdn1s0Y|S) zLoad credentials.r)Request)service_account) Credentials)InstalledAppFlowYouTubeTranscriptApiYou must run`pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib youtube-transcript-api` to use the Google Drive loaderN)portw)Zgoogle.auth.transport.requestsr/Z google.oauth2r0Zgoogle.oauth2.credentialsr1Zgoogle_auth_oauthlib.flowr2youtube_transcript_apir4 ImportErrorrexistsZfrom_service_account_filestrrZfrom_authorized_user_fileSCOPESZvalidZexpiredZ refresh_tokenrefreshZfrom_client_secrets_filerZrun_local_serveropenwriteto_json) rr/r0r1r2r4rZflowtokenrrr r@s6           .z!GoogleApiClient._load_credentialsN)__name__ __module__ __qualname____doc__rhomer__annotations__rrr!r classmethodr.rrrrr rs  rhttphttps>zyoutu.bezwww.youtube-nocookie.comz youtube.comz m.youtube.comzvid.pluszwww.youtube.comr; Optional[str])urlrcCst|}|jtvrdS|jtvr$dS|j}|drp|j}t|}d|vrj|d}t |t r`|n|d}qdSn|j d}| dd}t |dkrdS|S)zEParse a YouTube URL and return the video ID if valid, otherwise None.Nz/watchvr/ )r schemeALLOWED_SCHEMESnetlocALLOWED_NETLOCSpathendswithqueryr isinstancer;lstripsplitlen)rL parsed_urlrUrWZ parsed_queryZidsvideo_idrrr _parse_video_idts$     r^c@seZdZdZdZdZdZdS)TranscriptFormatz3Output formats of transcripts from `YoutubeLoader`.textlineschunksN)rBrCrDrETEXTLINESCHUNKSrrrr r_sr_c @seZdZdZdddejddfdddd d dd d d dZedddddZe ddddddZ dd ddddZ dddddZ dd d!d"Z d#d d$d%ZdS)& YoutubeLoaderz!Load `YouTube` video transcripts.FenNxr;boolzUnion[str, Sequence[str]]rKr_int)r]add_video_infolanguage translationtranscript_formatcontinue_on_failurechunk_size_secondscCsR||_d|i|_||_||_t|tr0|g|_n||_||_||_||_||_ dS)z!Initialize with YouTube video ID.sourceN) r] _metadatarkrlrXr;rmrnrorp)rr]rkrlrmrnrorprrr __init__s    zYoutubeLoader.__init__) youtube_urlrcCs t|}|std|d|S)z*Extract video ID from common YouTube URLs.z.Could not determine the video ID for the URL "z".)r^r+)rtr]rrr extract_video_ids  zYoutubeLoader.extract_video_idr)rtr)rcKs||}||fi|S)z|Given a YouTube URL, construct a loader. See `YoutubeLoader()` constructor for a list of keyword arguments. )ru)r-rtr)r]rrr from_youtube_urls zYoutubeLoader.from_youtube_urlz List[Dict]r) chunk_pieceschunk_start_secondsrc Cspt|d\}}t|d\}}tdtdd|i|j||dd|dd|dd|jd|d d d S) z0Create Document from chunk of transcript pieces.< cSs|ddSNr`rzstrip)Z chunk_piecerrr z4YoutubeLoader._make_chunk_document..02d: https://www.youtube.com/watch?v=z&t=s)Z start_secondsZstart_timestamprq page_contentmetadata)divmodrjoinmaprrr])rrwrxmrhrrr _make_chunk_documents   z"YoutubeLoader._make_chunk_documentzGenerator[Document, None, None])transcript_piecesrccs|g}d}|j}|D]J}|d|d}||krR|r@|||Vg}|}||j7}||qt|dkrx|||VdS)Nrstartduration)rprappendr[)rrrwrxZchunk_time_limittranscript_pieceZ piece_endrrr _get_transcript_chunkss   z$YoutubeLoader._get_transcript_chunksList[Document]rcCs<zddlm}m}m}Wnty2tdYn0|jrN|}|j|z| |j }Wn|ytgYS0z| |j }Wn|y| dg}Yn0|j dur||j }|}|jtjkrdtdd|}t||jd gS|jtjkrttd d|S|jtjkr0t||Std dS) z1Load YouTube transcripts into `Document` objects.r)NoTranscriptFoundTranscriptsDisabledr4zvCould not import "youtube_transcript_api" Python package. Please install it with `pip install youtube-transcript-api`.rgNrzcSs|ddSr{r|rrrr r~rz$YoutubeLoader.load..rcSs(t|ddttdd|dS)Nr`rzcSs |ddkS)Nrr`r)itemrrr r~"rz6YoutubeLoader.load....r)rr}dictfilteritemsrrrr r~s  zUnknown transcript format.)r8rrr4r9rk_get_video_inforrupdatelist_transcriptsr]find_transcriptrlrm translatefetchrnr_rcrrrrdlistrerr+)rrrr4 video_infotranscript_list transcriptrrrr loadsL          zYoutubeLoader.loadrcCszddlm}Wnty*tdYn0|d|j}|jpDd|jpLd|jpTd|jp\d|jrn|j dnd|j pxd|j pdd}|S)zGet important video information. Components include: - title - description - thumbnail URL, - publish_date - channel author - and more. r)YouTubezVCould not import "pytube" Python package. Please install it with `pip install pytube`.rUnknownz%Y-%m-%d %H:%M:%S)title descriptionZ view_count thumbnail_url publish_datelengthauthor) Zpytuberr9r]rrZviewsrrstrftimerr)rrZytrrrr r/s$    zYoutubeLoader._get_video_info)rBrCrDrEr_rcrs staticmethodrurHrvrrrrrrrr rfs  ?rfc@seZdZUdZded<dZded<dZded<d Zd ed <d Zd ed<dZ d ed<ddddZ dddddZ e dde dddddZd d dd d!Zd dd"d#d$d%Zd d d&d'd(Zd d d)d*d+Zd dd,d-d.d/Zd,dd0d1ZdS)2GoogleApiYoutubeLoaderaLoad all Videos from a `YouTube` Channel. To use, you should have the ``googleapiclient,youtube_transcript_api`` python package installed. As the service needs a google_api_client, you first have to initialize the GoogleApiClient. Additionally you have to either provide a channel name or a list of videoids "https://developers.google.com/docs/api/quickstart/python" Example: .. code-block:: python from langchain_community.document_loaders import GoogleApiClient from langchain_community.document_loaders import GoogleApiYoutubeLoader google_api_client = GoogleApiClient( service_account_path=Path("path_to_your_sec_file.json") ) loader = GoogleApiYoutubeLoader( google_api_client=google_api_client, channel_name = "CodeAesthetic" ) load.load() rgoogle_api_clientNrK channel_namezOptional[List[str]] video_idsTrirkrgr;captions_languageFrorrcCs||jj|_dSr)_build_youtube_clientrryoutube_clientrrrr r!vsz$GoogleApiYoutubeLoader.__post_init__r)rrcCsFzddlm}ddlm}Wnty6tdYn0|dd|dS)Nr)buildr3r5ZyoutubeZv3) credentials)Zgoogleapiclient.discoveryrr8r4r9)rrrr4rrr rys   z,GoogleApiYoutubeLoader._build_youtube_clientr"r#zDict[str, Any]r%cCs |ds|dstd|S)r'rrr()r*r+r,rrr r.sz:GoogleApiYoutubeLoader.validate_channel_or_videoIds_is_set)r]rcCstddlm}m}||}z||jg}Wn*|yV|D]}||j}q>q>Yn0|}ddd|DS)Nr)rr4rzcSsg|]}|ddqS)r`rzr|).0trrr rzGGoogleApiYoutubeLoader._get_transcripe_for_video_id..) r8rr4rrrrrr)rr]rr4rrZavailable_transcriptrrrr _get_transcripe_for_video_ids    z3GoogleApiYoutubeLoader._get_transcripe_for_video_idr)r]r)rcKs8||}|jjd|d}t||dddS)N id,snippetpartidrrr)rrZvideosrexecuterr*)rr]r)ZcaptionsZvideo_responserrr _get_document_for_video_ids   z1GoogleApiYoutubeLoader._get_document_for_video_id)rrcCs8|jjd|ddd}|}|dddd}|S)Nrchannel)rqtype maxResultsrrZ channelId)rsearchrr)rrrequestresponse channel_idrrr _get_channel_ids z&GoogleApiYoutubeLoader._get_channel_id)rrcCs4|jjd|d}|}|dddddS)NZcontentDetailsrrrZrelatedPlaylistsZuploads)rZchannelsrr)rrrrrrr _get_uploads_playlist_ids  z/GoogleApiYoutubeLoader._get_uploads_playlist_idr)rr)rc KsNzddlm}m}Wnty.tdYn0||}||}|jjd|dd}g}|durJ| } | dD]} | dd d } d | i} |j r| d d | | dz | | } |t| | d Wqx||tfy2}z<|jrtd d| dd d|n|WYd}~qxd}~00qx|j|| }q^|S)Nr)rrzTYou must run`pip install --upgrade youtube-transcript-api` to use the youtube loaderr2)rZ playlistIdrrZsnippetZ resourceIdZvideoIdZ thumbnailsrzError fetching transscript rzrz , exception: )r8rrr9rrrZ playlistItemsrrrkpoprrrrrrologgererrorrZ list_next)rrr)rrrZuploads_playlist_idrrrrr]Z meta_datarerrr _get_document_for_channelsR        z0GoogleApiYoutubeLoader._get_document_for_channelcsLg}jr|jn*jr@|fddjDntd|S)zLoad documents.csg|]}|qSr)r)rr]rrr rsz/GoogleApiYoutubeLoader.load..r()rextendrrr+)rZ document_listrrr rs zGoogleApiYoutubeLoader.load)rBrCrDrErGrrrkrror!rrrHr.rrrrrrrrrr rQs$       5r)(rE __future__rloggingenumrpathlibrtypingrrrrr r r urllib.parser r Zxml.etree.ElementTreerZlangchain_core.documentsrZpydanticrZpydantic.dataclassesrZ)langchain_community.document_loaders.baser getLoggerrBrr<rrRrTr^r_rfrrrrr s.   $      Q 8