a bgD"@slddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZGdddeZdS) N)PathLike)Path)AnyCallableDictIteratorOptionalUnion)Document) BaseLoaderc @seZdZdZdeeefeeeeeee e e ge feedddZ e e dd d Zeee e d d d ZeedddZe eefee eefdddZeddddZdS) JSONLoadera Load a `JSON` file using a `jq` schema. Setup: .. code-block:: bash pip install -U jq Instantiate: .. code-block:: python from langchain_community.document_loaders import JSONLoader import json from pathlib import Path file_path='./sample_quiz.json' data = json.loads(Path(file_path).read_text()) loader = JSONLoader( file_path=file_path, jq_schema='.quiz', text_content=False) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quizg .json', 'seq_num': 1} Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1} NFT) file_path jq_schema content_keyis_content_key_jq_parsable metadata_func text_content json_linesc Csjzddl}||_Wnty,tdYn0t||_|||_||_||_||_ ||_ ||_ dS)a~Initialize the JSONLoader. Args: file_path (Union[str, PathLike]): The path to the JSON or JSON Lines file. jq_schema (str): The jq schema to use to extract the data or text from the JSON. content_key (str): The key to use to extract the content from the JSON if the jq_schema results to a list of objects (dict). If is_content_key_jq_parsable is True, this has to be a jq compatible schema. If is_content_key_jq_parsable is False, this should be a simple string key. is_content_key_jq_parsable (bool): A flag to determine if content_key is parsable by jq or not. If True, content_key is treated as a jq schema and compiled accordingly. If False or if content_key is None, content_key is used as a simple string. Default is False. metadata_func (Callable[Dict, Dict]): A function that takes in the JSON object extracted by the jq_schema and the default metadata and returns a dict of the updated metadata. text_content (bool): Boolean flag to indicate whether the content is in string format, default to True. json_lines (bool): Boolean flag to indicate whether the input is in JSON Lines format. rNz=jq package not found, please install it with `pip install jq`) jq ImportErrorrresolver compile _jq_schema_is_content_key_jq_parsable _content_key_metadata_func _text_content _json_lines) selfr rrrrrrrr~/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_community/document_loaders/json_loader.py__init__Ts"    zJSONLoader.__init__)returnccsd}|jrp|jjddF}|D]0}|}|r|||D]}|V|d7}q:qWdq1sd0Yn*||jjdd|D]}|V|d7}qdS)z-Load and return documents from the JSON file.rzutf-8)encodingN)rr openstrip_parse read_text)rindexflinedocrrr lazy_loads,zJSONLoader.lazy_load)contentr)r"ccsr|jt|}|jdur&||t||dD]8\}}|j|d}|j|t |j |d}t ||dVq4dS)z#Convert given content to documents.Nr$)sample)r/sourceZseq_num)Z page_contentmetadata) rinputjsonloadsr_validate_content_key enumerate _get_text _get_metadatastrr r )rr.r)datair/textr1rrr r's    zJSONLoader._parse)r/r"cCs|jdur:|jr.|j|j}||}q>||j}n|}|jrlt|tsl|durlt dt |dnBt|trz|St|t t fr|rt |SdS|durt|SdSdS)zConvert sample to string formatNz%Expected page_content is string, got z instead. Set `text_content=False` if the desired input for `page_content` is not a string)rrrrr2firstr isinstancer9 ValueErrortypedictlistr3dumps)rr/Zcompiled_content_keyr.rrr r7s   zJSONLoader._get_text)r/additional_fieldsr"cKs@|jdur8|||}t|ts4tdt|d|S|SdS)z Return a metadata dictionary base on the existence of metadata_func :param sample: single data payload :param additional_fields: key-word arguments to be added as metadata values :return: NzUExpected the metadata_func to return a dict but got ``)rr?rBr@rA)rr/rEresultrrr r8s   zJSONLoader._get_metadata)r:r"cCs|}t|ts&tdt|d|jsN||jdurNtd|jd|jr|j |j | durtd|jddS)zCheck if a content key is validztExpected the jq schema to result in a list of objects (dict), so sample must be a dict but got `rFNz_Expected the jq schema to result in a list of objects (dict) with the key `z ` which should be parsable by jq) r>r?rBr@rArgetrrrr2r<)rr:r/rrr r5s2 z JSONLoader._validate_content_key)NFNTF)__name__ __module__ __qualname____doc__r r9rrboolrrr!rr r-intr'rr7r8r5rrrr r s,L  3  r )r3osrpathlibrtypingrrrrrr Zlangchain_core.documentsr Z)langchain_community.document_loaders.baser r rrrr s