a dgˡ @s:ddlZddlZddlmZmZmZmZmZmZm Z m Z m Z ddl m Z mZmZddlmZddlmZddlmZddlmZddlmZmZmZdd lmZdd lmZm Z m!Z!d Z"d d dddddd d ddddddddddddddddddddddddddgZ#dZ$dPe%ed!d"d#Z&e%e%d$d%d&Z'dQee ee%eee%e%e%ffe%e%ee%ee%eed(d)d*Z(Gd+d,d,eZ)Gd-d.d.eZ*dReee%ee ee%eee%e%e%ffee%ee%ed/d0d1Z+dSeee%ee ee%eee%e%e%ffe e,ee%fee%e e,ee%fee%e e)d3d4d5Z-eed6d7d8Z.eed9d:d;Z/ee%efeeeeefd<d=d>Z0eeeed?d@dAZ1eeeedBdCdDZ2e%e%dEdFdGZ3eeefeeeeefdHdIdJZ4e ee%eee%e%e%ffeee%ee%dKdLdMZ5GdNdOdOZ6dS)TN) AnyDictListOptionalSequenceTupleTypeUnioncast) GraphDocumentNode Relationship)Document)BaseLanguageModel) SystemMessage)JsonOutputParser)ChatPromptTemplateHumanMessagePromptTemplatePromptTemplate)RunnableConfig) BaseModelField create_modelr zeAdam is a software engineer in Microsoft since 2009, and last year he got an award as the Best TalentZAdamZPersonZ WORKS_FOR MicrosoftZCompany)texthead head_typerelationtail tail_typeZ HAS_AWARDz Best TalentZAwardzPMicrosoft is a tech company that provide several products such as Microsoft WordzMicrosoft WordZProductZ PRODUCED_BYz;Microsoft Word is a lightweight app that accessible offlineZHAS_CHARACTERISTICzlightweight appZCharacteristiczaccessible offlinea# Knowledge Graph Instructions for GPT-4 ## 1. Overview You are a top-tier algorithm designed for extracting information in structured formats to build a knowledge graph. Try to capture as much information from the text as possible without sacrificing accuracy. Do not add any information that is not explicitly mentioned in the text. - **Nodes** represent entities and concepts. - The aim is to achieve simplicity and clarity in the knowledge graph, making it accessible for a vast audience. ## 2. Labeling Nodes - **Consistency**: Ensure you use available types for node labels. Ensure you use basic or elementary types for node labels. - For example, when you identify an entity representing a person, always label it as **'person'**. Avoid using more specific terms like 'mathematician' or 'scientist'.- **Node IDs**: Never utilize integers as node IDs. Node IDs should be names or human-readable identifiers found in the text. - **Relationships** represent connections between entities or concepts. Ensure consistency and generality in relationship types when constructing knowledge graphs. Instead of using specific and momentary types such as 'BECAME_PROFESSOR', use more general and timeless relationship types like 'PROFESSOR'. Make sure to use general and timeless relationship types! ## 3. Coreference Resolution - **Maintain Entity Consistency**: When extracting entities, it's vital to ensure consistency. If an entity, such as "John Doe", is mentioned multiple times in the text but is referred to by different names or pronouns (e.g., "Joe", "he"),always use the most complete identifier for that entity throughout the knowledge graph. In this example, use "John Doe" as the entity ID. Remember, the knowledge graph should be coherent and easily understandable, so maintaining consistency in entity references is crucial. ## 4. Strict Compliance Adhere to the rules strictly. Non-compliance will result in termination.)additional_instructionsreturncCstdtfd|dfgS)NsystemZhumanz Tip: Make sure to answer in the correct format and do not include any explanations. Use the given format to extract information from the following input: {input})r from_messages system_prompt)r!r&{/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_experimental/graph_transformers/llm.pyget_default_promptnsr() input_typer"cCs8|dvrtd|dkrdS|dkr(dS|dkr4dSdS) N)node relationshippropertyz8input_type must be 'node', 'relationship', or 'property'r*zEnsure you use basic or elementary types for node labels. For example, when you identify an entity representing a person, always label it as **'Person'**. Avoid using more specific terms like 'Mathematician' or 'Scientist'r+zInstead of using specific and momentary types such as 'BECAME_PROFESSOR', use more general and timeless relationship types like 'PROFESSOR'. However, do not sacrifice any accuracy for generalityr,r ) ValueError)r)r&r&r'_get_additional_infosr.r*) enum_values descriptionr)llm_typerelationship_type field_kwargsr"cKs|}|dkrtdd|D}|rH|dkrHtd ||d|d|S|rhtd d|d|i|St|}td d||i|Sd S) zIUtility function to conditionally create a field with an enum constraint.tuplecSsh|] }|dqSr&.0elr&r&r' z&optional_enum_field..z openai-chat.z. Available options are )enumr0r0N).).).)listrr.)r/r0r)r1r2r3Zparsed_enum_valuesZadditional_infor&r&r'optional_enum_fields,    r>c@s&eZdZUeeed<eeed<dS)_Graphnodes relationshipsN)__name__ __module__ __qualname__rr__annotations__r&r&r&r'r?s  r?c@sheZdZUeddZeed<eddZeed<eddZeed<eddZ eed <ed dZ eed <d S) UnstructuredRelationz]extracted head entity like Microsoft, Apple, John. Must use human-readable unique identifier.r0rz;type of the extracted head entity like Person, Company, etcrz/relation between the head and the tail entitiesrz]extracted tail entity like Microsoft, Apple, John. Must use human-readable unique identifier.rz;type of the extracted tail entity like Person, Company, etcrN) rBrCrDrrstrrErrrrr&r&r&r'rFs rF) node_labels rel_typesr2r!r"cCs:|r t|nd}|r>|dkr4ttdd|D}qBt|}nd}d|rTd|dnd|rfd|dnd|rxd |dnd|dkrd |ndd d |g}d td|}t|d}ttd} d|rdnd|rdnd|dkrd |ndd|dg} d td| } t| dg| ||t dd} t | d} t || g}|S)Nr r4cSsh|] }|dqSr5r&r8itemr&r&r'r:r;z-create_unstructured_prompt..aYou are a top-tier algorithm designed for extracting information in structured formats to build a knowledge graph. Your task is to identify the entities and relations requested with the user prompt from a given text. You must generate the output in a JSON format containing a list with JSON objects. Each object should have the keys: "head", "head_type", "relation", "tail", and "tail_type". The "head" key must contain the text of the extracted entity with one of the types from the provided list in the user prompt.zlThe "head_type" key must contain the type of the extracted head entity, which must be one of the types from .zThe "relation" key must contain the type of relation between the "head" and the "tail", which must be one of the relations from zThe "tail" key must represent the text of an extracted entity which is the tail of the relation, and the "tail_type" key must contain the type of the tail entity from Your task is to extract relationships from text strictly adhering to the provided schema. The relationships can only appear between specific node types are presented in the schema format like: (Entity1Type, RELATIONSHIP_TYPE, Entity2Type) /nProvided schema is aAttempt to extract as many entities and relations as you can. Maintain Entity Consistency: When extracting entities, it's vital to ensure consistency. If an entity, such as "John Doe", is mentioned multiple times in the text but is referred to by different names or pronouns (e.g., "Joe", "he"), always use the most complete identifier for that entity. The knowledge graph should be coherent and easily understandable, so maintaining consistency in entity references is crucial.z7IMPORTANT NOTES: - Don't add any explanation and text.  )content)Zpydantic_objectzXBased on the following example, extract entities and relations from the provided text. zmUse the following entity types, don't use other entity that is not defined below:# ENTITY TYPES:{node_labels}zqUse the following relation types, don't use other relation that is not defined below:# RELATION TYPES:{rel_types}zaBelow are a number of examples of text and their extracted entities and relationships.{examples} zuFor the following text, extract entities and relations as in the provided example.{format_instructions} Text: {input}input)Zformat_instructionsrIrJexamples)templateZinput_variablesZpartial_variables)prompt) rHr=joinfilterrrrFrZget_format_instructionsrRrrr$)rIrJr2r!Znode_labels_strZ rel_types_strZbase_string_partsr%Zsystem_messageparserZhuman_string_partsZhuman_prompt_stringZ human_promptZhuman_message_promptZ chat_promptr&r&r'create_unstructured_prompts  (   rXF)rIrJnode_propertiesr1relationship_propertiesr2r"c sttdddftt|dddfd}|rt|trDd|vrDtd |d urPgn|Gfd d d t}tt|td ddf|d<t d"i|ttdddftt|dddfttdddftt|dddftt|dd|dfd}|rVt|trd|vrtd|d ur gn|Gfdddt} tt| td ddf|d<t d#i||dkrzd|_ Gfd d!d!t } | S)$a Create a simple graph model with optional constraints on node and relationship types. Args: node_labels (Optional[List[str]]): Specifies the allowed node types. Defaults to None, allowing all node types. rel_types (Optional[List[str]]): Specifies the allowed relationship types. Defaults to None, allowing all relationship types. node_properties (Union[bool, List[str]]): Specifies if node properties should be included. If a list is provided, only properties with keys in the list will be included. If True, all properties are included. Defaults to False. relationship_properties (Union[bool, List[str]]): Specifies if relationship properties should be included. If a list is provided, only properties with keys in the list will be included. If True, all properties are included. Defaults to False. llm_type (Optional[str]): The type of the language model. Defaults to None. Only openai supports enum param: openai-chat. Returns: Type[_Graph]: A graph model with the specified constraints. Raises: ValueError: If 'id' is included in the node or relationship properties list. .z)Name or human-readable unique identifier.rGzThe type or label of the node.r*r0r)r1idtyper]z6The node property 'id' is reserved and cannot be used.Tcs>eZdZUdZedddZeed<edddZ eed <d S) z%create_simple_model..Property-A single property consisting of key and value Property key.r,r[key.BExtracted value. Any date value should be formatted as yyyy-mm-dd.rGvalueN rBrCrD__doc__r>rarHrErrcr&)r1node_properties_mappedr&r'Property{s rgNzList of node properties properties SimpleNodez7Name or human-readable unique identifier of source nodez%The type or label of the source node.z7Name or human-readable unique identifier of target nodez%The type or label of the target node.zThe type of the relationship.r+)r0r)r1r2)source_node_idsource_node_typetarget_node_idtarget_node_typer^z>The relationship property 'id' is reserved and cannot be used.cs>eZdZUdZedddZeed<edddZ eed <d S) z1create_simple_model..RelationshipPropertyr_r`r,r[ra.rbrGrcNrdr&)r1relationship_properties_mappedr&r'RelationshipPropertys rozList of relationship propertiesSimpleRelationshipr4rNcsFeZdZUdZeddZeeed<eddZ eeed<dS)z)create_simple_model..DynamicGraphzBRepresents a graph document consisting of nodes and relationships.z List of nodesrGr@zList of relationshipsrAN) rBrCrDrerr@rrrErAr&)rirpr&r' DynamicGraphs rq)ri)rp) rHrr> isinstancer=r-rrrrrer?) rIrJrYr1rZr2Z node_fieldsrgZrelationship_fieldsrorqr&)rirpr1rfrnr'create_simple_modelAs$     ,   rs)r*r"cCsBi}t|dr0|jr0|jD]}|j|t|j<qt|j|j|dS)z$Map the SimpleNode to the base Node.rhr]r^rh)hasattrrhrcformat_property_keyrar r]r^)r*rhpr&r&r'map_to_base_nodes  rx)relr"cCsbt|j|jd}t|j|jd}i}t|drP|jrP|jD]}|j|t|j <q:t |||j |dS)z4Map the SimpleRelationship to the base Relationship.r\rhsourcetargetr^rh) r rjrkrlrmrurhrcrvrar r^)ryr{r|rhrwr&r&r'map_to_base_relationships  r}) argument_jsonr"c sg}|dD]f}|dsq i}d|vrR|drR|dD]}|d|t|d<q8|t|d|dt|dq g}|dD]6d rd rdsqd sz"fd d |dDdd <Wntytd <Yn0ds:z"fdd |dDdd<Wnty8td<Yn0i}dvrvdrvdD]}|d|t|d<qZtd d d}td dd}|t||d|dq||fS)Nr@r]rhrcrar^rtrArjrlrkcs&g|]}|ddkr|dqS)r]rjr^getr7ryr&r' +sz)_parse_and_clean_json..rrmcs&g|]}|ddkr|dqS)r]rlr^rr7rr&r'r4sr\rz)rrvappendr DEFAULT_NODE_TYPE IndexErrorr ) r~r@r*rYrwrAZrel_properties source_node target_noder&rr'_parse_and_clean_json s~            r)r@r"cCsdd|DS)NcSsDg|]<}tt|jtr |jn|j|jr4|jnt|jdqS)rt) r rrr]rHtitler^ capitalizerrhr7r&r&r'rUs z!_format_nodes..r&)r@r&r&r' _format_nodesTsr)relsr"cCsdd|DS)Nc SsDg|]<}tt|jgdt|jgd|jdd|jdqS)r _rz)r rr{r|r^replaceupperrhr7r&r&r'rbsz)_format_relationships..r&)rr&r&r'_format_relationshipsasr)sr"cCsB|}|s|S|d}dd|ddD}d|g|S)NrcSsg|] }|qSr&)r)r8wordr&r&r'rrr;z'format_property_key..r6r )splitlowerrU)rwordsZ first_wordZcapitalized_wordsr&r&r'rvms  rv) raw_schemar"c Cs@|dszz$t|djdddd}Wntyzt|djdd}Wndty|djdd}t|d trt|d |d <t|d trt|d |d <Yn0Yn0t|\}}WntyggfYS0n@|d}|jrd d |jDng}|j r,d d |j Dng}t |t |fS)Nparsedraw tool_callsrfunction argumentsZ function_callargsr@rAcSsg|]}|jrt|qSr&)r]rxr8r*r&r&r'rr;z._convert_to_graph_document..cSs&g|]}|jr|jr|jrt|qSr&)r^rjrlr}r8ryr&r&r'rs) jsonloadsZadditional_kwargs ExceptionrrrrHrr@rArr)rr~r@rAZ parsed_schemar&r&r'_convert_to_graph_documentvsL       r)allowed_relationships allowed_nodesr"csZ|rt|tstd|sdStdd|Dr4dStfdd|DrNdStddS)Nz1`allowed_relationships` attribute must be a list.css|]}t|tVqdSNrrrHrKr&r&r' r;z5validate_and_get_relationship_type..stringc3sN|]F}t|toDt|dkoDtdd|DoD|dvoD|dvVqdS)css|]}t|tVqdSrr)r8subitemr&r&r'rr;z?validate_and_get_relationship_type...rN)rrr4lenallrKrr&r'rs   r4z`allowed_relationships` must be list of strings or a list of 3-item tuples. For tuples, the first and last elements must be in the `allowed_nodes` list.)rrr=r-r)rrr&rr'"validate_and_get_relationship_types  rc @seZdZdZggddddddfeeeeeeeeeeeffe e e ee eefee eefe edd ddZ de e eed d d Zdee e eeed d dZde e eed ddZdee e eeed ddZdS)LLMGraphTransformeraD Transform documents into graph-based documents using a LLM. It allows specifying constraints on the types of nodes and relationships to include in the output graph. The class supports extracting properties for both nodes and relationships. Args: llm (BaseLanguageModel): An instance of a language model supporting structured output. allowed_nodes (List[str], optional): Specifies which node types are allowed in the graph. Defaults to an empty list, allowing all node types. allowed_relationships (List[str], optional): Specifies which relationship types are allowed in the graph. Defaults to an empty list, allowing all relationship types. prompt (Optional[ChatPromptTemplate], optional): The prompt to pass to the LLM with additional instructions. strict_mode (bool, optional): Determines whether the transformer should apply filtering to strictly adhere to `allowed_nodes` and `allowed_relationships`. Defaults to True. node_properties (Union[bool, List[str]]): If True, the LLM can extract any node properties from text. Alternatively, a list of valid properties can be provided for the LLM to extract, restricting extraction to those specified. relationship_properties (Union[bool, List[str]]): If True, the LLM can extract any relationship properties from text. Alternatively, a list of valid properties can be provided for the LLM to extract, restricting extraction to those specified. ignore_tool_usage (bool): Indicates whether the transformer should bypass the use of structured output functionality of the language model. If set to True, the transformer will not use the language model's native function calling capabilities to handle structured output. Defaults to False. additional_instructions (str): Allows you to add additional instructions to the prompt without having to change the whole prompt. Example: .. code-block:: python from langchain_experimental.graph_transformers import LLMGraphTransformer from langchain_core.documents import Document from langchain_openai import ChatOpenAI llm=ChatOpenAI(temperature=0) transformer = LLMGraphTransformer( llm=llm, allowed_nodes=["Person", "Organization"]) doc = Document(page_content="Elon Musk is suing OpenAI") graph_documents = transformer.convert_to_graph_documents([doc]) NTFr ) llmrrrT strict_moderYrZignore_tool_usager!r"c Cst|||_||_||_||_| |_|jrTz|tWntyRd|_Yn0|js|sb|rjt dzddl } | |_ Wnt yt dYn0|pt |||j| }||B|_ n\z |j} Wntyd} Yn0t|||| ||j} |j| dd} |pt| }|| B|_ dS)NFzThe 'node_properties' and 'relationship_properties' parameters cannot be used in combination with a LLM that doesn't support native function calling.rz^Could not import json_repair python package. Please install it with `pip install json-repair`.T)Z include_raw)r_relationship_typerrr_function_callZwith_structured_outputr?NotImplementedErrorr- json_repair ImportErrorrXchainZ _llm_typeAttributeErrorrsr()selfrrrrTrrYrZrr!rr1ZschemaZstructured_llmr&r&r'__init__s\          zLLMGraphTransformer.__init__)documentconfigr"c s|j}jjd|i|d}jr@ttttf|}t|\}}nt}g}t |t sZ|j }j |}t |trv|g}|D]} t | trz| drz| drz| dsqz|| d| dtf|| d| dtft| d| dtd} t| d| dtd} |t| | | dd qzd d t|D}jrƈjsPjrƈjrd d jDfd d |D}fdd |D}jrƈjdkrfdd |D}nfdd |D}t|||dS)z Processes a single document, transforming it into a graph document using an LLM based on the model's schema and constraints. rQrrrrrrr\r{r|r^cSs g|]}t|d|ddqSrr6r\r r7r&r&r'rmr;z8LLMGraphTransformer.process_response..cSsg|] }|qSr&rr7r&r&r'rrr;csg|]}|jvr|qSr&r^rrlower_allowed_nodesr&r'rsscs0g|](}|jjvr|jjvr|qSr&r{r^rr|rrr&r'rvsr4cs@g|]8}|jj|j|jjfddjDvr|qS)cSs(g|] \}}}|||fqSr&rr8Zs_tZr_tZt_tr&r&r'rsCLLMGraphTransformer.process_response...r{r^rr|rrrr&r'rs  cs*g|]"}|jddjDvr|qS)cSsg|] }|qSr&rr7r&r&r'rr;rr^rrrrr&r'rs r@rAr{) page_contentrZinvokerr rrrsetrrrHrPrrdictraddrr rr r=rrrrr rrrrrr@rAZ nodes_setZ parsed_jsonryrrr&rrr'process_response?sl         z$LLMGraphTransformer.process_response) documentsrr"csfdd|DS)aConvert a sequence of documents into graph documents. Args: documents (Sequence[Document]): The original documents. kwargs: Additional keyword arguments. Returns: Sequence[GraphDocument]: The transformed documents as graphs. csg|]}|qSr&)rr8rrrr&r'rr;zBLLMGraphTransformer.convert_to_graph_documents..r&)rrrr&rr'convert_to_graph_documentss z.LLMGraphTransformer.convert_to_graph_documentsc s|j}jjd|i|dIdH}jrFttttf|}t|\}}nt}g}t |t s`|j }j |}t |tr||g}|D]} | dr| dr| dsq|| d| dtf|| d| dtft| d| dtd } t| d| dtd } |t| | | dd qd d t|D}jrˆjsLjrˆjrd d jDfdd |D}fdd |D}jrˆjdkrfdd |D}nfdd |D}t|||dS)zl Asynchronously processes a single document, transforming it into a graph document. rQrNrrrrrr\rcSs g|]}t|d|ddqSrrr7r&r&r'rr;z9LLMGraphTransformer.aprocess_response..cSsg|] }|qSr&rr7r&r&r'rr;csg|]}|jvr|qSr&rrrr&r'rscs0g|](}|jjvr|jjvr|qSr&rrrr&r'rsr4cs@g|]8}|jj|j|jjfddjDvr|qS)cSs(g|] \}}}|||fqSr&rrr&r&r'rsDLLMGraphTransformer.aprocess_response...rrrr&r'rs  cs*g|]"}|jddjDvr|qS)cSsg|] }|qSr&rr7r&r&r'rr;rrrrr&r'rs r)rrZainvokerr rrrrrrrHrPrrrrrrr rr r=rrrrr rr&rr'aprocess_responsesh         z%LLMGraphTransformer.aprocess_responsecs(fdd|D}tj|IdH}|S)zV Asynchronously convert a sequence of documents into graph documents. csg|]}t|qSr&)asyncio create_taskrrrr&r'rszCLLMGraphTransformer.aconvert_to_graph_documents..N)rgather)rrrtasksresultsr&rr'aconvert_to_graph_documentss  z/LLMGraphTransformer.aconvert_to_graph_documents)N)N)N)N)rBrCrDrerrrHr rrrboolrrrr rrrrrr&r&r&r'rsR3 E Z   X r)r )Nr r*NN)NNNr )NNFNFN)7rrtypingrrrrrrrr r Z)langchain_community.graphs.graph_documentr r r Zlangchain_core.documentsrZlangchain_core.language_modelsrZlangchain_core.messagesrZlangchain_core.output_parsersrZlangchain_core.promptsrrrZlangchain_core.runnablesrZpydanticrrrrrRr%rHr(r.r>r?rFrXrrsrxr}rrrrvrrrr&r&r&r's,         5'   !   n   6   H    4  !