a dg C@sDddlmZddlZddlmZddlmZmZmZm Z m Z m Z ddl Z ddl mZmZmZddlmZmZmZddlmZddlmZerdd lmZmZdd lmZdd lmZdd l m!Z!m"Z"d dddZ#ddddZ$ddddZ%ddddZ&ddddgdZ'Gdd d eZ(Gd!d"d"e(Z)Gd#d$d$e(eZ*dS)%) annotationsN)Path) TYPE_CHECKINGCallableDictListOptionalUnion)&DEFAULT_DEANONYMIZER_MATCHING_STRATEGYAnonymizerBaseReversibleAnonymizerBase)DeanonymizerMappingMappingDataTypecreate_anonymizer_mapping)exact_matching_strategy)get_pseudoanonymizer_mapping)AnalyzerEngineEntityRecognizerNlpEngineProviderAnonymizerEngine)ConflictResolutionStrategyOperatorConfigz'AnalyzerEngine'returnc CsDzddlm}Wn.ty>}ztd|WYd}~n d}~00|S)Nr)rCould not import presidio_analyzer, please install with `pip install presidio-analyzer`. You will also need to download a spaCy model to use the analyzer, e.g. `python -m spacy download en_core_web_lg`.)presidio_analyzerr ImportError)rer }/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain_experimental/data_anonymizer/presidio.py_import_analyzer_engine!sr"z'NlpEngineProvider'c CsDzddlm}Wn.ty>}ztd|WYd}~n d}~00|S)Nrrr)presidio_analyzer.nlp_enginerr)rrr r r!_import_nlp_engine_provider/sr$z'AnonymizerEngine'c CsDzddlm}Wn.ty>}ztd|WYd}~n d}~00|S)Nrr\Could not import presidio_anonymizer, please install with `pip install presidio-anonymizer`.)presidio_anonymizerrr)rrr r r!_import_anonymizer_engine=sr'z'OperatorConfig'c CsDzddlm}Wn.ty>}ztd|WYd}~n d}~00|S)Nrrr%)presidio_anonymizer.entitiesrr)rrr r r!_import_operator_configHsr*ZspacyenZen_core_web_lg)Z lang_codeZ model_name)Znlp_engine_namemodelsc@sHeZdZdZddddddd d d Zd d dddZdd dddZdS)PresidioAnonymizerBasezcBase Anonymizer using Microsoft Presidio. See more: https://microsoft.github.io/presidio/ NTOptional[List[str]]#Optional[Dict[str, OperatorConfig]]Optional[Dict]bool Optional[int]analyzed_fields operatorslanguages_configadd_default_faker_operators faker_seedc s|dur t}tt}t}t}|dur0|n tt|_|rbfddt| D|_ ni|_ |rv| |||d} | } t| j |_||j| d|_||_dS)a Args: analyzed_fields: List of fields to detect and then anonymize. Defaults to all entities supported by Microsoft Presidio. operators: Operators to use for anonymization. Operators allow for custom anonymization of detected PII. Learn more: https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/ languages_config: Configuration for the NLP engine. First language in the list will be used as the main language in self.anonymize(...) when no language is specified. Learn more: https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/ faker_seed: Seed used to initialize faker. Defaults to None, in which case faker will be seeded randomly and provide random values. Ncs"i|]\}}|dd|idqS)Zcustomlambda)Z operator_nameparamsr ).0fieldZfaker_functionr(r r! sz3PresidioAnonymizerBase.__init__..)Znlp_configuration)supported_languages nlp_engine)DEFAULT_LANGUAGES_CONFIGr*r"r$r'listrkeysr4itemsr5 add_operatorsZ create_engineZnlpr> _analyzer _anonymizer) selfr4r5r6r7r8rrrproviderr?r r(r!__init__js4     zPresidioAnonymizerBase.__init__rNone) recognizerrcCs |jj||j|jdS)zsAdd a recognizer to the analyzer Args: recognizer: Recognizer to add to the analyzer. N)rEregistryadd_recognizerr4extendsupported_entities)rGrKr r r!rMsz%PresidioAnonymizerBase.add_recognizerzDict[str, OperatorConfig])r5rcCs|j|dS)zrAdd operators to the anonymizer Args: operators: Operators to add to the anonymizer. N)r5update)rGr5r r r!rDsz$PresidioAnonymizerBase.add_operators)NNNTN)__name__ __module__ __qualname____doc__rIrMrDr r r r!r-ds? r-c@s(eZdZdZd dddddddd ZdS) PresidioAnonymizerz$Anonymizer using Microsoft Presidio.Nstr Optional[str]r.$Optional[ConflictResolutionStrategy]textlanguage allow_listconflict_resolutionrc Cs|dur|jd}n"||jvr6td|d|jdg}|j|D],}|}|d|vrh|dgn|dqFtt|t|j }|jj ||||d} |j | |} |j j || |jd } t|| | } t|| S) a8Anonymize text. Each PII entity is replaced with a fake value. Each time fake values will be different, as they are generated randomly. PresidioAnonymizer has no built-in memory - so it will not remember the effects of anonymizing previous texts. >>> anonymizer = PresidioAnonymizer() >>> anonymizer.anonymize("My name is John Doe. Hi John Doe!") 'My name is Noah Rhodes. Hi Noah Rhodes!' >>> anonymizer.anonymize("My name is John Doe. Hi John Doe!") 'My name is Brett Russell. Hi Brett Russell!' Args: text: text to anonymize language: language to use for analysis of PII If None, the first (main) language in the list of languages specified in the configuration will be used. Nr Language '-' is not supported. Supported languages are: @. Change your language configuration file to add more languages.supported_entityrOentitiesr[r\analyzer_resultsr5)r> ValueErrorrEget_recognizersto_dictrNrAset intersectionr4analyzerF0_remove_conflicts_and_get_text_manipulation_data anonymizer5rr) rGrZr[r\r]rOrKrecognizer_dictentities_to_analyzerefiltered_analyzer_resultsanonymizer_resultsanonymizer_mappingr r r! _anonymizesP    zPresidioAnonymizer._anonymize)NNN)rQrRrSrTrsr r r r!rUs rUcseZdZdZd%dddddd fd d Zed d ddZed d ddZd&ddddddddZe fddddddZ dd ddZ ddd d!d"Z ddd d#d$Z ZS)'PresidioReversibleAnonymizerz/Reversible Anonymizer using Microsoft Presidio.NTr.r/r0r1r2r3cs,|dur t}t|||||t|_dS)N)r@superrIr _deanonymizer_mapping)rGr4r5r6r7r8 __class__r r!rIsz%PresidioReversibleAnonymizer.__init__rrcCs|jjS)zReturn the deanonymizer mapping)rvdatarGr r r!deanonymizer_mapping!sz1PresidioReversibleAnonymizer.deanonymizer_mappingcCsdd|jDS)zcReturn the anonymizer mapping This is just the reverse version of the deanonymizer mapping.cSs$i|]\}}|dd|DqS)cSsi|]\}}||qSr r )r;kvr r r!r=+zNPresidioReversibleAnonymizer.anonymizer_mapping...)rC)r;keyZ inner_dictr r r!r=*szCPresidioReversibleAnonymizer.anonymizer_mapping..)r{rCrzr r r!rr&sz/PresidioReversibleAnonymizer.anonymizer_mappingrVrWrXrYc Cs|dur|jd}||jvr4td|d|jdg}|j|D],}|}|d|vrf|dgn|dqDtt|t|j }|jj ||||d} |j | |} |j j || |jd } t|| | d d } |j| t||jS) aAnonymize text. Each PII entity is replaced with a fake value. Each time fake values will be different, as they are generated randomly. At the same time, we will create a mapping from each anonymized entity back to its original text value. Thanks to the built-in memory, all previously anonymised entities will be remembered and replaced by the same fake values: >>> anonymizer = PresidioReversibleAnonymizer() >>> anonymizer.anonymize("My name is John Doe. Hi John Doe!") 'My name is Noah Rhodes. Hi Noah Rhodes!' >>> anonymizer.anonymize("My name is John Doe. Hi John Doe!") 'My name is Noah Rhodes. Hi Noah Rhodes!' Args: text: text to anonymize language: language to use for analysis of PII If None, the first (main) language in the list of languages specified in the configuration will be used. Nrr^r_r`rarOrbrdT) is_reversed)r>rfrErgrhrNrArirjr4rkrFrlrmr5rrvrPrrr) rGrZr[r\r]rOrKrnrorerprqZnew_deanonymizer_mappingr r r!rs/sT     z'PresidioReversibleAnonymizer._anonymizez%Callable[[str, MappingDataType], str])text_to_deanonymizedeanonymizer_matching_strategyrcCs |jstdd|||j}|S)aDeanonymize text. Each anonymized entity is replaced with its original value. This method exploits the mapping created during the anonymization process. Args: text_to_deanonymize: text to deanonymize deanonymizer_matching_strategy: function to use to match anonymized entities with their original values and replace them. zDeanonymizer mapping is empty.z6Please call anonymize() and anonymize some text first.)rvrfr{)rGrrr r r! _deanonymizesz)PresidioReversibleAnonymizer._deanonymizerJcCs t|_dS)zReset the deanonymizer mappingN)r rvrzr r r!reset_deanonymizer_mappingsz7PresidioReversibleAnonymizer.reset_deanonymizer_mappingzUnion[Path, str]) file_pathrcCst|}|jdvr t|d|jjddd|jdkrxt|d"}tj|j|ddWd q1sl0YnH|j d rt|d"}t j|j|d d Wd n1s0Yd S) a Save the deanonymizer mapping to a JSON or YAML file. Args: file_path: Path to file to save the mapping to. Example: .. code-block:: python anonymizer.save_deanonymizer_mapping(file_path="path/mapping.json") .json.yaml) must have an extension of .json or .yamlT)parentsexist_okrw)indentNrz.ymlF)Zdefault_flow_style) rsuffixrfparentmkdiropenjsondumpr{endswithyaml)rGrZ save_pathfr r r!save_deanonymizer_mappings    2  z6PresidioReversibleAnonymizer.save_deanonymizer_mappingcCst|}|jdvr t|d|jdkr`t|d}t|}Wdq1sT0YnF|jdrt|d }tj|tjd}Wdn1s0Y|j |dS)aLoad the deanonymizer mapping from a JSON or YAML file. Args: file_path: Path to file to load the mapping from. Example: .. code-block:: python anonymizer.load_deanonymizer_mapping(file_path="path/mapping.json") rrrrNr)Loader) rrrfrrloadrrZ FullLoaderrvrP)rGrZ load_pathrZloaded_mappingr r r!load_deanonymizer_mappings    *  .z6PresidioReversibleAnonymizer.load_deanonymizer_mapping)NNNTN)NNN)rQrRrSrTrIpropertyr{rrrsr rrrr __classcell__r r rwr!rt s( Wrt)+ __future__rrpathlibrtypingrrrrrr rZ+langchain_experimental.data_anonymizer.baser r r Z;langchain_experimental.data_anonymizer.deanonymizer_mappingr rrZGlangchain_experimental.data_anonymizer.deanonymizer_matching_strategiesrZ=langchain_experimental.data_anonymizer.faker_presidio_mappingrrrrr#rr&rr)rrr"r$r'r*r@r-rUrtr r r r!s0        WP