a ag~)@s dZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lm Z m!Z!dd l"m#Z#edZ$ededZ%e e$e e$ge%fe e$dddZ&GdddeZ'dS)zo Ensemble retriever that ensemble the results of multiple retrievers by using weighted Reciprocal Rank Fusion N) defaultdict)Hashable)chain) AnyCallableDictIterableIteratorListOptionalTypeVarcast)#AsyncCallbackManagerForRetrieverRunCallbackManagerForRetrieverRun)Document) BaseRetriever RetrieverLike)RunnableConfig) ensure_config patch_config)ConfigurableFieldSpecget_unique_config_specs)model_validatorTH)bound)iterablekeyreturnccs4t}|D]$}||}|vr |||Vq dS)a Yield unique elements of an iterable based on a key function. Args: iterable: The iterable to filter. key: A function that returns a hashable key for each element. Yields: Unique elements of the iterable based on the key function. N)setadd)rrseenekr$k/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/langchain/retrievers/ensemble.py unique_by_key(s  r&c@sJeZdZUdZeeed<eeed<dZe ed<dZ e e ed<e eedd d Zed d eee efed ddZd#e e eeeedddZd$e e eeeedddZe eeedddZe eeedddZdde ee eeedddZdde ee eeedddZeeeeed d!d"ZdS)%EnsembleRetrieveraeRetriever that ensembles the multiple retrievers. It uses a rank fusion. Args: retrievers: A list of retrievers to ensemble. weights: A list of weights corresponding to the retrievers. Defaults to equal weighting for all retrievers. c: A constant added to the rank, controlling the balance between the importance of high-ranked items and the consideration given to lower-ranked items. Default is 60. id_key: The key in the document's metadata used to determine unique documents. If not specified, page_content is used. retrieversweights<cNid_key)rcCstdd|jDS)z+List configurable fields for this runnable.css|]}|jD] }|Vq qdSN) config_specs).0 retrieverspecr$r$r% Qsz1EnsembleRetriever.config_specs..)rr(selfr$r$r%r.NszEnsembleRetriever.config_specsbefore)mode)valuesrcCs,|ds(t|d}d|g||d<|S)Nr)r()getlen)clsr7Z n_retrieversr$r$r% set_weightsUs  zEnsembleRetriever.set_weights)inputconfigkwargsrc Ksddlm}t|}|j|dd|dd|dg|j|di|jd}|jd|fd |d pj|i|}z|j |||d }Wn2t y}z| ||WYd}~n d}~00|j |fi||SdS) Nr)CallbackManager callbacksverboseFtagsmetadatarBZinheritable_tagsZ local_tagsZinheritable_metadataZlocal_metadatanamerun_name run_managerr>) langchain_core.callbacksr@r configurer9rCrDon_retriever_startget_name rank_fusion Exceptionon_retriever_erroron_retriever_end) r4r=r>r?r@callback_managerrIresultr"r$r$r%invoke]s<      zEnsembleRetriever.invokec sddlm}t|}|j|dd|dd|dg|j|di|jd}|jd|fd |d pj|i|IdH}z|j |||d IdH}Wn8t y}z | |IdH|WYd}~n&d}~00|j |fi|IdH|SdS) Nr)AsyncCallbackManagerrArBFrCrDrErFrGrH) rJrUrrKr9rCrDrLrM arank_fusionrOrPrQ) r4r=r>r?rUrRrIrSr"r$r$r%ainvoke~s@       zEnsembleRetriever.ainvoke)queryrIrcCs|||}|S)z Get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. )rNr4rXrIfused_documentsr$r$r%_get_relevant_documentss z)EnsembleRetriever._get_relevant_documentscs|||IdH}|S)z Asynchronously get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. N)rVrYr$r$r%_aget_relevant_documentssz*EnsembleRetriever._aget_relevant_documents)r>)rXrIr>rcsRfddt|jD}tt|D]}dd||D||<q(||}|S)z Retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. c s6g|].\}}|tjd|dddqSZ retriever_r8)tag)rA)rTr get_childr/ir0r>rXrIr$r% sz1EnsembleRetriever.rank_fusion..cSs*g|]"}t|tr"ttt|dn|qS) page_content) isinstancestrrr r/docr$r$r%rcs) enumerater(ranger:weighted_reciprocal_rankr4rXrIr>Zretriever_docsrarZr$rbr%rNs   zEnsembleRetriever.rank_fusioncs^tjfddt|jDIdH}tt|D]}dd||D||<q4||}|S)z Asynchronously retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. c s6g|].\}}|tjd|dddqSr])rWrr_r`rbr$r%rcsz2EnsembleRetriever.arank_fusion..NcSs$g|]}t|tst|dn|qSrd)rfrrhr$r$r%rcs)asynciogatherrjr(rkr:rlrmr$rbr%rVs   zEnsembleRetriever.arank_fusion) doc_listsrcst|tjkrtdttt|jD]P\}}t|ddD]:\}}jdur\|jn |j j||j 7<qBq.t |}t t|fdddfddd }|S) a Perform weighted Reciprocal Rank Fusion on multiple rank lists. You can find more details about RRF here: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf Args: doc_lists: A list of rank lists, where each rank list contains unique items. Returns: list: The final aggregated list of items sorted by their weighted RRF scores in descending order. zFsz.Tcs jdur|jn |jjSr-rrrsZ rrf_scorer4r$r%rtMs)reverser)r:r) ValueErrorrfloatziprjr,rerDr+r from_iterablesortedr&)r4rpZdoc_listweightZrankriZall_docsZ sorted_docsr$rur%rl s0    z*EnsembleRetriever.weighted_reciprocal_rank)N)N) __name__ __module__ __qualname____doc__r r__annotations__rxr+intr,r rgpropertyrr.r classmethodrrr<rrrTrWrr[rr\rNrVrlr$r$r$r%r'9sZ      "  %   . - r')(rrn collectionsrcollections.abcr itertoolsrtypingrrrrr r r r r rJrrZlangchain_core.documentsrZlangchain_core.retrieversrrZlangchain_core.runnablesrZlangchain_core.runnables.configrrZlangchain_core.runnables.utilsrrZpydanticrrrr&r'r$r$r$r%s    ,     $