a ]g P@sddlZddlTddlZddlZddZddZedfddZ d2d d Z d3d d Z e Z d4ddZ ddZeZd5ddZddZddZeZd6ddZeZd7ddZGdddZd8d d!ZGd"d#d#Zed$fd%d&Zd9d(d)ZGd*d+d+Zd,d-ZeZd.d/Ze Z!d:d0d1Z dS);N)*cCstj|dd}|j\}}tj||fdd}tj||fdd}t}t||_t||_||_ ||_ | | |t|| ||fS)zPreturn k smallest values (and their indices) of the lines of a float32 arrayfloat32dtypeint64)npascontiguousarrayshapezerosfaissfloat_maxheap_array_tswig_ptridsvalnhkheapifyaddnreorderarrayrmnIDZharb/var/www/html/cobodadashboardai.evdpl.com/venv/lib/python3.9/site-packages/faiss/extra_wrappers.pykmins   rcCstj|dd}|j\}}tj||fdd}tj||fdd}t}t||_t||_||_ ||_ | | |t|| ||fS)zOreturn k largest values (and their indices) of the lines of a float32 arrayrrr)rrr r r float_minheap_array_tr rrrrrrrrrrrkmax+s   rc Cstj|dd}tj|dd}|j\}}|j\}}||ks|tkr||j|dd<n"t ||t||t|||t||S)zJcompute the whole pairwise distance matrix between two sets of vectorsrrN) rrr empty METRIC_L2Zpairwise_L2sqrr METRIC_INNER_PRODUCTTZpairwise_extra_distances) xqxbmetric metric_argnqdnbd2disrrrpairwise_distances=s*     r-90cCs$tj|dd}tt||j||SNrr)rr Z float_randr sizerseedresrrrrandVsr4cCsBtj|dd}|dur*tt||j|ntt||j|||SNrr)rr Z int64_randr r0Zint64_rand_max)rr2Zvmaxr3rrrrandint\s r6cCs$tj|dd}tt||j||Sr/)rr Z float_randnr r0r1rrrrandnhsr7cCsV|d}|jdkr$t|jt|S|j\}}tj|dd}t||t|t||S)z> compute a checksum for quick-and-dirty comparisons of arrays uint8uint64r) viewndimZ bvec_checksumr0r r rr Zbvecs_checksum)arr)csrrrchecksumns   r?cCs(tj||fdd}t||t|||Sr/)rr rand_smooth_vectors_cr )rr)r2r3rrrrand_smooth_vectorszsrBc Cstj|dd}tj|dd}|jd}|jd|ks8J|jd|jd}}d}t|D]&}|t|t|||t||7}qZ|S)z< size of intersection between each line of two result tablesrrrr9)rrr rangeZranklist_intersection_sizer )ZI1ZI2rZk1Zk2Zninterirrreval_intersections  rEcCs t|jd|jdt|dS)Nr9r)Zfvec_renorm_L2r r xrrr normalize_L2srHcCs|tj|dd}|dur&t|d}tj|ddd}tj|jdd}t|jt| d|t|t||||fS)aPerform a bucket sort on a table of integers. Parameters ---------- tab : array_like elements to sort, max value nbucket - 1 nbucket : integer number of buckets, None if unknown nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) perm : array_like perm[lims[i] : lims[i + 1]] contains the indices of bucket #i (size tab.size) rrNr9r:) rrintmaxr r0 bucket_sort_cr r r;)tabnbucketntlimspermrrr bucket_sortsrQcCsn|jdks|jdksJ|j\}}|dur:t|d}tj|ddd}t||t||t|||S)aPerform a bucket sort on a matrix, recording the original row of each element. Parameters ---------- tab : array_like array of size (N, ncol) that contains the bucket ids, maximum value nbucket - 1. On output, it the elements are shuffled such that the flat array tab.ravel()[lims[i] : lims[i + 1]] contains the row numbers of each bucket entry. nbucket : integer number of buckets (the maximum value in tab should be nbucket - 1) nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) int32rNr9r) rr rIrJrr matrix_bucket_sort_inplace_cr r )rLrMrNZnrowZncolrOrrrmatrix_bucket_sort_inplaces   rTc@s2eZdZdZd ddZddZddZd d Zd S) ResultHeapz_Accumulate query results from a sliced dataset. The final result will be in self.D, self.I.FcCs~tj||fdd|_tj||fdd|_|||_|_|rBt}nt}||_||_t |j|_ t |j|_ | ||_ dS)z nq: number of query vectors, k: number of results per query keep_max: keep the top-k maximum values instead of the minima rrrN)rr rrr(rrr rr rrrheaps)selfr(rkeep_maxrVrrr__init__s  zResultHeap.__init__cCsd|j\}}tj|dd}tj|dd}|j||fks8J||jksFJ|j|t|t||dS)z Add results for all heaps D, I should be of size (nh, nres) D, I do not need to be in a particular order (heap or sorted) rrrN)r rrr(rVZ addn_with_idsr )rWrrr(kdrrr add_results zResultHeap.add_resultcCs|j\}}|t|ksJ|jdkr0|j|jksJ|jdkrF|j|fksJJtj|dd}tj|dd}tj|dd}|jdkrdn|}|j|t||t|t||dS)z Add results for a subset of heaps. D, I should hold resutls for all the subset as a special case, if I is 1D, then all ids are assumed to be the same r9rrrrN)r lenr<rrrVZaddn_query_subset_with_idsr )rWZsubsetrrZnsubsetrZZ id_striderrradd_result_subsets$   zResultHeap.add_result_subsetcCs|jdSN)rVr)rWrrrfinalizeszResultHeap.finalizeN)F)__name__ __module__ __qualname____doc__rYr[r^r`rrrrrUs  rUFc Cs||j|jksJ|j\}}}tj||f|jd}tj||f|jd}|rLtnt}||||t|t|t|t|||fS)z Merge a set of sorted knn-results obtained from different shards in a dataset Dall and Iall are of size (nshard, nq, k) each D[i, j] should be sorted returns D, I of size (nq, k) as the merged result set r)r rr rZmerge_knn_results_CMaxZmerge_knn_results_CMinr ) ZDallZIallrXZnshardrrZDnewZInewfuncrrrmerge_knn_resultss    rfc@s$eZdZddZddZddZdS)MapInt64ToInt64cCsXtt||_|d|jks&Jd||_tj|dfdd|_t|jt |jdS)Nr\zneed power of 2 capacityrr) rIrlog2 log2_capacitycapacityr rLr Zhashtable_int64_to_int64_initr )rWrjrrrrY3s zMapInt64ToInt64.__init__cCs>|j\}|j|fksJt|jt|j|t|t|dSr_)r r Zhashtable_int64_to_int64_addrir rL)rWkeysvalsrrrradd:s  zMapInt64ToInt64.addcCs>|j\}tj|fdd}t|jt|j|t|t||Sr5)r rr r Zhashtable_int64_to_int64_lookuprir rL)rWrkrrlrrrlookupAs zMapInt64ToInt64.lookupN)rarbrcrYrmrnrrrrrg1srgc Cstj|dd}tj|dd}|j\}}|j\}}||ks|dkrt t|t|||||t|t| nt || fS)a Compute the k nearest neighbors of a set of vectors without constructing an index. Parameters ---------- xq : array_like Query vectors, shape (nq, d) where d is the number of bits / 8 `dtype` must be uint8. xb : array_like Database vectors, shape (nb, d) where d is the number of bits / 8 `dtype` must be uint8. k : int Number of nearest neighbors. variant : string Function variant to use, either "mc" (counter) or "hc" (heap) Returns ------- D : array_like Distances of the nearest neighbors, shape (nq, k) I : array_like Labels of the nearest neighbors, shape (nq, k) rRrrrqr9Zmc) r rr r Zint_maxheap_array_trrr rrZhammings_knn_hcZhammings_knn_mcNotImplementedError) r$r%rvariantr(r)r*r+rrheaprrr knn_hammings,     ruc@s<eZdZdZddZddZd ddZdd d Zd d ZdS)KmeansaObject that performs k-means clustering and manages the centroids. The `Kmeans` class is essentially a wrapper around the C++ `Clustering` object. Parameters ---------- d : int dimension of the vectors to cluster k : int number of clusters gpu: bool or int, optional False: don't use GPU True: use all GPUs number: use this many GPUs progressive_dim_steps: use a progressive dimension clustering (with that number of steps) Subsequent parameters are fields of the Clustring object. The most important are: niter: int, optional clustering iterations nredo: int, optional redo clustering this many times and keep best verbose: bool, optional spherical: bool, optional do we want normalized centroids? int_centroids: bool, optional round centroids coordinates to integer seed: int, optional seed for the random number generator cKs||_||d|_d|vr(t|_nt|_|D]H\}}|dkrf|dksX|dkr^t}||_q8t|j|t |j||q8| dS)zd: input dimension, k: nb of centroids. Additional parameters are passed on the ClusteringParameters object, including niter=25, verbose=False, spherical = False FZprogressive_dim_stepsgpuTN) r)resetrwZ"ProgressiveDimClusteringParameterscpClusteringParametersitemsZ get_num_gpusgetattrsetattr set_index)rWr)rkwargsvrrrrYs   zKmeans.__init__cCsp|j}|jjtkrL|jjr&t||_n t||_|jrlt j |j|jd|_n |jr`t |jd}nt }||_ dS)N)Zngpu)r)rz __class__r{ sphericalZ IndexFlatIPindexZ IndexFlatL2rwr Zindex_cpu_to_all_gpusZGpuProgressiveDimIndexFactoryZProgressiveDimIndexFactoryfac)rWr)rrrrrs   zKmeans.set_indexNcCs(|durt||_d|_d|_d|_dS)zg prepare k-means object to perform a new clustering, possibly with another number of centroids N)rIr centroidsobjiteration_stats)rWrrrrrys  z Kmeans.resetc sVtj|dd}|j\}}||jks&J|jjtkrt||j|j}|durr|j\}}||ks`Jt | |j | ||j|nH|dusJ|dusJ|jjrJt||j|j}| |t||jt |j } | |j||_ |jfddtDtddD|_dfddD|_|jjd krR|jd Sd S) a  Perform k-means clustering. On output of the function call: - the centroids are in the centroids field of size (`k`, `d`). - the objective value at each iteration is in the array obj (size `niter`) - detailed optimization statistics are in the array iteration_stats. Parameters ---------- x : array_like Training vectors, shape (n, d), `dtype` must be float32 and n should be larger than the number of clusters `k`. weights : array_like weight associated to each vector, shape `n` init_centroids : array_like initial set of centroids, shape (n, d) Returns ------- final_obj: float final optimization objective rrNcsg|]}|qSr)at).0rD)statsrr >z Kmeans.train..cSsg|] }|jqSr)r)rstrrrr?rz,obj time time_search imbalance_factor nsplitcsg|]fddDqS)csi|]}|t|qSr)r})rfieldrrr Crz+Kmeans.train...r)r) stat_fieldsrrrBsrrxro)rrr r)rzrr{Z Clusteringrr Zcopy_array_to_vectorravelrtrainrrZProgressiveDimClusteringr rZvector_float_to_arrayZreshaperrCr0rrsplit) rWrGweightsZinit_centroidsrr)Zclusncr+rr)rrrr s2         z Kmeans.traincCsZtj|dd}|jdus Jd|j|j|j|j|d\}}||fS)Nrrzshould train before assigningr9)rrrrryrmsearchr)rWrGrrrrrassignHs  z Kmeans.assign)N)NN) rarbrcrdrYrryrrrrrrrvs   Jt|dd}tj||fdd}t||t|t|t||n<||dd}tj||fdd}t|||t|t|||S)a> Pack a set integers (i, j) where i=0:n and j=0:M into n bitstrings. Output is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. If nbit is an integer: all entries takes nbit bits. If nbit is an array: entry (i, j) takes nbit[j] bits. rRrr8) r rrrrIsumr pack_bitstrings_cr )r=nbitrM code_sizebrrrpack_bitstringsZs rcCs|j\}}|durztj|dd}t|}t|dd}||ksHJtj||fdd}t||t|t||t|nL|}||dd}||ksJtj||fdd}t|||t||t||S)a Unpack a set integers (i, j) where i=0:n and j=0:M from n bitstrings (encoded as uint8s). Input is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. Two forms: - when called with (array, M, nbit): there are M entries of size nbit per row - when called with (array, nbits): element (i, j) is encoded in nbits[j] bits NrRrrr) r rrr]rIrr unpack_bitstrings_cr )rZ M_or_nbitsrrrrZ min_code_sizer=rrrunpack_bitstringsus&    r)r.)r.N)r.)r@)Nr)Nr)F)rq)N)"numpyrZ faiss.loaderr collections.abcrrrr!r-r4r6Zlrandr7r?rBrArErHrQrKrTrSrUrfrgrprurvrrrrrrrrr s>       '@ 6 8