+j<6ddlmZddlmZmZddlZddlZddlm Z ddlm Z m Z d0d Z d1d Z d1d Zd2dZd1dZd3dZed4dZed5dZd6dZed7dZed8dZed9dZd:dZd;d"Zdd.Zd?d/ZdS)@) annotations)AnyoverloadN) coo_matrix)Tensordevicea np.ndarrayreturnrc|jjr% tj|S#t$rYnwxYwtj|S)aView a numpy array as a tensor without copying its buffer, which on a corpus of embeddings costs as much as the scoring it feeds. Two kinds of array cannot be viewed and are copied with :func:`torch.tensor` instead: read-only buffers (memmaps, broadcast views), and dtypes with no torch equivalent. Arrays with a negative stride are rejected outright by both routes. The returned tensor aliases the caller's array, so consumers must write only into fresh outputs. )flags writeabletorch from_numpy TypeErrortensorr s OC:\Projects\DSCons\.venv\Lib\site-packages\sentence_transformers\util\tensor.py _wrap_numpyr sU w #A&& &    D  <??s " //list | np.ndarray | Tensorc0t|trtd|Drtjd|DS|r8td|Drtjd|D}nhtj|}nSt|t jrt|}n)t|tstj|}|j r | tj S|S)a Converts the input `a` to a PyTorch tensor if it is not already a tensor. Lists are stacked into one tensor: a list of sparse tensors keeps its sparsity, and a list of numpy arrays is viewed rather than read element by element (see :func:`_wrap_numpy`). Args: a (Union[list, np.ndarray, Tensor]): The input array or tensor. Returns: Tensor: The converted tensor. c3NK|] }t|to|jV!dSN) isinstancer is_sparse.0xs r z%_convert_to_tensor..)s3@@z!V$$4@@@@@@crg|]4}|tj5S)dtype)coalescetorfloat32rs r z&_convert_to_tensor..+s0PPP em D DPPPr c3JK|]}t|tjVdSr)rnpndarrayrs rrz%_convert_to_tensor..,s.<./s777[^^777r r") rlistallrstackrr)r*rrrr%r&rs r_convert_to_tensorr0s!T @@a@@@ @ @ ;PPaPPPQQ Q  3<rFrG row_normsmasknormalized_valuess rnormalize_embeddingsrRos#  Ex",,Z1!,DDD$$&&J ((**J,=,=,?,?VG JOOA..z7HIIII GAJ 222 9%%221gajAAI q=D dy.  "7,=z?P?P Q QQr truncate_dim int | NonecdSrr,r>rSs rtruncate_embeddingsrWY\Y\r torch.TensorcdSrr,rVs rrWrW]`]`r np.ndarray | torch.Tensorc|dd|fS)a Truncates the embeddings matrix. Args: embeddings (Union[np.ndarray, torch.Tensor]): Embeddings to truncate. truncate_dim (Optional[int]): The dimension to truncate sentence embeddings to. `None` does no truncation. Example: >>> from sentence_transformers import SentenceTransformer >>> from sentence_transformers.util import truncate_embeddings >>> model = SentenceTransformer("tomaarsen/mpnet-base-nli-matryoshka") >>> embeddings = model.encode(["It's so nice outside!", "Today is a beautiful day.", "He drove to work earlier"]) >>> embeddings.shape (3, 768) >>> model.similarity(embeddings, embeddings) tensor([[1.0000, 0.8100, 0.1426], [0.8100, 1.0000, 0.2121], [0.1426, 0.2121, 1.0000]]) >>> truncated_embeddings = truncate_embeddings(embeddings, 128) >>> truncated_embeddings.shape >>> model.similarity(truncated_embeddings, truncated_embeddings) tensor([[1.0000, 0.8092, 0.1987], [0.8092, 1.0000, 0.2716], [0.1987, 0.2716, 1.0000]]) Returns: Union[np.ndarray, torch.Tensor]: Truncated embeddings. .Nr,rVs rrWrWs: c=L=( ))r max_active_dimsintcdSrr,r>r^s rselect_max_active_dimsrbsililr NonecdSrr,ras rrbrbrXr cdSrr,ras rrbrbr[r c||S|dkrtd|dt|tjrt j|}|jd}t jt j|t||d\}}t j |}| d|| d||S)a Returns a new tensor with only the top-k values (in absolute terms) of each embedding, all others set to zero. The input embeddings are never modified in place. Args: embeddings (Union[np.ndarray, torch.Tensor]): Embeddings to sparsify by keeping only the largest values. max_active_dims (Optional[int]): Number of values to keep as non-zeros per embedding. `None` keeps all values, returning the embeddings as-is. Raises: ValueError: If `max_active_dims` is 0 or negative. Returns: Union[np.ndarray, torch.Tensor]: A new dense tensor of the same shape, with all but the top-k values per embedding set to zero. If `max_active_dims` is `None`, the embeddings are returned unchanged. Nrz0max_active_dims must be a positive integer, got .)kr9) ValueErrorrr)r*rrshapetopkabsmin zeros_likescatter_gather)r>r^ embedding_dim_ top_indicesselecteds rrbrbs(!^O^^^___*bj)).\*-- $R(MZ * 5 5_m9\9\bdeeeNA{ ++H b+z'8'8['I'IJJJ Or featuresdict[str, Any]c L|d}|dd|ddtjjjfdt |dd|ddDd d |d<tjjjfd t |dd|ddDd d |d<tjd t |dd|ddDj}|dj d}tj |j d| dk|d<dD]}| |d|S)aReverse FA2 input flattening on a features dict, in place. When :class:`Transformer` runs with FA2 unpadding, ``DataCollatorWithFlattening`` flattens the batch into ``token_embeddings: (1, sum_lens, D)`` / ``input_ids: (1, sum_lens)``, drops ``attention_mask``, and writes ``cu_seq_lens_q`` to mark sequence boundaries. This function reverses that: pad ``token_embeddings`` and ``input_ids`` back to ``(B, T_max, ...)``, rebuild ``attention_mask`` from the cumulative lengths, and drop the FA2-specific keys. Caller is responsible for gating on ``cu_seq_lens_q in features``. Args: features: Features dict containing flat FA2 outputs. Mutated in place. Returns: The same ``features`` dict, with the standard ``(B, T_max, ...)`` shape restored. cu_seq_lens_qtoken_embeddingsr input_idsc*g|]\}}||Sr,r,)rseflat_embs rr'z,repad_flattened_features..%88841a!A#888r Nrhr8Tg) batch_first padding_valuec*g|]\}}||Sr,r,)rr}r~flat_idss rr'z,repad_flattened_features..rr cg|] \}}||z  Sr,r,)rr}r~s rr'z,repad_flattened_features..s CCCdaAECCCr rBattention_mask)ry cu_seq_lens_k max_length_q max_length_kseq_idx position_ids) tolistrrCutilsrnn pad_sequenceziprrrkaranger:pop)rvculengthsT_maxkeyrrs @@rrepad_flattened_featuresrs / " ) ) + +B*+A.H $Q'H#(8>#5#B#B88883r#2#w122#7#7888dZ]$C$$H  "HN.;;88883r#2#w122#7#7888dZ[<H[lCCc"SbS'2abb6.B.BCCCHO\\\G [ ! ' *E!&eHO!L!L!L!V!VWX!Y!Y\c\m\mno\p\p!pH l   S$ Or list[Tensor]maskstuple[Tensor, Tensor]ctd|Dtfd|Drfd|D}fd|D}tj|dtj|dfS)aStack a list of ``(B, T_i, D)`` token embeddings and their ``(B, T_i)`` masks along ``dim=1``, padding each column up to the batch-wide max token count. Used by multi-vector / late-interaction losses to assemble a ``(B, N, T_max, D)`` document tensor plus matching ``(B, N, T_max)`` mask. ``F.pad``'s tail-axis padding handles the token dimension (the embedding and batch dims are left alone). Padded positions must be excluded downstream via the returned mask (MaxSim already honours it). Skips the pad and does a plain stack when every column already shares ``T`` (e.g. a single document column). Pads when columns differ, as with independently-padded text columns (different per-column batch-longest) or ragged-token-count VLMs (Qwen2-VL family). c3@K|]}|dVdSr8NrIrr~s rrz0stack_padded_token_embeddings..,..aq ......r c3JK|]}|dkVdSrrrr~rs rrz0stack_padded_token_embeddings..2 2 2!166!99  2 2 2 2 2 2r c g|]A}tjj|ddd|dz fBSrr8rrCrDpadrIrs rr'z1stack_padded_token_embeddings..GcccSTeh)--a!Q5166!99;L1MNNcccr c g|]?}tjj|d|dz f@Srrrmrs rr'z1stack_padded_token_embeddings..BSSS$((Qq 0A,BCCSSSr r8r9)maxanyrr/r>rrs @rstack_padded_token_embeddingsr s ..:... . .E 2 2 2 2z 2 2 222TccccXbccc SSSSUSSS ;zq ) ) )5;u!+D+D+D DDr ctd|Dtfd|Drfd|D}fd|D}tj|dtj|dfS)aConcatenate ``(B_i, T_i, D)`` token-embedding chunks and ``(B_i, T_i)`` mask chunks along ``dim=0``, padding each chunk up to the chunk-wide max token count. Used by GradCache-style multi-vector losses where each mini-batch is encoded independently and the resulting per-mini-batch chunks must be assembled into a single ``(sum(B_i), T_max, D)`` tensor + ``(sum(B_i), T_max)`` mask. Native-resolution VLMs (Qwen2-VL family) may emit a different ``T`` per mini-batch within the same column. Skips the pad entirely when all chunks already share the same ``T`` (text and fixed-resolution VLMs), paying a pad only for ragged-token-count VLMs. c3@K|]}|dVdSrrrs rrz.cat_padded_token_embeddings..)rr c3JK|]}|dkVdSrrrs rrz.cat_padded_token_embeddings..*rr c g|]A}tjj|ddd|dz fBSrrrs rr'z/cat_padded_token_embeddings..+rr c g|]?}tjj|d|dz f@Srrrs rr'z/cat_padded_token_embeddings..,rr rr)rrrcatrs @rcat_padded_token_embeddingsrs ..:... . .E 2 2 2 2z 2 2 222TccccXbccc SSSSUSSS 9ZQ ' ' '5a)@)@)@ @@r batch target_devicerc|D];}t||tr|||||<<|S)au Send a PyTorch batch (i.e., a dictionary of string keys to Tensors) to a device (e.g. "cpu", "cuda", "mps"). Args: batch (Dict[str, Tensor]): The batch to send to the device. target_device (torch.device): The target device (e.g. "cpu", "cuda", "mps"). Returns: Dict[str, Tensor]: The batch with tensors sent to the target device. )rrr%)rrrs rbatch_to_devicer0sH66 eCj& ) ) 6s}55E#J Lr rrcV|}|}|}t ||d|dff|jS)ac Converts a sparse PyTorch tensor to a SciPy COO sparse matrix. Args: x (torch.Tensor): A 2-dimensional sparse PyTorch tensor in COO layout. Example: >>> import torch >>> from sentence_transformers.util import to_scipy_coo >>> x = torch.sparse_coo_tensor([[0, 1], [1, 0]], [1.0, 2.0], (3, 3)) >>> to_scipy_coo(x).toarray() array([[0., 1., 0.], [2., 0., 0.], [0., 0., 0.]], dtype=float32) Returns: scipy.sparse.coo_matrix: A SciPy COO matrix with the same shape, indices and values as the input tensor. rr8)rk)r$rFcpunumpyrGrrk)rrFrGs r to_scipy_coorAs~& Aiikkoo%%''G XXZZ^^   # # % %F v GAJ78 H H HHr c|js|}|}tj|d|jtj}|dkr+d|| <|S|dkrv| dkrJ| }tj |dd\}}|||<|Std|d ) a Compute count vector from sparse embeddings indicating how many samples have non-zero values in each dimension. Args: embeddings: Sparse tensor of shape (batch_size, vocab_size) or (vocab_size,) Returns: Count vector of shape (vocab_size,) rh)rr#r8r@rT) return_countszExpected 1D or 2D tensor, got D)r to_sparser$rrHrIrint32r9rFsqueezerGnumeluniquer_rj)r> count_vectorrF unique_dimscountss rcompute_count_vectorrZs;  ,))++ $$&&J;zr22:;LTYT_```L~~178 Z''))11334   Q        $ $ & & * * ((**G"',wqz"N"N"N K(. L %M*..:J:JMMMNNNr )r r r r)r rr r)r rr r)r>rr r)r>r rSrTr r )r>rYrSrTr rY)r>r\rSrTr r\)r>r\r^r_r rY)r>r r^rcr r )r>rYr^rcr rY)r>r\r^rTr r\)rvrwr rw)r>rrrr r)rrwrrr rw)rrr r)r>rYr rY) __future__rtypingrrrr)r scipy.sparserrrrr0r6r;r=rRrWrbrrrrrrr,r rrs"""""""  ######          @    &         $RRRR: \\\ \ ``` `****@ lll l \\\ \ ``` `%%%%PBEEEE(AAAA&"IIII2OOOOOOr