+jddlmZddlZddlmZddlmZddlm Z ej e Z ddZ dd ZdddZddZ dddZdS)) annotationsN)logging)is_dist_initializedreturnintcHtrtjSdS)zXThe rank of the current process in the distributed group, or ``0`` when not distributed.r)rdistget_rankTC:\Projects\DSCons\.venv\Lib\site-packages\sentence_transformers\util\distributed.pyr r s!} 1r cHtrtjSdS)zPThe number of processes in the distributed group, or ``1`` when not distributed.r)rr get_world_sizer r rrrs$%"$$$ 1r Ftensor torch.Tensor with_gradboolctr|r%tjj}n[t j}fdt|D}t j|t j}||<tj |dSd}t |S)ap Gathers a tensor from each distributed rank into a list. Always retains gradients for the local rank's tensor, and optionally retains gradients for the gathered tensors if `with_grad` is True. Args: tensor (torch.Tensor): The tensor to gather from each rank. with_grad (bool, optional): If True, the local rank's tensor retains its gradients. Defaults to False. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. c8g|]}tjSr )torch zeros_like).0_rs r zall_gather...s$TTTQ 0 8 8TTTr r)dimzTrying to gather while torch.distributed is not available or has not been initialized, returning the original (local) tensor. This is expected if you are only using one GPU; consider not using gathering to remove this warning.) rr distributednn all_gatherr rranger catlogger warning_once)rrgathered_tensors world_size local_rankwarnings` rrrs 2  2$03>>vFF  ,..JTTTT% BSBSTTT  O,f 5 5 5J+1 Z (y)q1111 S     Mr c$t|dS)a Gathers a tensor from each distributed rank into a list, retaining gradients for the local rank's tensor. Args: tensor (torch.Tensor): The tensor to gather from each rank. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. Tr)r)rs rall_gather_with_gradr*Bs f - - --r mask!tuple[torch.Tensor, torch.Tensor]c rtr tj|d|j}t j|t jjt| }|d|kr|tj j |ddd||dz f}tj j |d||dz f}t||t|fS)afAll-gather a ``(B, T, D)`` token-embedding tensor and its ``(B, T)`` mask across ranks, padding the token axis to the cross-rank max ``T`` first. ``all_gather`` requires every rank to contribute an identically-shaped tensor, but multi-vector batches pad each column to its own per-rank batch-longest ``T``, which differs across ranks. This reduces the global max ``T``, pads both the embeddings and the mask up to it (keeping them aligned), then gathers. Embeddings are gathered with ``with_grad``. The mask never carries a gradient. Args: tensor (torch.Tensor): ``(B, T, D)`` token embeddings to gather. mask (torch.Tensor): ``(B, T)`` mask to gather, padded with ``False`` to match ``tensor``. with_grad (bool, optional): Retain gradients for the embeddings (see :func:`all_gather`). Returns: tuple[torch.Tensor, torch.Tensor]: Gathered ``(sum(B), T_max, D)`` embeddings and ``(sum(B), T_max)`` mask. Without an initialised process group this only forwards to :func:`all_gather` (no padding needed, all_gather returns the local tensor). r)device)oprr))rrrsizer.r all_reduceReduceOpMAXritemr functionalpadr)rr+r local_maxT_maxs rall_gather_paddedr9Ps,LLQ FFF   dm&78888INN$$%% ;;q>>E ! !X(,,VaAuv{{ST~~?U5VWWF8&**4!UTYYq\\5I1JKKD f 2 2 2Jt4D4D DDr )rr)F)rrrrrr)rrrr)rrr+rrrrr,) __future__rrtorch.distributedrr transformers.utilsr environmentr get_logger__name__r"r rrr*r9r r rr@s"""""" &&&&&&,,,,,,  H % %        $$$$$N . . . .AFEEEEEEEr