[j+$ddlmZddlZddlmZddlmZmZmZerddl m Z erddl Z dd Z dd Z d d Zd dZddZd!d"dZdZd#dZd$dZd%dZd%dZd%dZdS)&) annotationsN) TYPE_CHECKING)is_torch_availableis_torch_distributed_availableis_torch_greater_or_equal)DistributedConfigreturnboolc^tsdStjS)NF)rtorch distributedis_initializedLC:\Projects\DSCons\.venv\Lib\site-packages\transformers\distributed\utils.py!_is_torch_distributed_initializedrs* ) + +u   + + - --rcNtsdSddlm}t||S)NFr)DTensor)rtorch.distributed.tensorr isinstance)objrs r is_dtensorr$s7 ) + +u000000 c7 # ##rintc^tsdStjS)Nr)rrrget_rankrrr_get_torch_distributed_rankr,s* , . .q   % % ' ''rc^tsdStjS)Nr )rrrget_world_sizerrr!_get_torch_distributed_world_sizer!2s* , . .q   + + - --rcto0ttjdddkS)N LOCAL_RANKz-1r)rrosenvirongetrrris_local_dist_rank_0r'8s2 , . . _3rz~~lTX7Y7Y3Z3Z^_3__r device_type str | Nonectjs-|#tjj} t tjd}t tjd}t tjd}ddddd d d }| |}d}|d kr=tt| |tj ||}tj |||| dS#t$r}td|d}~wwxYwdS)zInitialize torch.distributed if not already initialized. If `device_type` is not given, it is detected from the current accelerator. NRANKr# WORLD_SIZEncclglooxcclhcclneurontpu_dist)cudacpuxpuhpur1tpur4)backendrank world_size device_idzWe tried to initialize torch.distributed for you, but it failed. Make sure you init torch distributed in your script to use distributed training.)rrr_C_get_acceleratortyperr$r%r&getattr set_devicedeviceinit_process_group ExceptionOSError)r(r9 local_rankr: backend_mapr8r;es r_ensure_torch_distributedrH<sg   + + - -   (3355:K rz&)**DRZ 566JRZ 566J"! K"ook22G Ie##{++66zBBB!LjAA   0 0dzY 1        ^  9  sC#D++ E 5EE cDtsdStjj}|dkrHtjtt|gdStjdS)u3Barrier bound to the current accelerator device. Passing `device_ids` is required when the process group was initialized without a `device_id`; with it, the call is a no-op compared to plain `barrier()`. Safe to call when torch.distributed has not been initialized — returns immediately. Nr4) device_ids) rrr<r=r>rbarrierr?current_device)r(s r_distributed_barrierrMds - . .(++--2Ke !!ge[.I.I.X.X.Z.Z-[!\\\\\ !!#####rdistributed_configr c|jdkrtdstdtjj}|dkrkttj dd}tt| |tj ||}ntj |}|j}gg}}|dkr*|||dtj|t#|t#|}t%|dkr(|d |||fS) Nr 2.7z?FSDP2 requires `torch>=2.7` (distributed checkpoint save/load).r4r#rfsdp)mesh_dim_names_) fsdp_sizerrDrr<r=r>rr$r%r&r?r@rAappendrinit_device_meshtuplelen_flattenjoin)rNr(rE device_maprTdimsnamesmeshs r)initialize_fully_sharded_data_parallelismr_tsP#a''0I%0P0P'WXXX(++--2Ke a8899 {##..z:::\+z:: \+.. ",Ib%D1}} I V   - -k5;;W\]bWcWc - d dD 4yy1}} chhuoo&&& t rdict[str, torch.Tensor]ctdstdddlm}m}|dd}|||}t dkr|SiS)z~Gather FSDP-sharded params to full plain CPU tensors. Only rank 0 accumulates the result; other ranks return ``{}``. rP0Distributed checkpointing requires `torch>=2.7`.r)StateDictOptionsget_model_state_dictT)full_state_dict cpu_offload)options)rrD'torch.distributed.checkpoint.state_dictrcrdr)modelrcrdrgres rgather_full_state_dictrjs %U + +JHIII_^^^^^^^tFFFG**5'BBBO"$$)) Ircheckpoint_dirstrNonectdstdddlmcm}ddlm}ddlm}||}| |||dd tdS) uSave model parameters as standard HF-format sharded safetensors using DCP + HuggingFaceStorageWriter with consolidation enabled. Every rank first writes its own shard in parallel under `/sharded/`, then a consolidation pass reads those shards and emits HF-compatible `model-*-of-N.safetensors` (+ index) at `/`. The result is a directory `from_pretrained` reads through its normal path — no special flag needed at load time. rPrbrN)HuggingFaceStorageWriter)rdT)pathsave_distributedenable_consolidation)storage_writer) rrDtorch.distributed.checkpointr checkpoint'torch.distributed.checkpoint.hf_storagerorhrdsaverM)rirkdcprord state_dicts r!save_model_checkpoint_distributedrzs %U + +JHIII/........PPPPPPLLLLLL%%e,,JHH//!!%    rctdstdddlmcm}ddlm}|||}|d|i|dS)zSave optimizer state via DCP.rPrbrN)get_optimizer_state_dict optimizer checkpoint_id)rrDrtrrurhr|rw)rir}rkrxr|optimizer_state_dicts rsave_optimizer_distributedrs $U + +JHIII/........PPPPPP33E9EEHHk/ 0HOOOOOrctdstdddlmcm}ddlm}m}|||}|d|i|||||dS)zLoad optimizer state via DCP.rPrbrN)r|set_optimizer_state_dictr}r~) rrDrtrrurhr|rload)rir}rkrxr|rrs rload_optimizer_distributedrs $U + +JHIII/........jjjjjjjj33E9EEHHk/ 0HOOOUI/CDDDDDr)r r )r r)N)r(r))rNr )r r`)rkrlr rm) __future__rr$typingrutilsrrrconfiguration_utilsr rrrrr!r'rHrMr_rjrzrrrrrrs#""""" aaaaaaaaaa7666666LLL.... $$$$(((( .... ````%%%%%P $ $ $ >&B P P P P E E E E E Er