Nj/8UddlmZmZddlmZddlZddlmZddl m Z ddl m Z ddl mZmZmZddlmZmZGdd eZdaedzed <d ed dfd Z ddejdedzd eejedzffdZ ddejdededzd ejfdZ ddejdedededejdedzd ejfdZ ddejdedededzd ejf dZ ddejdedzd eeje e ffdZ! ddededzdedzd ejfdZ"dS))ABCabstractmethod)AnyN) ShardedTensor)Shard)_all_gather_dtensor_create_chunk_dtensor_create_chunk_sharded_tensor) DeviceMeshDTensorceZdZdZedejdeejedzffdZ edejdedejfdZ e ddejde d e d e d e j d ejdzdejfd Zedejde dedejfdZedejdeejeeffdZedededzdejfdZdS)FSDPExtensionsz This enables some customizable hooks to enable composability with tensor parallelism. To activate these hooks, use :func:`_set_fsdp_extensions` to set a custom :class:`FSDPExtensions` that implements the hooks. tensorreturnNcdS)z6E.g. converting ``DistributedTensor`` to local tensor.Nselfrs UC:\Projects\DSCons\.venv\Lib\site-packages\torch\distributed\fsdp\_fsdp_extensions.pypre_flatten_transformz$FSDPExtensions.pre_flatten_transforms param_extensioncdS)z6E.g. converting local tensor to ``DistributedTensor``.Nr)rrrs rpost_unflatten_transformz'FSDPExtensions.post_unflatten_transforms  rrank world_sizenum_devices_per_nodepgdevicecdS)z6Shards a tensor to chunks and returns the local chunk.Nr)rrrrrrrs r chunk_tensorzFSDPExtensions.chunk_tensor(  r device_meshcdS)zAShards a tensor/DTensor to DTensor and returns the local DTensor.Nr)rrrr#s r chunk_dtensorzFSDPExtensions.chunk_dtensor5s  rcdS)z This is to be called before loading a *sharded* model state dict and should return the tensor and list of shards from which to load data. Nrrs rpre_load_state_dict_transformz,FSDPExtensions.pre_load_state_dict_transform?s  r parent_meshcdS)z This is to be called before loading a *sharded* DTensor state dict. This gathers tensor in FSDP dimension and returns local tensor of TP DTensor. Nr)rrr(s rall_gather_dtensorz!FSDPExtensions.all_gather_dtensorJr"rN)__name__ __module__ __qualname____doc__rtorchTensortuplerrrintdist ProcessGrouprr!r r%listrr'r r*rrrrrs     u|S4Z' (   ^          ^ '+          "      t#       ^            ^     u|T%[( )   ^      $&       ^    rr _extensions flattenerrc |adSr+)r7)r8s r_set_fsdp_extensionsr:[s KKKrrfsdp_extensioncJ|||\}}|||fS|dfSr+)r)rr; new_tensorrs r_ext_pre_flatten_transformr>`s=!&4&J&J6&R&R# O  &. . 4<rrc:|||||S|Sr+)r)rrr;s r_ext_post_unflatten_transformr@ks* !o&A66vOOO Mrrrrrc@||jnt}||||||Sr+)r!r )rrrrrr;chunk_tensor_fns r_ext_chunk_tensorrCusB  % ## ) ?    rr#c<||jnt}||||Sr+)r%r )rrr#r;chunk_dtensor_fns r_ext_chunk_dtensorrFs=  % $$ "     rc|||St|turtdt||}||fS)NzExpected ShardedTensor, got )r'typerAssertionError local_shards)rr;shardss r"_ext_pre_load_state_dict_transformrLsh!;;FCCC F||=((JDLLJJKKK  " "F F rr(c:||jnt}|||Sr+)r*r)rr(r;all_gather_dtensor_fns r_ext_all_gather_dtensorrOs2  % ))  !  5 55rr+)#abcrrtypingrr0torch.distributed distributedr4+torch.distributed._shard.sharded_tensor.apir-torch.distributed._shard.sharded_tensor.shardr#torch.distributed.fsdp._shard_utilsrr r torch.distributed.tensorr r rr7__annotations__r:r1r2r>r@r3r5rCrFr6rLrOrrrrYs+######### EEEEEE?????? 98888888E E E E E SE E E P&* ^d ")))Nt-1 L"T) 5<t #$-1 L#T) \  -1  L     #T)  \4-1  L #T)   \ (-1   L "T)  5<e $%     -1 6 6  6d" 6#T) 6 \ 6 6 6 6 6 6r