j9ddlZddlZddlmZmZmZddlZddlmZddl m Z m Z ddl m Z ddlmZddlmZdd lmZmZdd lmZdd lmZd d lmZd dlmZmZerddlmZGddeZ dZ!dZ"Gdde Z#Gdde Z$Gdde Z%Gdde Z&Gdde Z'GddeZ(dS) N)DictList TYPE_CHECKING) BackwardState)compiled_autograd variables) trace_wrapped) unimplemented)%call_module_hooks_from_backward_state) GuardBuilder install_guard) AttrSource)istype)VariableTracker)ConstantVariable EnumVariable)InstructionTranslatorc@eZdZdZdfd ZdZedZxZS)DistributedVariablea{ The base distributed variable that encapsulates common methods for the distributed objects (i.e. ProcessGroup, DeviceMesh, etc.). Concrete distributed objects could inherit this class and add object specific logic. i.e. It provides the check on the distributed package existance and hold the tracking value for the corresponding distributed object. returnNc tjdi|tst d||_dS)Nz+torch.distributed package is not available!)super__init__r is_availabler value)selfrkwargs __class__s TC:\Projects\Historical\venv\Lib\site-packages\torch/_dynamo/variables/distributed.pyrzDistributedVariable.__init__#sO""6""""//11 I G H H H c*t|jSN)typerrs r" python_typezDistributedVariable.python_type)sDJr#c>tjSr%)torch distributedrrr#r"rz DistributedVariable.is_available,s --///r#)rN) __name__ __module__ __qualname____doc__rr( staticmethodr __classcell__r!s@r"rrsq    00\00000r#rc~tsdSddlm}t j|o||juS)NFr)DTensor)rrtorch.distributed.tensorr4inspect isfunction from_local)rr4s r" is_from_localr92sK  + + - -u000000  e $ $ D'2D)DDr#ctsdSddlm}m}m}m}m}|||||g}tj |o||vS)NFr)_get_group_size_by_name_get_group_tag_rank_not_in_group$_resolve_group_name_by_ranks_and_tagget_process_group_ranks) rr"torch.distributed.distributed_c10dr;r<r=r>r?r6r7)rr;r<r=r>r?constant_processgroup_functionss r"is_constant_pg_functionsrB:s  + + - -u , '#  e $ $ Q2Q)QQr#cFeZdZdZedZdddedeffd ZxZ S)WorldMetaClassVariablez Tracks torch.distributed.GroupMember and torch.distributed.group, which are instances of the metaclass _WorldMeta. c\|sdSddlm}t||uS)NFr) _WorldMeta)rr@rFr&)clsrrFs r"is_group_member_typez+WorldMetaClassVariable.is_group_member_typeWs>!! 5AAAAAAE{{j((r#txrnamerc|dkr[t|jd}t|tjt |jjS|dkr[t|jd}t|tjt|jj St ||S)NWORLD)basememberNON_GROUP_MEMBER) rsourcer make_guardr ID_MATCHProcessGroupVariablerrLrrOr var_getattr)rrIrJrPr!s r"rTz"WorldMetaClassVariable.var_getattr`s 7??T[AAAF &++L,ABB C C C' (899 9 ' ' 'T[9KLLLF &++L,ABB C C C ;<< <ww""2t,,,r#) r,r-r.r/ classmethodrHstrrrTr1r2s@r"rDrDQsx ))[) -5 -S -_ - - - - - - - - - -r#rDcLeZdZedZdZ d fd ZxZS) PlacementClassVariablectsdSddlm}t |tuot ||SNFr) Placement)rr(torch.distributed.tensor.placement_typesr[r& issubclassrr[s r"is_placement_typez(PlacementClassVariable.is_placement_typemsO#//11 5FFFFFFE{{d"Cz%'C'CCr#c|jSr%rr's r"as_python_constantz)PlacementClassVariable.as_python_constantw zr#rIrargsList[VariableTracker]r Dict[str, VariableTracker]rrcntj|jddtjfvrj|jrct|j}t |}tj|jddr||d|||St |||S)N__new__r) r6getattr_staticrobjectrhrPPlacementVariable call_methodr call_function)rrIrdr new_objvarr!s r"rmz$PlacementClassVariable.call_functionzs  "4:y$ ? ?FNCT T T  U nnTZ00G#G,,C%dj*dCC Jf=== ww$$Rv666r#)rIrrdrer rfrr)r,r-r.r0r_rbrmr1r2s@r"rXrXlsDD\D7 #7&7- 7  7777777777r#rXc`eZdZedZdZdddedeffd Z dfd Z xZ S)rkcdtsdSddlm}t ||SrZ)rrr\r[ isinstancer^s r" is_placementzPlacementVariable.is_placements?#//11 5FFFFFF%+++r#c|jSr%rar's r"rbz$PlacementVariable.as_python_constantrcr#rIrrJrc|dkrtj|jjSt ||S)Ndim)rcreaterrvrrTrrIrJr!s r"rTzPlacementVariable.var_getattrs; 5==#*4:>:: :ww""2t,,,r#rdrer rfrc&ddlm}gd}||vr t|j}t j|dd Jdt j||}n#t $rd}YnwxYw|tjur|j dSd|D}d| D}|dkr||jg|Ri||S||jg|Ri|} |j | St ||||S) Nr)r)r __setattr__is_shard is_partial is_replicate __getattr__zno custom getattr allowed!c6g|]}|Srrb.0xs r" z1PlacementVariable.call_method..s$999qA((**999r#c>i|]\}}||Srrrkvs r" z1PlacementVariable.call_method..s*KKKDAqa--//KKKr#rz) rr&rr6riAttributeErrorrjrrwitemsrrl) rrIrJrdr rconstant_fold_functions value_typemethod constant_valr!s r"rlzPlacementVariable.call_methods~ '&&&&&# # #  * * * !$*-- *:}dKKSS/TSS / DAA!    ((.'.t44499D999DKKFLLNNKKKF}$$tz3D333F333 !6$*>t>>>v>>L*#*<88 8ww""2tT6:::sA A A*)A*rdrer rfrr) r,r-r.r0rsrbrVrrTrlr1r2s@r"rkrks,,\,-5-S-_------ *;& *; - *;  *;*;*;*;*;*;*;*;*;*;r#rkc`eZdZedZdZdddedeffd Z dfd Z xZ S)DeviceMeshVariablecdtsdSddlm}t ||S)NFr) DeviceMesh)rrtorch.distributed.device_meshrr)rrs r"is_device_meshz!DeviceMeshVariable.is_device_meshs?#//11 5<<<<<<eZ(((r#c|jSr%rar's r"rbz%DeviceMeshVariable.as_python_constantrcr#rIrrJrc|dkrtj|jjS|dkrtj|jjSt ||S)Nndim device_type)rrwrrrrrTrxs r"rTzDeviceMeshVariable.var_getattrs\ 6>>#*4:?;; ; = #*4:+ABB Bww""2t,,,r#rdrer rfrc|dkrNd|D}d|D}tj|jj|i|S|dkr+tj|jS|dkr&t |jS|dkr&t |jSt ||||S)Nsizec6g|]}|Srrrs r"rz2DeviceMeshVariable.call_method..s$???Q!..00???r#c>i|]\}}||Srrrs r"rz2DeviceMeshVariable.call_method..s*QQQ$!QAq3355QQQr#get_coordinate get_group_get_or_create_default_group) rrrwrrrrSrrrrl)rrIrJrdr const_args const_kwargsr!s r"rlzDeviceMeshVariable.call_methods 6>>??$???JQQ&,,..QQQL#*?4:?J+W,+W+WXX X # # ##*4:+D+D+F+FGG G ;  ' (<(<(>(>?? ? 1 1 1' (O(O(Q(QRR Rww""2tT6:::r#r) r,r-r.r0rrbrVrrTrlr1r2s@r"rrs))\)-5-S-_------;& ; - ;  ;;;;;;;;;;r#rcXeZdZdZdZ dfd Zdfd Zed ZxZ S)rSa We don't want a ProcessGroup object to end up in our output graph. But it's common for dynamo to intercept a PG that is then used to get info like rank() or world_size(), as well as passed to utility functions in distributed_c10d which desugar it into plain types like a ranklist and tag. For convenience and proper guarding, we construct a variable type. TODO: make it possible to use ProcessGroupVariable as input to simple functions like _expand_group without dynamo complaining about making a proxy for it. It is not a tensor-like type, and we don't want a proxy- but dynamo assumes torch library functions are dealing with tensor-like types and would have proxies for their args. TODO: should we make this inherit VT instead of UDOV? Do we want any of the default behaviors or just graph-break whenever one of our special cases is not hit? c|jSr%rar's r"rbz'ProcessGroupVariable.as_python_constant rcr#rdrer rfrrc|dkr6tj|jS|dkr6tj|jS|dkr6tj|jSt||||S)Nrankr_get_backend_name) r rrwrrrrrrlrrIrJrdr r!s r"rlz ProcessGroupVariable.call_methods 6>>-44TZ__5F5FGG G 6>>-44TZ__5F5FGG G & & &-44TZ5Q5Q5S5STT Tww""2tT6:::r#rIrcdkr)tjjjSdvrtjfdSt S)N group_name)rrc4||Sr%)rl)rdr rJrrIs r"z2ProcessGroupVariable.var_getattr..#s(8(8T4(P(Pr#)r rrwrrLambdaVariablerrTrxs```r"rTz ProcessGroupVariable.var_getattrs| <  -44TZ5JKK K # # #+PPPPPP ww""2t,,,r#cttsdSddlm}ddlm}t |||fS)NFr) ProcessGroup)FakeProcessGroup)rrtorch._C._distributed_c10dr+torch.testing._internal.distributed.fake_pgrr)rrrs r"is_process_groupz%ProcessGroupVariable.is_process_group(sW#//11 5;;;;;;PPPPPPel,<=>>>r#rrIr) r,r-r.r/rbrlrTr0rr1r2s@r"rSrSs$;& ; - ;  ;;;;;; ------??\?????r#rSc eZdZdZedededefdZdejj dedededdf fd Z d Z d e ed e eefdeffd ZddZxZS)BackwardHookVariablezU Handles torch.utils.hooks.BackwardHook for module-level backward hooks. module user_hooksuser_pre_hooksc tjstddtf fd }|j|d\}|j|\ }|j|\ }|jd||fi}tjj ddd|j j d<t||||S)Nz6module-level backwards hooks require compiled autogradbw_statec tjjdt jt t|ft jt t|fS)a  Rather than installing the user hooks in the graph (which don't survive AotAutograd), we install hooks that will call trace_wrapped in the backward pass that CompiledAutograd can turn into actual hook calls. N)fnr hooks_name module_name)r*utilshooks BackwardHook functoolspartialr r )rruser_hooks_nameuser_pre_hooks_names r"_in_graph_bw_hooksz7BackwardHookVariable.create.._in_graph_bw_hooksCsy;$11%%@!)#2$/ %%@!)#6$/  r#modrmr example_value)rcompiled_autograd_enabledr routputadd_backward_state_hook create_proxyr*rrrnodemetar) rIrrrrbw_state_proxy_proxyrrrs @@@r"rwzBackwardHookVariable.create9s!: T R S S S         :')i&G&GPU&V&V# ^!#!B!B>!R!RQY>>zJJ &&         ,1;+<+I+I$PRTV+W+W (#E6:~NNNr#rrNc rtjdi|||_||_||_||_dS)Nr)rrrrrr)rrrrroptionsr!s r"rzBackwardHookVariable.__init__lsE ##7###  $,r#c|jSr%)rr's r"as_proxyzBackwardHookVariable.as_proxyzrcr#rdr cx|dvr|j||g|Ri|St||||S)N)setup_input_hooksetup_output_hook) _setup_hookrrlrs r"rlz BackwardHookVariable.call_method}sT < < <#4#B>t>>>v>> >ww""2tT6:::r#rIrc ddlm}|||jd|||fiS)Nr) wrap_fx_proxyrl)builderrrrr)rrIhook_method_namerdrs r"rz BackwardHookVariable._setup_hooksa******}  I " " $--//2      r#r)r,r-r.r/r0rrwr*fxProxyrrrrrVrlrr1r2s@r"rr3s- 0O0O$0O( 0O0O0O\0Od -x~ -  -$ - ( -  - - - - - - ;?# ; S/)* ;  ; ; ; ; ; ;         r#r))rr6typingrrrr*%torch.fx.experimental._backward_staterrrr _trace_wrapped_higher_order_opr excr external_utilsr guardsr rrPrrrrMrconstantrrtorch._dynamo.symbolic_convertrrr9rBrDrXrkrrSrrr#r"rs,,,,,,,,,, ??????++++++++::::::BBBBBB00000000!!!!!!44444444EDDDDDD00000/0004EEERRR.-----0---6 7 7 7 7 70 7 7 7F=;=;=;=;=;+=;=;=;@&;&;&;&;&;,&;&;&;R8?8?8?8?8?.8?8?8?v` ` ` ` ` ?` ` ` ` ` r#