jݴddlZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z mZddlmZddlZddlmZmZddlmZmZddlmZmZddlmZmZdd lmZm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*d d l+m,Z,m-Z-m.Z.d d l/m0Z0 ddl1m2Z2n #e3$rdZ2YnwxYwe rddl4m5Z5ddl6m7Z7m8Z8edeZ9dGdde.Z?Gdde?Z@Gdde@ZAGd d!eAZBGd"d#e@ZCd=d$ZDGd%d&e?ZEGd'd(e.ZFGd)d*e.ZGd+ZHd=d,ZIGd-d.e@ZJGd/d0e.ZKGd1d2e.ZLdd3l6mMZMmNZNGd4d5eNZOeOZPGd6d7e.ZQGd8d9e.ZRGd:d;e.ZSdS)>N)AnyCallableDictListOptionalTuple TYPE_CHECKINGTypeVar)Never) polyfills variables)create_call_function create_rot_n) unimplemented Unsupported) GuardBuilder install_guard) AttrSourceConstantSourceDefaultsSource GetItemSource)check_constant_argscheck_unspec_or_constant_argsidentity is_functionis_wrapper_or_member_descriptoristype make_cell)typestrValueMutationNewVariableTrackerConstantVariable)_fsdp_param_group)InstructionTranslator)TritonGridTypeTritonKernelType_F)boundtxr'ct|tr|S|stj||Stj||SN) isinstancer#buildrLazyVariableTrackercreate)r,valsources RC:\Projects\Historical\venv\Lib\site-packages\torch/_dynamo/variables/functions.pywrap_bound_argr61sQ#''A A$R---,33C@@@ct|D]4\}}t|ttfrt ||||<5dSr.)listitemsr/tupledictr6)r,resultkvs r5wrap_args_kwargsr@=sYV\\^^$$..1 a% ' ' .&r1--F1I..r7r=c|jj}|jD]H}|}||vr)|||||||<IdS)z Update `result` to add mapping from local name to new cells created directly by `code`, or update SideEffects in `parent` if the a local cell is already in `result` (cell argument). N)output side_effects co_cellvarstrack_cell_new store_cellpop)parentr=coderCnamenew_cells r5 init_cellvarsrLDss =-L   ..00 6>>  # #Hfjj.>.> ? ? ?t   r7cddlm}||||||}||_t|trddlm} t| |}|t|tsJ||_|S)Nr) FunctionTypepairwise) typesrN__kwdefaults__r/r; itertoolsrPr<__annotations__) rI f_globalsrJdefaultsclosure kwdefaults annotationsrNfuncrPs r5_create_nested_fnr[Us#""""" <ix A AD$D+u%%2&&&&&&88K0011   *[$"?"?   ?&D Kr7cReZdZdZdZ dd Zddd ed efd ZdZ dZ dS)BaseUserFunctionVariablec4|jSr.)get_code co_filenameselfs r5 get_filenamez%BaseUserFunctionVariable.get_filenamejs}}**r7c4|jSr.)r_co_nameras r5get_namez!BaseUserFunctionVariable.get_namems}}&&r7r,r'argsList[VariableTracker]kwargsDict[str, VariableTracker]returnr#c\||g|||Sr.)inline_user_function_return self_argsrbr,rgris r5 call_functionz&BaseUserFunctionVariable.call_functionps2 --d4Ndnn6F6F4N4NPVWWWr7rJcd} t||}n-#t$r |dkrt|trd}YnwxYwt j|S)NF__name__T)hasattr get_functionNotImplementedErrorr/NestedUserFunctionVariablerr%r2rbr,rJr=s r5 call_hasattrz%BaseUserFunctionVariable.call_hasattrxs T..00$77FF"   z!!j7Q&R&R! )00888s"''AAcrttj|jSr.)r9inspect signaturert parametersras r5inspect_parameter_namesz0BaseUserFunctionVariable.inspect_parameter_namess*G%d&7&7&9&9::EFFFr7ciSr.)rbr,s r5 closure_varsz%BaseUserFunctionVariable.closure_vars r7Nr,r'rgrhrirjrkr#) rr __module__ __qualname__rcrfrpstrr#rxr}rrr7r5r]r]is+++'''X #X&X- X  XXXX969c9o9999GGGr7r]ceZdZdZddhejZedZddfd Zfd Z d Z d Z d Z d Z dZdZdeeeffdZdddefdZdddedefdZ dfd ZxZS)UserFunctionVariablez-Some unsupported user-defined global functionfn is_constantctt|tj|||SNr4)r make_guardr CLOSURE_MATCHclsvaluer4s r5create_with_sourcez'UserFunctionVariable.create_with_sources6f'' (BCCDDDs5((((r7FrkNc Dtjdi|t|ddrd|_nd|_t |t jtjj fsJdt|d|tj |d|}||_ dS)N_dynamo_marked_constantFTzexpected FunctionType found  _torchdynamo_inliner)super__init__getattrrr/rQrNtorchjitScriptFunctionr!rzgetattr_staticr)rbrrri __class__s r5rzUserFunctionVariable.__init__s""6""" 20% 8 8 %#D  $D  #UY%=>   = = <'"++ < < < < = =  #B(=r B B&(r7c|t|tr|jStSr.)rrrras_python_constantrbrs r5rz'UserFunctionVariable.as_python_constants3 $, - - 7Nww))+++r7cgSr.rras r5rnzUserFunctionVariable.self_argsrr7c|jSr.rras r5rtz!UserFunctionVariable.get_function wr7c|jjSr.)r__code__ras r5r_zUserFunctionVariable.get_codes wr7ctjSr.)rQrNras r5 python_typez UserFunctionVariable.python_types !!r7c2t|jddduS)N__self__)rrras r5has_selfzUserFunctionVariable.has_selfstw D11==r7c|jjSr.)r __globals__ras r5 get_globalsz UserFunctionVariable.get_globalss w""r7c jrJ|jj}tjt |j}|jpg}fdt|D}tj |j |j |j tfdt||D|j}|jr>fd|jDfd|jD|_t'j|j|i|} | t/| j} t3|| t5|| |j jjpd} t7| t7jj jksJtt;jjj j| D] \} } }|jj}||vr ||}nj rtCtEj d| }tE|d} tGj$||j%|}n##tL$rtOj(}YnwxYw|)|||}nV tGj$||j%}n##tL$rtOj(}YnwxYw|)d ||}|| | <| S) z Assume `args` and `kwargs` are VariableTracker arguments for a call to this function, create new bindings for initial locals. )r,cRg|]#\}}jdntj|$Sr.r4r).0idx_rbs r5 z2UserFunctionVariable.bind_args..sD   QK'DD^DK-M-M   r7c0g|]\}}||S)r3r4r)rargr4wraps r5rz2UserFunctionVariable.bind_args..s<#VDS000r7cRi|]#}|jdntj|d$S)NT)is_kwr)rr>rbs r5 z2UserFunctionVariable.bind_args..sN"""  {*D' QdCCC """r7c>i|]\}}|||Srr)rr>r?kwdefaults_sourcesrs r5rz2UserFunctionVariable.bind_args..sD(((Aq44A&8&;<<<(((r7r __closure__ cell_contentsN)*rrBroot_tx functoolspartialr6r __defaults__ enumeraterQrNrrrrr;ziprrRr:rzr{bindapply_defaultsr< argumentsr@rLlen co_freevarsrScountrCr4rrr#r0r ValueErrorrDeletedVariabletrack_cell_existing)rbrHrgrirrrVdefaults_sources fake_funcr+r=rWrrJcellrCcell_var closure_cellclosure_cell_contents contents_varrrs` @@r5 bind_argszUserFunctionVariable.bind_argss ####-' G<<<!%?(b    #H--   & K N K '*85E'F'F   N      """" * """ (((((-3355(((I $ 2!),,14B6BB eo++--..&)))ffbk222'%+7||s47#3#?@@@@@@" O  tw/;W  " $" $OCt"=5L|##'- V,t{M::C   )3<(Q(Q%?#2#8 24I$$LL"???#,#<#>#>LLL?(;; $ ?#2#8AS#T#TLL!???#,#<#>#>LLL?(;;D$ UU#F4LL s$II=<I=J44KKr,r'rJc.|jot|j|} tj|j|}n'#t $rt j|||cYSwxYw|r t j ||Stj ||Sr) r4rrzrrAttributeErrorrGetAttrVariabler1r2r#r0)rbr,rJr4subobjs r5 var_getattrz UserFunctionVariable.var_getattrs>DK!>!> H+DGT::FF H H H,T4GGG G G G H  H077GG G$R000s9!AAcjt|j|}tj|Sr.)rsrrr%r2rws r5rxz!UserFunctionVariable.call_hasattr!s*$'')00888r7rgrhrirjr#c|jr*t||j|||S|jjjr|jjjs ddlm }n#t$rd}YnwxYw|t|j|j |j fvr_tjj|5t!|||cdddS#1swxYwYt!|||S)Nr) FSDPState)rinvoke_and_store_as_constantrrfrBcurrent_tracerunder_activation_checkpoint#allow_side_effects_under_checkpoint/torch.distributed.fsdp._fully_shard._fsdp_stater Exception _pre_forward _post_forwardr_dynamorCrrp)rbr,rgrirrs r5rpz"UserFunctionVariable.call_function%s{   /DGT]]__dF  I $ @ CI,P C !UUUUUUU ! ! !  !$&'5**]/SSTVWWCC 7700T6BBCCCCCCCCCCCCCCCCww$$Rv666s$A A,+A,*#CC!C)FrkNr)rrrr__doc__r]_nonvar_fields classmethodrrrrnrtr_rrrrrr#rrrxrp __classcell__rs@r5rrs77  " 0N ))[))))))))",,,,,    """>>>###Uc?6J1KUUUUn151S1111969c9o99997 #7&7- 7  7777777777r7rcbeZdZdZdfd ZdefdZdZdZ dfd Z fdZ xZ S)UserMethodVariablez$Some unsupported user-defined methodrkNc Ltjdd|i|||_dS)Nrr)rrobj)rbrrrirs r5rzUserMethodVariable.__init__Cs0))B)&)))r7c@|jjd|jd|jdS)N(z, ))rrrrrras r5__repr__zUserMethodVariable.__repr__Gs*.)BBDGBBtxBBBBr7c|jgSr.)rras r5rnzUserMethodVariable.self_argsJs zr7ctjSr.)rQ MethodTyperas r5rzUserMethodVariable.python_typeMs r7r,r'rgrhrirjr#c|jrt|jtjrit |jdd}||dr|dks|j r.|j ||jj |||j SnVtO|jtj jur7t j|j||jg|R|S|j rIt |jj|jj }t%|||||St)|||S)Nrz torch.nn.ztorch.nn.utils.parametrize)constant)rBis_root_tracerr/rrNNModuleVariablerr startswithr call_methodrrr&FSDPParamGroupuse_training_stateTorchCtxManagerClassVariablerprrrfr)rbr,rgri module_attrrrs r5rpz UserMethodVariable.call_functionPsp$ 9 # # % % * Hi0+ +  "$'<<.converts; a1 2 2 &##%% %##%%%r7c&g|] }|Srr)rrrs r5rz0invoke_and_store_as_constant..s! % % %1GGAJJ % % %r7c.i|]\}}||Srr)rr>r?rs r5rz0invoke_and_store_as_constant..s' 7 7 71a 7 7 7r7r)r:rBregister_attr_or_moduler)r,rrJrgriresrs @r5rrs&&& & % % % % % %D 7 7 7 7  7 7 7F "d f  C 9 , ,  d## -  r7cjeZdZdhejZ d dfd ZdZdZdZdZ d Z d Z d Z d Z xZS)rvrUNrkc  tjdi| t|tsJt|t jsJt|tsJ||_||_ ||_ ||_ ||_ ||_ ||_||_dSNr)rrr/rrrQCodeTyper<fn_namerIrUrVrXrYrW wrapped_fn) rbr(rIrUrVrXrYrWr)rirs r5rz#NestedUserFunctionVariable.__init__s ""6"""'4466<<<<<$1133U^DDDDD)T*****  "  $& 5?r7cgSr.rras r5rnz$NestedUserFunctionVariable.self_argsrr7c4|jSr.)rIrras r5r_z#NestedUserFunctionVariable.get_codesy++---r7c*|jrttj|j|j|j}|jr|j|_ |j r|j |_ |j rj|j }t|trddlm}t#||}t|t"sJ||_|S)NrrO)rWrurQrNrIrrUr(rVrrXrRrYr/r;rSrPr<rT)rbrZrYrPs r5rtz'NestedUserFunctionVariable.get_functions < &% %! I ( ( * * N L + + - -   = C $ @ @ B BD  ? G"&/"D"D"F"FD    /*==??K+u-- :......"88K#8#899 k400 0 00#.D  r7c|jduSr.)rWras r5 has_closurez&NestedUserFunctionVariable.has_closures|4''r7cdS)NFrras r5rz#NestedUserFunctionVariable.has_selfsur7c|jSr.)rUras r5rz&NestedUserFunctionVariable.get_globalss ~r7c&|}tj||j|j|jrt|jjndtdtt|j D}|j r|j |_tj|j|i|}|t'|j}t+|jj|t1|||t3|j D]"\}} | |vsJ|jj|} | || <#|S)Nc34K|]}tdVdSr.)r)rrs r5 z7NestedUserFunctionVariable.bind_args..s(SSa)D//SSSSSSr7)r_rQrNrUr(rrVr;r:rangerrrXkeys_as_python_constantrRrzr{rrr<rr@rBrrLrrW) rbrHrgrirIrZr+r=rrJrs r5rz$NestedUserFunctionVariable.bind_argssr}}!  N L + + - -*.- AE$-% & & &T SS5T]]__5P1Q1Q+R+RSSS S S    ? L"&/"I"I"K"KD ,!$'',d=f== eo++--...777ffd+++"4#344  ICv%%%%<%c*DF4LL r7c2fd|j|jgt j|jjj|j r|j n) dg|j r|j n) dg|j r|j n) dg|j rd |j }|gnI#t$r|j Yn-wxYw dgt!dd|jrfd|jt!ddt%dt!dddSdS)Nc:tdS)Nr[)load_import_fromrrcodegensr5z8NestedUserFunctionVariable.reconstruct..sG,,X7JKKr7Fc0ddS)Nrwrapsr8r9sr5r;z8NestedUserFunctionVariable.reconstruct..>s00gFFr7r r T) add_push_nullrI extend_outputcreate_load_const_uncheckedrUr%r2rrerVcreate_load_constrWrXrYrrurr)r)rbr:rYs ` r5 reconstructz&NestedUserFunctionVariable.reconstructs K K K K     wBB4>RRSTTT ' (?@@AAA = E GDM " " " "  ! !7#<#.ts$777Q!&&((777r7c>i|]\}}||SrrUrr>r?s r5rz6SkipFunctionVariable.call_function..us*HHHA1a**,,HHHr7) mutation_typer ct|tjr|dSt d|ddS)Nr)r)zfunctools.wraps(r)r/rrvcloner)rrgs r5r>z1SkipFunctionVariable.call_function..wraps|sPb)"FGG888tAw8777666677777r7z'skip function z in file '_abc _warningsz.Graph break due to unsupported Python builtin .zM. Please file an issue on GitHub so the PyTorch team can add support for it. optreez)Graph break for an optree C/C++ function zj. Consider using torch.utils._pytree - https://github.com/pytorch/pytorch/blob/main/torch/utils/_pytree.pyz'Graph break due to unsupported builtin a. This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind). If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround. If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use torch.compiler.allow_in_graph.allow_in_graphzFound an allow_in_graph decorator to a function which is created inside the parent function that is getting compiled. This is not supported for now.z', r)rzrrrrQkeysrr:getr"rr>rrLambdaVariablegetfiler TypeErrorrrrrutils warn_oncerG) rbr,rgrirr>pathmsgknown_python_builtin_moduless ` r5rpz"SkipFunctionVariable.call_functiones   !$*.De L LD  W4:WW X X X X X J$??AAFFHH H H#D&11 IDJ77$777HHHHHEK488::>>tzJJ%5%7%7 Z9? * *6 *c$ii1nn 8 8 8 8 8 +E22 2$ 7tz22Q (?QQ$QQQ! 7! 7! 706 /D,:(,HHHHI^HHaeakaxHHHC J)5 -88BB6_DJDY__\`\f\s___ M'11#6666 :$*BW : :Z^ZdZq : : :M'11#666C! 7Dz&*:::? 4;>'''''B >C #     s+,EC(IIr.rr)rrrrr#rrrrr staticmethodr lru_cacherQrprrs@r5rFrFFs  'N  ))[)Y  \ J #J&J- J  JJJJJJJJr7rFcHeZdZdZdfd Zdfd Z dd ZxZS)WrapperUserFunctionVariablea3 Used to represent a wrapper object that contains the actual callable as an attribute. For example, torch.jit.script/trace have the original function at their _torchdynamo_inline attribute. Similarly, functions with __script_if_tracing_wrapper have the original attr at "__original_fn". rkNc Vtjdi|||_||_dSr&)rr wrapper_obj attr_to_trace)rbrqrrrirs r5rz$WrapperUserFunctionVariable.__init__s6""6"""&*r7r,r'c||jkrLt|j|j}|jot |j|}t j|||St||Sr.) rrrrqr4rr#r0rr)rbr,rJr3r4rs r5rz'WrapperUserFunctionVariable.var_getattrsm 4% % %$*D,>??C[BZ T%B%BF"(S&99 9ww""2t,,,r7rgrhrirjr#ctjtj||tj|jg||Sr.)rrr getattr_and_tracerpr%rrros r5rpz)WrapperUserFunctionVariable.call_functionsM -  '  - y1$2DEEMMv   r7rr,r'r)rrrrrrrrprrs@r5roros++++++ ------  #  &  -           r7rocRtjrddlm}|SiS)Nrtraceable_collective_remaps)r distributed is_available)torch.distributed._functional_collectivesryrxs r5_traceable_collective_remapsr}sD %%''+      +* Ir7ctjs Jd|tvsJ|j}|d}t||S)NzIllegal invocation.z)torch.distributed._functional_collectives)rrzr{r}valuesrr import_sourcer)r,r inner_name path_sources r5_traceable_collectives_sourcersu   ) ) + +BB-BBB + -//6688 8 8 8 8J""#NOOK k: . ..r7ceZdZdZdfd ZeddZedZedd Z ddZ xZ S)!CollectiveFunctionRewriteVariablea Some of the torch.distributed.* collective APIs are possible to rewrite to 'traceable' collectives. This class provides both a way to check if a function is remappable, and perform the remapping. In the case that a function is 'remappable' but only for some combinations of call-time arguments, we check the args at `call_function` time and fall back to graph-breaking if needed. This is no worse than status-quo as we currently graph-break on all distributed.* collectives. rkNc xtj|fi|t|tsJ||_dSr.)rrr/rreplacement_var)rbrrrirs r5rz*CollectiveFunctionRewriteVariable.__init__sG&&v&&&/+?@@@@@.r7r,r'c xt||\}}t|ft|fd|i||d|S)Nr4)rr4)rrewriter)r,old_fnr4optionsnew_fn new_sources r5r2z(CollectiveFunctionRewriteVariable.creates`>FFr6RR 0  0VV VgVV      r7cJtj|o|tvSr.)rz isfunctionr})variables r5 can_rewritez-CollectiveFunctionRewriteVariable.can_rewrites(  x ( ( WX9U9W9W-W r7cNt|}|t||fSr.)r}r)r,rrs r5rz)CollectiveFunctionRewriteVariable.rewrites(-//34R@@@@r7rgrhrirjr#cBddlm}ddlm}t j|j}t|j|i|j }d}d|vr1|d rtd|j|j|j |j |jfvrn|d}||jn|jdj}||vrt'd|t(j|||d<|j|||S)Nr)REDUCE_OP_TO_STRrasync_opzBCollectiveFunctionRewriteVariable can't support async_op=True for opzUnsupported all_reduce op: )torch.distributedrzr|rrzr{rr<rrrr all_reducereduce_scatter_tensor_reduce_scatter_basercrr|defaultrrr%r2rrp) rbr,rgridistrr{ reduce_op_var reduce_ops r5rpz/CollectiveFunctionRewriteVariable.call_function sj )(((((NNNNNN%dg.. nind5f55?@@   F:$6$I$I$K$K  ^UYU\^^    7 O  &  %   #JJt,,M!,##)$/7   000 !Jy!J!JKKK$5<< +F4L#11"dFCCCr7rrvr) rrrrrrrlr2rrrprrs@r5rrs//////    \   \ AAA\A'D #'D&'D- 'D  'D'D'D'D'D'D'D'Dr7rcleZdZdeddffd ZdZdZ ddZdddedefdZ dZ dZ xZ S)FunctoolsPartialVariablerZrkNc tjdi|||_t|tsJ||_t|t sJ||_dSr&)rrrZr/r9rgr<keywords)rbrZrgrrirs r5rz!FunctoolsPartialVariable.__init__8se""6""" $%%%%% (D)))))  r7c^fd|j|jr|j|js:t t|jdzddS|jt|j } t|t|jzdz|ddS)Nc0ddS)Nrrr?r9sr5r;z6FunctoolsPartialVariable.reconstruct..Asg&>&>{I&V&Vr7r F) r@rZrgforeachrrArrrr;rbcreate_call_function_kw)rbr:rbs ` r5rDz$FunctoolsPartialVariable.reconstruct@sVVVVWWW  9 ' OODI & & &}   ! !"6s49~~7I5"Q"Q R R R F ,,..///T]''))**  + +CIIDI,F,JDRW X X     r7c*|Sr.rUras r5rtz%FunctoolsPartialVariable.get_functionO&&(((r7r,r'rgrhrirjr#cd|j|z}i|j|}|j|||Sr.)rgrrZrp)rbr,rgri merged_args merged_kwargss r5rpz&FunctoolsPartialVariable.call_functionRs; i$& 34=3F3 y&&r; FFFr7rJctjtt jt |Sr.)rr%r2rsrrr)rbr,rJs r5rxz%FunctoolsPartialVariable.call_hasattr\s5)00 I%h// 6 6   r7ctj|jgd|jDRid|jDS)Nc6g|]}|SrrU)rrs r5rz?FunctoolsPartialVariable.as_python_constant..es$ < < <3c$$&& < < i|]\}}||SrrUrXs r5rz?FunctoolsPartialVariable.as_python_constant..fs*KKKTQq!&&((KKKr7)rrrZrrgrr:ras r5rz+FunctoolsPartialVariable.as_python_constantbsq I ( ( * * < <$) < < <   LKT]5H5H5J5JKKK   r7ctj|jgd|jDRid|jDS)zcSimilar to as_python_constant(), but add ID_MATCH guards to try to force things to become constantsc6g|]}|Srguard_as_python_constant)rr?s r5rzEFunctoolsPartialVariable.guard_as_python_constant..ms$ > > >qa((** > > >r7c>i|]\}}||SrrrXs r5rzEFunctoolsPartialVariable.guard_as_python_constant..ns*QQQ41aq!,,..QQQr7)rrrZrrgrr:ras r5rz1FunctoolsPartialVariable.guard_as_python_constantisq I . . 0 0 > >DI > > >   RQ4=;N;N;P;PQQQ   r7r) rrrrr#rrDrtrprrxrrrrs@r5rr7s!_!4!!!!!!    )))G #G&G- G  GGGG 6 c o              r7rc"eZdZdddhejZeejddee de fe j ffdZ edZdeddffd Zedefd Zd Zd Z ddZ ddZdZxZS)PolyfilledFunctionVariablerr) traceable_fnNrk.ciSr.r)rs r5_get_polyfill_handlersz1PolyfilledFunctionVariable._get_polyfill_handlerszs  r7ctt|tj|||Sr)rrrrLrs r5rz-PolyfilledFunctionVariable.create_with_sources6f'' (CDDEEEs5((((r7c rtjdi|||_|||}t |sJd|d|dD]*}t ||d}|rt |sJ|}n+td|d||_||_ dS)NzPolyfill handler z is not callable for )__torch_dynamo_polyfill____python_implementation__z# does not have a traceable functionr) rrrrrccallabler RuntimeErrorr)r)rbrrihandlercandidate_attr candidaterrs r5rz#PolyfilledFunctionVariable.__init__s""6"""--//33B;;  XX"Xg"X"XTV"X"XXX   N >>I  *****(   PGPPP & ,r7c|jSr.)rras r5 polyfill_fnz&PolyfilledFunctionVariable.polyfill_fns   r7c.t|jddS)N*__torch_dynamo_can_constant_fold_through__F)rr)ras r5can_constant_fold_throughz4PolyfilledFunctionVariable.can_constant_fold_throughs OI5   r7c*|Sr.rUras r5rtz'PolyfilledFunctionVariable.get_functionrr7r,r'rgrhrirjr#c N|rXt||rH|jd|Did|D}t j||S|jt jurt|dkr|st|dtj tj frtd|djDrtj||jdt$jt)d|djDfit%jd|djD St j||j}||||S) Nc6g|]}|SrrUrVs r5rz.s$;;;a**,,;;;r7c>i|]\}}||SrrUrXs r5rz.s*LLLTQq!..00LLLr7r rc3K|]m}t|tjrt|jtp4t|tjo|tuVndSr.)r/rr%rintSymNodeVariablerrVs r5r3z;PolyfilledFunctionVariable.call_function..sAy9::Wz!'SV?W?WYq)";<<WTWAWr7rpc3>K|]}|VdSr.as_proxy)ras r5r3z;PolyfilledFunctionVariable.call_function..s*??A1::<<??????r7c^g|]*}t|tjr|jn|j+Sr)r/rr%rsym_numrVs r5rz.sG  *!Y-GHH+AGG!" r7)r)rrrr:r#r0builtinssumrr/r ListVariable TupleVariableallrr2rB create_proxyrsym_sumr;rrp)rbr,rgrir=traceable_function_variables r5rpz(PolyfilledFunctionVariable.call_functions  ) ) + + 50M &1 1 5;;d;;;LLV\\^^LLL  #(V44 4 Gx| # #D Q47Y%;Y=T$UVVa,33 &&#M??a?????A    "&a   4 ('6&;B@Q&R&R#*88T6JJJr7ch|dkr||||St|j|d}|Jd|d|jt|sJd|d|ji}|jrt |j||d<t |fi|}||||S)N__call__zMember z not found in z is not callable in r4)rprrrr4rr)rbr,rJrgrimethodrpolyfilled_method_variables r5rz&PolyfilledFunctionVariable.call_methods :  %%b$77 7$--!!#JT#J#J#J#J!!!6""QQ$Qd$Q$Q$Q$QQQ" ; > *4; = =GH %?%R%R'%R%R")77D&IIIr7c|jSr.rras r5rz-PolyfilledFunctionVariable.as_python_constantrr7rrgrhrirjrkr#)rrrrr#rrrrmrrrrQrNrrr*rpropertyrrrtrprrrrs@r5rrrs   ' NYtHS#X,>@R,R'S[))[) -2-D------.!R!!!X!   )))2K #2K&2K- 2K  2K2K2K2KhJ& J - J  JJJJ&r7r)TMADescriptorMetadataTritonHOPifierceZdZdedefdZdedefdZdedefdZ de e j j jdffd Zd Zdefd Zd S) DynamoTritonHOPifierrjrkc t|r.)r)rbrjs r5raise_unsupportedz&DynamoTritonHOPifier.raise_unsupporteds#r7maybe_callablec:t|ttfSr.)r/rvr)rbrs r5 is_callablez DynamoTritonHOPifier.is_callables  79MN   r7r3c|jSr.rJ)rbr3s r5 get_valuezDynamoTritonHOPifier.get_values yr7.cddlm}t||r|St dt |dS)Nr )BaseListVariablezgrid for the triton kernel is )listsrr/rrtype)rbgridrs r5 check_gridzDynamoTritonHOPifier.check_gridsY++++++ d, - - I==?? " G4::GG H H H H Hr7crd|D}|||gi}|S)NcTi|]%\}}tj||&Srrr%r2rXs r5rz2DynamoTritonHOPifier.call_grid..s/QQQDAq *11!44aQQQr7)r:rp)rbrmetar,s r5 call_gridzDynamoTritonHOPifier.call_grid s;QQDJJLLQQQ!!"tfb11 r7c ddlmddlm}i}t |D]E}||}t |tr&|||<|j j ||<Fd| D} ddl m } m} fd| D} fd| D} | D]b}|}t |t"jt"jfs&|d t+|d c| | }|| t.}|jd | d |j||||d t#jdS)Nr r$)ConstDictVariablecTi|]%\}}tj||&SrrrXs r5rz1DynamoTritonHOPifier.call_HOP.."s?   1  & - -a 0 0!   r7r)kernel_side_tabletriton_kernel_wrapper_mutationc`i|]*\}}t|||+Sr)r/rrr>r?r%s r5rz1DynamoTritonHOPifier.call_HOP../sL   1!-.. q##%%   r7c<i|]\}}t|||Sr)r/rs r5rz1DynamoTritonHOPifier.call_HOP..4s@   1a!122 q   r7z.Unexpected argument type for a Triton kernel: r_rpr) kernel_idxconstant_args_idxrtma_descriptor_metadatari)rr%dictsrr9rbr/TMADescriptorVariable to_metadatadata_ptr from_tensorr:*torch._higher_order_ops.triton_kernel_wraprrrrealizerrrrrepradd_constant_argsr<rBrrr)rbrgridscombined_args_rawr,rrr>r? combined_argsrr constant_argsnon_constant_argsrrr%s @r5call_HOPzDynamoTritonHOPifier.call_HOPsM......,,,,,,:<',,..// > >A!!$A!233 >-.]]__'*'(z'=!!$  )//11               )//11       %++--    #))++  A Aa)":I+>?? ? r7rrr) rrrrrTrrrrprrr(rrs@r5rrXs   WWWWWW # & -        ;& ; - ;  ; ; ; ; ; ;ssr7rc<eZdZ d fd Zd Zd ZxZS) rrvariables.DataPtrVariabledimsList[ConstantVariable] block_dims element_sizer%c t|tjsJtjdi|||_||_||_||_dSr&) r/rDataPtrVariablerrrr+r-r.)rbrr+r-r.rirs r5rzTMADescriptorVariable.__init__s^(I$=>>>>>""6"""   $(r7cvd|jDd|jD|jfS)Nc6g|]}|Srrrdims r5rz5TMADescriptorVariable.to_metadata..s 1 1 1S\\^^ 1 1 1r7c6g|]}|Srrr3s r5rz5TMADescriptorVariable.to_metadata..s 7 7 7S\\^^ 7 7 7r7)r+r-r.rras r5rz!TMADescriptorVariable.to_metadatasD 1 1ty 1 1 1 7 7t 7 7 7   & & ( (  r7cfdjgjjj}|t|dzddS)Nc\ddtjdS)Nz$triton.tools.experimental_descriptorcreate_d_tma_descriptor)r8rr+)r:rbsr5r;z3TMADescriptorVariable.reconstruct..s0G,,6:#di..:::r7r F) r@rrDr+r-r.rrpr)rbr:rgs`` r5rDz!TMADescriptorVariable.reconstructs        !!'***@@T_@d.?@c$ii!mU33333r7)rr*r+r,r-r,r.r%)rrrrrrrDrrs@r5rrs )- )' )- ) ) ) ) ) ) ) )    4 4 4 4 4 4 4r7rc@eZdZdeddffd Z d d ZxZS)CreateTMADescriptorVariablerankrkNc T|dvsJtjdi|||_dS)N)r r r)rrr<)rbr<rirs r5rz$CreateTMADescriptorVariable.__init__s: v~~~~""6""" r7r,r'rgrhrirjr#c|d|vr|dn|d}t|tjstd|jd|jdkrPt |t |zdksJd|vr|dn|dg}d|vr|dn|d g}nut |t |zd ksJd |vr|d n|dd |vr|d n|d g}d |vr|d n|dd|vr|dn|dg}d|vr|dn|d}t ||||S)Nptrrz8Please ensure there were no graph breaks between create_z3d_tma_descriptor and the upstream .data_ptr() call.r r4 block_dimr dim1dim0 block_dim1 block_dim0r.)rr+r-r.)r/rr0rr<rr)rbr,rgrir?r+r-r.s r5rpz)CreateTMADescriptorVariable.call_functions %fUmmDG#y899 $)$$$  9>>t99s6{{*a////!&&u d1gD(3f'<'<{##$q'JJt99s6{{*a////"(F"2"2vQ"(F"2"2vQD )5(>(>|$$DG(4(>(>|$$DGJ2@61I1Ivn--tTVx $!%     r7r)rrrrrrrprrs@r5r;r;s  ( #( &( - (  ( ( ( ( ( ( ( ( r7r;r.rv)TrrNrrzrSrQtypingrrrrrrr r typing_extensionsr rrr rbytecode_transformationrrexcrrguardsrrr4rrrrrgrrrrrrrbaser!r"r#rr%#torch.distributed.fsdp._fully_shardr&ModuleNotFoundErrortorch._dynamo.symbolic_convertr'r r(r)r*r6r@rrLr[r]rrrrrrvrFror}rrrrrrrrrrr;rr7r5rRs UUUUUUUUUUUUUUUUUUUU###### ########HHHHHHHH,,,,,,,,00000000NNNNNNNNNNNN=<<<<<<<<<&&&&&&EEEEEEEDDDDDD  WT""" A A A A A.... $sO';"<    "(@t7t7t7t7t73t7t7t7n=5=5=5=5=5-=5=5=5@ 2("6(     KAKAKAKAKA!9KAKAKA\iiiii?iiiX     /   D////LDLDLDLDLD(<LDLDLD^8 8 8 8 8 8 8 8 v~~~~~~~~B Y Y Y Y Y >Y Y Y x$8#7#9#9 )))))?)))X!4!4!4!4!4O!4!4!4H2 2 2 2 2 /2 2 2 2 2 sB BB