j<UddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z mZmZmZmZmZmZmZmZddlmZddlZddlmZddlmZddlmZddlmZm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)m*Z*m+Z+dd l m,Z,dd l-m.Z.dZ/ee j0e1d < ddl2Z/n #e3$rdZ/YnwxYwe j4Z4dZ5ej6e7Z8deej9deej9fdZ:de;de;fdZj?de de de dee f dZ@de deAfdZBedej9dej9fdZCedeee ee dfee e ffdeDfdZCde deej9eDffdZCde;fd ZEdgde;d"e jFd#e;ddfd$ZGdd%d&e,d'eHd(e d)e d*eHdee+f d+ZIGd,d-ZJGd.d/ZKGd0d1ZLGd2d3ZMd"e;de;fd4ZNd"e;de;fd5ZOd6e;de;fd7ZPd"e;de;fd8ZQ dhd9e d:e de fd;eHdeHddfd?ZRd@ejSdeej9de de ffdAZT didDeDdEeDdFeAdGeDde;f dHZUejVdIdfdJeeHdKeeHdLejWdMee;ejXfdNeHdej9f dOZYedPZZdeAfdQZ[d:e deZfdRe de deZffdSZ\ddTdUdVe]dWe;dXe;dRe dYee;dZe e de fge de ffde]fd[Z^d:e de fde de ffd\Z_d:e de fde de ffd]Z`d:e de fde de ffd^Zad:e de fde de ffd_Zbd:e de fde de ffd`Zcd:e de fde de ffdaZdd:e de fde de ffdbZed:e de fde de ffdcZfdjdeeAddfdfZgdS)kN) AnyCallableDictListOptionaloverloadSequenceTupleTypeVarUnion)patch)fx) aot_eager) OutputGraph)config eval_frameoptimize_assertreset)create_instruction debug_checks is_generatortransform_code_object)CheckFunctionManager CompileId GuardedCode)DynamoFrameType)samenpxreturnc|dS||jSN)detachclonerequires_grad_ requires_gradr!s FC:\Projects\Historical\venv\Lib\site-packages\torch/_dynamo/testing.pyclone_mer+7s6yt 88::     , ,Q_ = ==namec.tjdd|S)Nz ^_orig_mod[.]resub)r-s r*remove_optimized_module_prefixr3= 6"B - --r,cddlmddfd}tj|d||i|jfS)Nr)InstructionTranslatorcH|jj|Sr$) current_txoutputregion_tracker)_gmargskwargsr6gmr:s r*extract_graph_backendz8extract_graph_and_tracker..extract_graph_backendGs&.99;;BQ r,T)backend fullgraph)torch._dynamo.symbolic_convertr6torchcompilegraph)fnr<r=r?r6r>r:s @@@r*extract_graph_and_trackerrGAsDDDDDD BNE@EM/4@@@DDdUfUUU 8^ ##r,model predictionlossexample_inputscJg}||||i}i}|D]_\}}t|tjrt |}|} |j} |jtj|} | ||dz<| ||<`||||i} | D]3\}} t|tjrt |}| | |<4|| |D]r} t| ttfr | d| D>t| tj r|| js|S)Nz.gradc3XK|]%}t|tj|jV&dSr$) isinstancerCTensorgrad).0inps r* z"collect_results..ss5XX*S%,:W:WX38XXXXXXr,)appendnamed_parametersrNrOptimizedModuler3rPrC zeros_like named_bufferstuplelistextendrO)rHrIrJrKresultsgradsparamsr-param param_copyrPbuffersbufferexamples r*collect_resultsrdRsG NN: NN4 E F--// " " e eZ7 8 8 81$77D z : #E**D $dWn!t  NN5 NN6G++-- f eZ7 8 8 81$77D  NN7!-- gt} - - - NNXXwXXX X X X X'5<00 -w|,,, Nr,outct|tjr|jSt|tt frt d|DS|dSt|trdStdt|)Nc34K|]}t|VdSr$)requires_bwd_passrQr!s r*rSz$requires_bwd_pass..~s+55A$Q''555555r,FDon't know how to reduce) rNrCrOr(rZrYanyintNotImplementedErrortyperes r*rhrhzs#u|$$  C$ ' '55555555 u C  u 8$s)) D DDr,cdSr$ros r*reduce_to_scalar_lossrrsCr,.cdSr$rqros r*rrrrs Cr,ct|tjr)||z St|t t fr)td|Dt|z St|j dvrt|j St|j dkr|j St|trMtd|Dt|z St!dt|)z/Reduce the output of a model to get scalar lossc34K|]}t|VdSr$rrris r*rSz(reduce_to_scalar_loss..s+99(++999999r,)MaskedLMOutputSeq2SeqLMOutput!CausalLMOutputWithCrossAttentionsSquashedNormalc34K|]}t|VdSr$rv)rQvalues r*rSz(reduce_to_scalar_loss..s+JJE(//JJJJJJr,rj)rNrCrOsumnumelrZrYlenrn__name__rrlogitsmeandictvalueskeysrmros r*rrrrs1#u|$$ wwyy399;;&& C$ ' '  99S99999CHHDD c     %SZ000 c / / /x||~~ C   JJSZZ\\JJJJJS HHJJN N    8$s)) D DDr,ctjtjtd}tj|stj||S)Nz../debug)ospathjoindirname__file__existsmkdir)rs r* debug_dirrsM 7<<11: > >D 7>>$    Kr,r/codeextrac xttjt |d5}|t j|dt j|d|dddddS#1swxYwYdS)Nwz  ) openrrrrwritedisBytecodeinfo)r-rrfds r* debug_dumprs bgll9;;--s 3 3 r |D!!&&(( U Ucl4.@.@.D.D.F.F U UE U U U                     sA'B//B36B3)skipframe cache_sizehooks_rc dttdtddfd}t|jrdSt |jt |j|}t idddddditt|jg }t|t|j tddS) zused to debug jump updates instructions code_optionsr"Nc|dtd|dtddS)NrNOP)insertr)rrs r* insert_nopsz&debug_insert_nops..insert_nopssFA1%88999A1%8899999r,F_idr) r compiler_fnroot_txexportexport_constraints frame_state local_scope global_scopef_codetorch_function_mode_stack) rrrrrrrlocalsglobalsrr guard_managerr)rrrrrrrrEs r*debug_insert_nopsrs :$s):3:4::::EL!!t { ; ;D AJHHYY|"$   E t1%88F RSUV X XXr,cleZdZd dZdejjdeejde de ffdZ d dZ dS) CompileCounterr"Nc"d|_d|_dSNr frame_countop_countselfs r*__init__zCompileCounter.__init__ r,r>rK.c|xjdz c_|jjD]}d|jvr|xjdz c_|jS)Nrcall)rrEnodesoprforward)rr>rKnodes r*__call__zCompileCounter.__call__sQ AHN # #D   " zr,c"d|_d|_dSrrrs r*clearzCompileCounter.clearrr,r"N) r __module__ __qualname__rrCr GraphModulerrOrrrrrqr,r*rrs}(&8d|_d|_||_g|_dSr)rrr@graphs)rr@s r*rz"CompileCounterWithBackend.__init__s#  24 r,r>rK.cddlm}|xjdz c_|jjD]}d|jvr|xjdz c_|j|||j ||S)Nr)lookup_backendr) backends.registryrrrErrrrrTr@)rr>rKrrs r*rz"CompileCounterWithBackend.__call__s 655555 AHN # #D   "  2+~~dl++B???r,) rrrstrrrCrrrrOrrrrqr,r*rrs555555 @(& @8rK.cD|j||jSr$)rrTr)rr>rKs r*rzEagerAndRecordGraphs.__call__s" 2zr,r rrrrrCrrrrOrrrrqr,r*rrsi5555(&8rK.cfj|dtjjdt tjdtdtfffd }dtjjdt tjdtdtfffd }t||||S)Nr>rKr".cFj||jSr$)rrTrr>rKrs r* fw_compilerz5AotEagerAndRecordGraphs.__call__..fw_compiler# N ! !" % % %: r,cFj||jSr$)rrTrrs r* bw_compilerz5AotEagerAndRecordGraphs.__call__..bw_compilerrr,)rr) rrTrCrrrrOrrr)rr>rKrrs` r*rz AotEagerAndRecordGraphs.__call__s 2 $ 6:5<6H  c3h         $ 6:5<6H  c3h          ##     r,rrrqr,r*rr si8888  (& 8z)remove_trailing_space..-s AAAdkkmmAAAr,)rsplitrs r*remove_trailing_spacer,s/ 99AA 40@0@AAA B BBr,gm_strc:tt|Sr$)rr)rs r* normalize_gmr0s !v!6!6 7 77r,c2tjdd|}|S)z- Normalize code: remove empty lines. z[\r\n]+rr0)r normal_codes r*empty_line_normalizerr6s&T400K r,rrFnargs expected_opsexpected_ops_dynamicexpected_frame_countctjs||}t}dt|D}dt|D}||} ||} t t ||} | |} | |} | |}| |}t |t| | |t|| |t| | |t|| ||j ||||j |dSdS)Nc8g|]}tjddS rCrandnrQrs r*rz!standard_test..K$ 7 7 7QU[R 7 7 7r,c8g|]}tjddSrrrs r*rz!standard_test..Lrr,) rassume_static_by_defaultrrangerr assertTruer assertEqualrr)rrFrrrractualargs1args2correct1correct2opt_fnval1aval2aval1bval2bs r* standard_testr>sx  *,/C/O+   F 7 7%,, 7 7 7E 7 7%,, 7 7 7Er5zHr5zH GGG $_V $ $R ( (F FENE FENE FENE FENE GGGOOD))***OOD))***OOD))***OOD))***V')=>>> ,77777 r,r>c|jSr$)r)r>rKs r*dummy_fx_compiler_s  :r,T皙?speeduppvalue is_correctpvalue_thresholdc4|sdS||kr|ddS|dd|dS)NERRORz.3fzx SAMEzx p=z.2frq)rrrrs r*format_speedupresI w    %%%%% + + +v + + ++r,cpusizestridedtypedevice extra_sizecztdt||Ddz|z}|jrZ|jdkr7 t j|tj||}n0t j|||}nt j|g||}t j |||S)Nc3,K|]\}}|dz |zVdS)rNrq)rQshapers r*rSzrand_strided..zs/ H H]UFUQY& H H H H H Hr,r)rr )r)rrr ) r}zipis_floating_pointitemsizerCrfloat16tozeros as_strided)rrrr r! needed_sizerbs r* rand_stridedr-rs H Hc$6G6G H H HHH      M >Q   [EM&QQQTTUFF[E&IIIFF;-uVLLL  FD& 1 11r,_Tctj Sr$)rrrqr,r*check_dynamic_shape_capturer0s. ..r,patchescvtjdtdtdtffd }|S)Nr<r=r"c tj5}D]/\}}}|tj|||0|i|cdddS#1swxYwYdSr$) contextlib ExitStack enter_contextr object)r<r=stackmoduleattrvalrFr1s r*_fnz"_make_fn_with_patches.._fns  ! # # 'u%, E E!c##ELs$C$CDDDD2t&v&&  ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' 's:AA #A ) functoolswrapsrr.)rFr1r<s`` r*_make_fn_with_patchesr?sX_R'3'#'"''''''' Jr,c|Sr$rqr)s r*rAsar,) xfail_prop decoratorcls cls_prefix fn_suffixrBrCc <t||j|ji}|j|_t |D]}|drt ||}t|s t||t ||V||} t|g|R} | | _|$t||rtj | } t|| || t||st||t |||S)Ntest_) rnr __bases__rdir startswithgetattrcallablesetattrr?hasattrunittestexpectedFailure) rDrErFrBrCr1DummyTestClassr-rFnew_namenew_fns r*make_test_cls_with_patchesrUs9Z777KKN"0"9NC>> ??7 # # >d##BB<< gc4.@.@AAA+ ++H*28888F&FO%'"j*A*A%!1&99 NHii.?.? @ @ @ @.. > ND'#t*<*< = = = r,cNtjdkr|Stj|S)N)r sys version_inforPrrFs r*skipIfNotPy311r\s% 7"" =  r,c`tjdkr|Stjd|S)Nr zRequires Python 3.12+rXr[s r*skipIfNotPy312r`s0 7"" 18=0 1 1" 5 55r,cNtjdkrtj|S|S)Nr^)rYrZrPrQr[s r* xfailIfPy312rbs& 7""'+++ Ir,c`tjdkrtjd|S|S)Nr^zNot supported in Python 3.12+rXr[s r* skipIfPy312rds0 7""=x}<==bAAA Ir,c`tjdkr|Stjd|S)N)r rzRequires Python 3.10+rXr[s r* requiresPy310rfs0 7"" 5x}455b999r,cd|_|SNT)_expected_failure_dynamicr[s r*expectedFailureDynamicrjs#'B Ir,cd|_|Srh)!_expected_failure_codegen_dynamicr[s r*expectedFailureCodegenDynamicrm+/B( Ir,cd|_|Srh)!_expected_failure_dynamic_wrapperr[s r*expectedFailureDynamicWrapperrqrnr,Fuse_xlac&tjdtjdtrtjd|r@ddlmcm}|dt| dSdS)Ni9r) rC manual_seedrandomseedrtorch_xla.core.xla_modelcore xla_model set_rng_stater xla_device)rrxms r*reset_rng_stater}s d K  t5--------- s2==??334444455r,)r/)NNr)Tr)F)hr4rr=loggingos.pathrrur1rYtypesrPtypingrrrrrrr r r r unittest.mockr rCr torch._dynamo.backends.debuggingrtorch._dynamo.output_graphrr/rrrrbytecode_transformationrrrrguardsrrrrutilsrr ModuleType__annotations__numpyModuleNotFoundError unsupportedthree getLoggerrlogrOr+rr3rGnnModulerdboolrhrrfloatrCodeTyperrlrrrrrrrrrrrrrfloat32rr r-r.r0r?rnrUr\r`rbrdrfrjrmrqr}rqr,r*rs                             666666222222888888888888 A@@@@@@@@@"""""""&HU %%% BBB$  g!!>&>8EL+A>>>> ......$$$"% 8?%(+%36%HK% #Y%%%%P E3 E4 E E E E u|    tCy%S/4S>9 :   EsEuU\5-@'AEEEE,3  S  s D    QRYYY Y(+Y47Y&@@@@@@@@,        >......CCCCCC888888 #'*. ! 88 8c8 83- 8 #3- 8  8 8888B (,U\(: c3h! , ,  ,  , , ,  , , , , ', 22 3-2 SM2 ;2 #u|# $ 2  2  \ 22228 WT]]/T//// hsBw/ 3 8CQSGCT    "!%DOK      #s(+,hsCx.@@A  >xS)hsCx.@ 6xS)6hsCx.@6666 Xc3h'HS#X,> HS#X&8CH+= :hsCx(:Xc3h-?::::xS1hsCx6H hsCx&8Xc3h=O hsCx&8Xc3h=O 55T5d555555sB$$B.-B.