jaXPddlZddlZddlZddlmZeejdsHedejjd<edejjd<edejjd<ddlm Z m Z m Z d Z d Z Gd d ejj ZGd dZ ddZdS)N) _dummy_type_CudaStreamBase _CUDAGraph_graph_pool_handle_cuda_isCurrentStreamCapturing)rrrctS)zReturn True if CUDA graph capture is underway on the current CUDA stream, False otherwise. If a CUDA context does not exist on the current device, returns False without initializing the context. )rBC:\Projects\Historical\venv\Lib\site-packages\torch/cuda/graphs.pyis_current_stream_capturingr s * + ++r ctS)zReturn an opaque token representing the id of a graph memory pool. See :ref:`Graph memory management`. .. warning:: This API is in beta and may change in future releases. )rr r r graph_pool_handler"s   r cjeZdZdZfdZd fd ZfdZfdZfdZfd Z fd Z fd Z xZ S) CUDAGraphzrWrapper around a CUDA graph. .. warning:: This API is in beta and may change in future releases. cFt|SN)super__new__)cls __class__s r rzCUDAGraph.__new__5swws###r NglobalcNt||dS)aBegin capturing CUDA work on the current stream. Typically, you shouldn't call ``capture_begin`` yourself. Use :class:`~torch.cuda.graph` or :func:`~torch.cuda.make_graphed_callables`, which call ``capture_begin`` internally. Arguments: pool (optional): Token (returned by :func:`~torch.cuda.graph_pool_handle` or :meth:`other_Graph_instance.pool()`) that hints this graph may share memory with the indicated pool. See :ref:`Graph memory management`. capture_error_mode (str, optional): specifies the cudaStreamCaptureMode for the graph capture stream. Can be "global", "thread_local" or "relaxed". During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. "global" will error on actions in other threads, "thread_local" will only error for actions in the current thread, and "relaxed" will not error on these actions. Do NOT change this setting unless you're familiar with `cudaStreamCaptureMode `_ )poolcapture_error_modeN)r capture_begin)selfrrrs r rzCUDAGraph.capture_begin8s)" 4` for a general introduction, detailed use, and constraints. Arguments: cuda_graph (torch.cuda.CUDAGraph): Graph object used for capture. pool (optional): Opaque token (returned by a call to :func:`~torch.cuda.graph_pool_handle()` or :meth:`other_Graph_instance.pool()`) hinting this graph's capture may share memory from the specified pool. See :ref:`Graph memory management`. stream (torch.cuda.Stream, optional): If supplied, will be set as the current stream in the context. If not supplied, ``graph`` sets its own internal side stream as the current stream in the context. capture_error_mode (str, optional): specifies the cudaStreamCaptureMode for the graph capture stream. Can be "global", "thread_local" or "relaxed". During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. "global" will error on actions in other threads, "thread_local" will only error for actions in the current thread, and "relaxed" will not error on actions. Do NOT change this setting unless you're familiar with `cudaStreamCaptureMode `_ .. note:: For effective memory sharing, if you pass a ``pool`` used by a previous capture and the previous capture used an explicit ``stream`` argument, you should pass the same ``stream`` argument to this capture. .. warning:: This API is in beta and may change in future releases. .. _cudaStreamCaptureMode: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html#group__CUDART__STREAM_1g9d0535d93a214cbf126835257b16ba85 Nztorch.cuda.Streamdefault_capture_streamrrc0|jj(tj|j_|dn|f|_||n |jj|_|jJtj|j|_||_ ||_ dS)Nr ) rr2torchcudaStreamrcapture_streamstream stream_ctx cuda_graphr)rr:rr8rs r __init__zgraph.__init__s > 0 849J4E4E4G4GDN 1,BBTG (FFdn.S "...*++D,?@@$"4r ctjtjtj|j|jj |j d|j idS)Nr) r4r5 synchronizegccollect empty_cacher9 __enter__r:rrr)rs r rAzgraph.__enter__s~          !!###%% Y +/+B     r cp|j|j|||dSr)r:rr9__exit__)rexc_type exc_value tracebacks r rCzgraph.__exit__s6 ##%%%   9i@@@@@r )NNr) r+r,r-r.r2typingOptional__annotations__strr;rArCr r r r1r1us:DHFO,?@GGG "* 55  5555,    AAAAAr r1Fc  '(tjr"tjrtdd}t |t sd}|f}|f}g't ||D] \}}t |tjjrt|j dkr0t|j dkrt|j dks Jdtd|Ds Jdtjjj|}'t |td|Ds Jd  d 'D} d |D('(fd t't|D} d t't|D} dt't|D} |t)n|} tjtjtj5t ||| D]\}}}d\}}}t'|D]}tjj||}t d|D}t|dkrRtj|t d|Dt d|Dd|}|||fD]}~ dddn #1swxYwYtjg}g}t ||| D]\}}}tj|| 5||}dddn #1swxYwYtjj|\}}|t |||g}g}t t=| t=|t=| D]Z\}}}t d|D}t d|D}d}t|dkrtj|| 5tj|t d|Dt d|Dd|}dddn #1swxYwYg}d} |D]A}!|!jr#|!||| | dz } ,|dBt |}||||\| | d}"g}#tC|D]\}$}|"| |$| |$(|$| |$||$| |$||$||$||$ }%t |tjjr6d}&|&||j"|%|j#|_#|#||#|%|r|#dSt |#S)aAccept callables (functions or :class:`nn.Module`\ s) and returns graphed versions. Each graphed callable's forward pass runs its source callable's forward CUDA work as a CUDA graph inside a single autograd node. The graphed callable's forward pass also appends a backward node to the autograd graph. During backward, this node runs the callable's backward work as a CUDA graph. Therefore, each graphed callable should be a drop-in replacement for its source callable in an autograd-enabled training loop. See :ref:`Partial-network capture` for detailed use and constraints. If you pass a tuple of several callables, their captures will use the same memory pool. See :ref:`Graph memory management` for when this is appropriate. Arguments: callables (torch.nn.Module or Python function, or tuple of these): Callable or callables to graph. See :ref:`Graph memory management` for when passing a tuple of callables is appropriate. If you pass a tuple of callables, their order in the tuple must be the same order they'll run in the live workload. sample_args (tuple of Tensors, or tuple of tuples of Tensors): Samples args for each callable. If a single callable was passed, ``sample_args`` must be a single tuple of argument Tensors. If a tuple of callables was passed, ``sample_args`` must be tuple of tuples of argument Tensors. num_warmup_iters (int): The number of warmup iterations. Currently, ``DataDistributedParallel`` needs 11 iterations for warm up. Default: ``3``. allow_unused_input (bool): If False, specifying inputs that were not used when computing outputs (and therefore their grad is always zero) is an error. Defaults to False. pool (optional): Token (returned by :func:`~torch.cuda.graph_pool_handle` or :meth:`other_Graph_instance.pool()`) that hints this graph may share memory with the indicated pool. See :ref:`Graph memory management`. .. note:: The ``requires_grad`` state of each Tensor in ``sample_args`` must match the state that's expected for the corresponding real input in the training loop. .. warning:: This API is in beta and may change in future releases. .. warning:: ``sample_args`` for each callable must contain only Tensors. Other types are not allowed. .. warning:: Returned callables do not support higher order differentiation (e.g., double backward). .. warning:: In any :class:`~torch.nn.Module` passed to :func:`~make_graphed_callables`, only parameters may be trainable. Buffers must have ``requires_grad=False``. .. warning:: After you pass a :class:`torch.nn.Module` through :func:`~make_graphed_callables`, you may not add or remove any of that Module's parameters or buffers. .. warning:: :class:`torch.nn.Module`\s passed to :func:`~torch.cuda.make_graphed_callables` must not have module hooks registered on them at the time they are passed. However, registering hooks on modules *after* passing them through :func:`~torch.cuda.make_graphed_callables` is allowed. .. warning:: When running a graphed callable, you must pass its arguments in the same order and format they appeared in that callable's ``sample_args``. .. warning:: The automatic mixed precision is supported in :func:`~torch.cuda.make_graphed_callables` only with disabled caching. The context manager `torch.cuda.amp.autocast()` must have `cache_enabled=False`. z_make_graphed_callables does not support the autocast caching. Please set `cache_enabled=False`.FTrzModules must not have hooks registered at the time they are passed. However, registering hooks on modules after passing them through make_graphed_callables is allowed.c3(K|] }|jduVdS)FN requires_grad.0bs r z)make_graphed_callables..s)EEAq%/EEEEEEr zIn any :class:`~torch.nn.Module` passed to :func:`~make_graphed_callables`, only parameters may be trainable. All buffers must have ``requires_grad=False``.c3JK|]}t|tjVdSr) isinstancer4Tensor)rQargs r rSz)make_graphed_callables..#s.HHS:c5<00HHHHHHr zfIn the beta API, sample_args for each callable must contain only Tensors. Other types are not allowed.c,g|]}t|Sr )len)rQargss r z*make_graphed_callables..*s!L!L!L#d))!L!L!Lr cg|]D}t|tjjr!t |ndES)r )rUr4nnModuletuple parameters)rQcs r r[z*make_graphed_callables..+sP""" ",Aux!?!?GallnnR"""r c2g|]}||zSr r )rQiflatten_sample_argsper_callable_module_paramss r r[z*make_graphed_callables../s9***  A!;A!>>***r cJg|] }tj!Sr r4r5rrQ_s r r[z*make_graphed_callables..4&HHHQ%*&&((HHHr cJg|] }tj!Sr rgrhs r r[z*make_graphed_callables..5rjr N)NNNc3(K|] }|j |VdSrrNrQos r rSz)make_graphed_callables..Ds)$K$K11?$KQ$K$K$K$K$K$Kr c3(K|] }|j |VdSrrNrQrcs r rSz)make_graphed_callables..Hs=%%"#q%%%%%%%r c3LK|]}|j tj|V dSrrOr4 empty_likerms r rSz)make_graphed_callables..KsH++45AO+!,Q//++++++r )outputsinputs grad_outputs only_inputs allow_unused)rc3PK|]!}|jrtj|ndV"dSrrrrms r rSz)make_graphed_callables..osJ$ $ AB1? .ss)JJ1!/JQJJJJJJr c3(K|] }|j |VdSrrNrps r rSz)make_graphed_callables..ys) T TqAO T T T T T T Tr c3K|]}||V dSrr rms r rSz)make_graphed_callables..zs"&W&WQq&W&Wr c r Gfddtjj fd} | S)Nc|eZdZefdZeejjjfdZ dS)Omake_graphed_callables..make_graphed_autograd_function..Graphedc^tD]Y}|||kr!|||Zt t sJt dDS)Nc3>K|]}|VdSrdetachrms r rSzjmake_graphed_callables..make_graphed_autograd_function..Graphed.forward..s*@@AQXXZZ@@@@@@r )rangedata_ptrcopy_r"rUr_)ctxrurc fwd_graph len_user_argsstatic_input_surfacestatic_outputss r forwardzWmake_graphed_callables..make_graphed_autograd_function..Graphed.forwards}--AAA+A.7799VAY=O=O=Q=QQQ,Q/55fQi@@@  """!.%88888@@@@@@@@r c~t|tksJt|D]F\}}|?||kr||Gt t sJt dDS)Nc3FK|]}||n|VdSrrrPs r rSzkmake_graphed_callables..make_graphed_autograd_function..Graphed.backward..sC;.make_graphed_autograd_function..Graphed.backwards5zzS)<%=%====="#6>>**GAt}::<<4==??::GGDMMM  """""4e<<<<<@Rr N) r+r,r- staticmethodrr4autogradfunctiononce_differentiabler)rrrrrrrsr Graphedrs  A A A A A A A\ A ^ $ 8      9 8\   r rctjjj|}jt |z}tjj|Sr)r4utils_pytreearg_tree_leavesapplyr_tree_unflatten) user_argsflatten_user_argsoutr module_paramsoutput_unflatten_specs r functionalizedzVmake_graphed_callables..make_graphed_autograd_function..functionalizedsP!& 3 CY O '-%(9":":]"JLC;&55c;PQQ Qr )r4rFunction) rrrrrrrrrrrs ````````` @r make_graphed_autograd_functionz>make_graphed_callables..make_graphed_autograd_functions             en-   : R R R R R R Rr cfd}|S)Nc.jkr|S|Sr)training)rfuncgraph_training_stategraphedorig_fwds r new_fwdzEmake_graphed_callables..make_graphed_forward..new_fwds-}(<<<&w 22'x33r r )rrrrrs```` r make_graphed_forwardz4make_graphed_callables..make_graphed_forwards544444444r )$r4is_autocast_enabledis_autocast_cache_enabled RuntimeErrorrUr_rr]r^rY_backward_hooks_forward_hooks_forward_pre_hooksallbuffersrrrappendrrr5r=r8r6 tree_leavesrrr1 tree_flattenreversedrOreverse enumeraterr)) callables sample_argsnum_warmup_itersallow_unused_inputrjust_one_callablerarZ flatten_argper_callable_len_user_args"per_callable_static_input_surfaces fwd_graphs bwd_graphsmempoolrr grad_inputsrt outputs_gradrivper_callable_static_outputs"per_callable_output_unflatten_specrflatten_outputsspec per_callable_static_grad_outputsper_callable_static_grad_inputsrrrrgrad_idxrWrretrcrrrdres) @@r make_graphed_callablesrsJ  "" u'F'H'H  m    i ' '% L "n y+..  4 a ) ) A%&&!++())Q..,--222]323EEEEEEE  -  E k)94@ ""5#5#5666HHKHHHHH   Z  H "M!L8K!L!L!L"""""*****s9~~&&***& IH%I2G2GHHHJHH%I2G2GHHHJ%)\!!!tG  J   5:,,.. / /03 {$F1 1    ,D$,2B .K,+,,  +-99$$+FF$$K$K$K$K$KKK |$$q(("'."5"5 ,$%%';%%%  &+++9@+++&&%)%7#6 # #K|[9  A ' . J#%)+&!$Y Z!H!H88dI Z  ig  6 6 " "dDkG " " " " " " " " " " " " " " "!& 3 @ @ I I#**5+A+ABBB*11$7777(*$&(#;>344,--<<%C%C7ni$$ $ FT$ $ $   JJJJJJJ  |  q !!)'!::  #n11( T T,@ T T TTT!&&W&W2E&W&W&W!W!W $!3 2                 ' 0 0C  0[%<"))+h*?@@@A "))$////"#566(//0CDDD'../ABBBB%,,...#++---000f CY''  400 qM qM &q ) &q ) .q 1 .q 1 ' * ,Q / +A .    dEHO , ,    0/dmWdl[L JJt     JJw    1v ::s8,CMMM=OO O $AUU U )rKFN)r>rGr4_utilsrhasattr_C__dict__torch._Crrrr rrr1rr r r rs wux*++&1k,&?&?EHl#.9k:N.O.OEH*+:E+(;;EH67,,,   D.D.D.D.D.#D.D.D.NFAFAFAFAFAFAFAFAVPTkkkkkkr