Ljh ddlZddlmZddlmZddlZgdZdZGddZGdd Z Gd d Z Gd d Z ej j jej j jej j jdZdeZ d9ddezej j zdej j fdZej jjej jjej jjej jjej jjej jjdZdeZ d9ddezej jzdej jfdZd9ddezdefdZd9ddezdefdZ d:ddezddezej jzdefdZ!ej j"jej j"j#ej j"jdZ$de$Z%ddl&m'Z(m)Z* d9ddezej j"zdej j"fdZ+de(_,de(_-de.fdZ/dZ0d e.fd!Z1d"Z2d e.fd#Z3d$Z4d e.fd%Z5d e.fd&Z6d'Z7de.fd(Z8d;d*e(d+e.de.fd,Z9d;d*e(d+e.de.fd-Z:d;d*e(d+e.de.fd.Z;d/Zed1e?2 d???r(N) r<r=r>__doc__r0rr$_cufft_get_plan_cache_sizesize_cufft_get_plan_cache_max_size_cufft_set_plan_cache_max_sizemax_sizerDr'r(r)rrEs))) ) ( ( N  D -, ,e.RH@@@@@r(rc8eZdZdZdZdZdZdZfdZxZ S)ra Represent all cuFFT plan caches, return the cuFFTPlanCache for a given device when indexed. Finally, this object, when used directly as a `cuFFTPlanCache` object (e.g., setting the `.max_size`) attribute, the current device's cuFFT plan cache is used. Fc"g|_d|_dS)NT)caches#_cuFFTPlanCacheManager__initializedrCs r)r0zcuFFTPlanCacheManager.__init__is !r(ctjj|}|dks"|tjkr1t dtjd|t |jdkrM|jdttjD|j|S)Nrz/cufft_plan_cache: expected 0 <= device index < z, but got device with index c34K|]}t|VdSr,)r).0indexs r) z4cuFFTPlanCacheManager.__getitem__..us;*/u%%r() r$cuda_utils_get_device_index device_countr9lenrMextendrange)r/devicerRs r) __getitem__z!cuFFTPlanCacheManager.__getitem__ms !33F;; 199!8!8!:!:::-%*BYBYB[B[--%*--  t{  q K  389P9P9R9R3S3S   {5!!r(cft|tj|Sr,)getattrr$rTcurrent_device)r/names r) __getattr__z!cuFFTPlanCacheManager.__getattr__zs%tEJ55778$???r(c|jr3t|tj||St ||Sr,)rNsetattrr$rTr_super __setattr__)r/r`value __class__s r)rez!cuFFTPlanCacheManager.__setattr__}sM   44 9 9 ; ;rErNr0r\rare __classcell__)rgs@r)rr^syM""" " " "@@@444444444r(rc NeZdZedededeeeffdZdZ dZ dS)r rf attr_namereturncdtdtdtffd }t|tr|dfSt|tt frx|st dt|dkrt d||d d }t|d kr|dfS||d d }||fSt d t|)Nr3whichrkc zt|tr|Std|dt|)Nz expects a bool for z , but got )r7bool TypeErrortype)r3rmrjs r) _ensure_boolz;cuBLASModule._parse_reduction_setting.._ensure_boolsK#t$$  PP%PP499PP r(Tz& expects at least one boolean argumentz& expects at most two boolean argumentsrallow_reduced_precision allow_splitkz2 expects a bool or a tuple/list of bools, but got ) rr8ror7listtuplerprXrq)rfrjrrrtrvs ` r)_parse_reduction_settingz%cuBLASModule._parse_reduction_settings+ c # $       eT " " $;  edE] + + 9 V9 T T TUUU5zzA~~9 T T TUUU&2l58=V&W&W #5zzQ.44'<a.AAL*L8 8 [ [DQVKK [ [   r(cJ|dkrtjS|dkr#tj\}}|S|dkr#tj\}}|S|dkr#tj\}}|S|dkr#tj\}}|S|dkrtjS|dkr tjdd Std |z) N allow_tf32&allow_fp16_reduced_precision_reduction.allow_fp16_reduced_precision_reduction_split_k&allow_bf16_reduced_precision_reduction.allow_bf16_reduced_precision_reduction_split_kallow_fp16_accumulationfp32_precisionrTrUnknown attribute )r$r%_get_cublas_allow_tf322_get_cublas_allow_fp16_reduced_precision_reduction2_get_cublas_allow_bf16_reduced_precision_reduction#_get_cublas_allow_fp16_accumulation_get_fp32_precision_getterAttributeError)r/r`rt_rvs r)razcuBLASModule.__getattr__s3 <  82244 4 = = =KKMM ' #Q+ * E E EKKMM A|  = = =KKMM ' #Q+ * E E EKKMM A|  . . .8??AA A % % %866vxHH H1D8999r(c|dkrtj|S|dkr9||d\}}tj||S|dkr9||d\}}tj||S|dkrtj|S|dkr!tjdd|Std|z) Nr{r|r~rrrTrr) r$r%_set_cublas_allow_tf32ry2_set_cublas_allow_fp16_reduced_precision_reduction2_set_cublas_allow_bf16_reduced_precision_reduction#_set_cublas_allow_fp16_accumulation_set_fp32_precision_setterr)r/r`rfrtrvs r)rezcuBLASModule.__setattr__s! <  822599 9 = = =484Q4Q?55 1 #\8NN' = = =484Q4Q?55 1 #\8NN' . . .8??FF F % % %866vxOO O1D8999r(N) r<r=r> staticmethodrr8rxroryrarer'r(r)r r sn   dDj@Q   \ 0:::::::::r(r )defaultcusolvermagmaz, backendrkc|nt|trL|tvrtdtdt jt|nNt|t jjr t j|ntdt j S)af Override the heuristic PyTorch uses to choose between cuSOLVER and MAGMA for CUDA linear algebra operations. .. warning:: This flag is experimental and subject to change. When PyTorch runs a CUDA linear algebra operation it often uses the cuSOLVER or MAGMA libraries, and if both are available it decides which to use with a heuristic. This flag (a :class:`str`) allows overriding those heuristics. * If `"cusolver"` is set then cuSOLVER will be used wherever possible. * If `"magma"` is set then MAGMA will be used wherever possible. * If `"default"` (the default) is set then heuristics will be used to pick between cuSOLVER and MAGMA if both are available. * When no input is given, this function returns the currently preferred library. * User may use the environment variable TORCH_LINALG_PREFER_CUSOLVER=1 to set the preferred library to cuSOLVER globally. This flag only sets the initial value of the preferred library and the preferred library may still be overridden by this function call later in your script. Note: When a library is preferred other libraries may still be used if the preferred library doesn't implement the operation(s) called. This flag may achieve better performance if PyTorch's heuristic library selection is incorrect for your application's inputs. Currently supported linalg operators: * :func:`torch.linalg.inv` * :func:`torch.linalg.inv_ex` * :func:`torch.linalg.cholesky` * :func:`torch.linalg.cholesky_ex` * :func:`torch.cholesky_solve` * :func:`torch.cholesky_inverse` * :func:`torch.linalg.lu_factor` * :func:`torch.linalg.lu` * :func:`torch.linalg.lu_solve` * :func:`torch.linalg.qr` * :func:`torch.linalg.eigh` * :func:`torch.linalg.eighvals` * :func:`torch.linalg.svd` * :func:`torch.linalg.svdvals` N"Unknown input value. Choose from: .Unknown input value type.) r7r8_LinalgBackendsr9_LinalgBackends_strr$r%_set_linalg_preferred_backend_LinalgBackend_get_linalg_preferred_backendrs r)r r sX GS ! ! 8 / ) )K5HKKK  ..w/GHHHH GUX4 5 58 ..w77776777 8 1 1 3 33r()rcublashipblascublaslt hipblasltckc|nt|trL|tvrtdtdt jt|nNt|t jjr t j|ntdt j S)a Override the library PyTorch uses for BLAS operations. Choose between cuBLAS, cuBLASLt, and CK [ROCm-only]. .. warning:: This flag is experimental and subject to change. When PyTorch runs a CUDA BLAS operation it defaults to cuBLAS even if both cuBLAS and cuBLASLt are available. For PyTorch built for ROCm, hipBLAS, hipBLASLt, and CK may offer different performance. This flag (a :class:`str`) allows overriding which BLAS library to use. * If `"cublas"` is set then cuBLAS will be used wherever possible. * If `"cublaslt"` is set then cuBLASLt will be used wherever possible. * If `"ck"` is set then CK will be used wherever possible. * If `"default"` (the default) is set then heuristics will be used to pick between the other options. * When no input is given, this function returns the currently preferred library. * User may use the environment variable TORCH_BLAS_PREFER_CUBLASLT=1 to set the preferred library to cuBLASLt globally. This flag only sets the initial value of the preferred library and the preferred library may still be overridden by this function call later in your script. Note: When a library is preferred other libraries may still be used if the preferred library doesn't implement the operation(s) called. This flag may achieve better performance if PyTorch's library selection is incorrect for your application's inputs. Nrrr) r7r8 _BlasBackendsr9_BlasBackends_strr$r%_set_blas_preferred_backend _BlasBackend_get_blas_preferred_backendrs r)r r $s8 GS ! ! 8 - ' 'I5FIII  ,,]7-CDDDD GUX2 3 38 ,,W55556777 8 / / 1 11r(rGc|tj|tjS)a?Query or set the cuBLAS workspace size in bytes. When called with no arguments, returns the current workspace size. When called with a size argument, sets the workspace size and returns the new value. Setting the workspace size will take precedence over the CUBLAS_WORKSPACE_CONFIG environment variable. Changes take effect lazily: only handles used after the change get new workspaces. Args: size (int, optional): workspace size in bytes. Must be non-negative. Returns: int: the current (or newly set) workspace size in bytes. )r$r%_cuda_setCublasWorkspaceSize_cuda_getCublasWorkspaceSizerGs r)r r Ps4  --d333 8 0 0 2 22r(c|tj|tjS)aAQuery or set the cuBLASLt workspace size in bytes. When called with no arguments, returns the current workspace size. When called with a size argument, sets the workspace size and returns the new value. Setting the workspace size will take precedence over the CUBLASLT_WORKSPACE_SIZE environment variable. Changes take effect lazily: only handles used after the change get new workspaces. Args: size (int, optional): workspace size in bytes. Must be non-negative. Returns: int: the current (or newly set) workspace size in bytes. )r$r%_cuda_setCublasLtWorkspaceSize_cuda_getCublasLtWorkspaceSizers r)r r cs4  //555 8 2 2 4 44r(c@|t}nut|tr/|tvrt dt dt|}n1t|t jjr|}nt d|t jjj krt j }|t jjj krt d|t jjj krt|St|S)a{Query or set the BLAS workspace size for a given backend. Convenience wrapper that dispatches to :func:`cublas_workspace_size` or :func:`cublaslt_workspace_size` depending on the backend. When *backend* is ``None`` the current :func:`preferred_blas_library` is used. ``Default`` is resolved to the platform's default backend (cuBLAS on NVIDIA, potentially hipBLASLt on supported ROCm architectures). .. note:: When ``TORCH_CUBLASLT_UNIFIED_WORKSPACE`` is enabled (the default on open-source CUDA builds), the cuBLASLt workspace is capped at the cuBLAS workspace size and physically reuses the same allocation. Setting a large cuBLASLt workspace via this function will therefore *not* increase memory beyond the cuBLAS workspace size. .. note:: Setting the workspace size for the cublas backend will take precedence over the CUBLAS_WORKSPACE_CONFIG environment variable, and setting the workspace size for the cublaslt backend will take precedence over the CUBLASLT_WORKSPACE_SIZE environment variable. Args: size (int, optional): workspace size in bytes. Must be non-negative. When omitted the current size is returned without modification. backend (str | torch._C._BlasBackend, optional): which backend's workspace to query/set. Accepts the same strings as :func:`preferred_blas_library` (e.g. ``"cublas"``, ``"cublaslt"``). Returns: int: the current (or newly set) workspace size in bytes. Raises: RuntimeError: if the resolved backend is CK (no workspace concept). Nz%Unknown backend string. Choose from: rzUnknown backend type.z$CK backend does not use a workspace.)r r7r8rr9rr$r%rDefault_get_blas_default_backendCkCublasltr r )rGrresolveds r)rrvsR)++ GS ! ! 4 - ' 'L8ILLL !) GUX2 3 34233358(0008557758(+++ABBB58(111&t,,,  & &&r()raotritonr) _SDPAParams _SDPBackendc|nt|trL|tvrtdtdt jt|nWt|t jjr t j|ntdtdt j S)aY [ROCm-only] Override the backend PyTorch uses in ROCm environments for Flash Attention. Choose between AOTriton and CK .. warning:: This flag is experimental and subject to change. When Flash Attention is enabled and desired, PyTorch defaults to using AOTriton as the backend. This flag (a :class:`str`) allows users to override this backend to use composable_kernel * If `"default"` is set then the default backend will be used wherever possible. Currently AOTriton. * If `"aotriton"` is set then AOTriton will be used wherever possible. * If `"ck"` is set then CK will be used wherever possible. * When no input is given, this function returns the currently preferred library. * User may use the environment variable TORCH_ROCM_FA_PREFER_CK=1 to set the preferred library to CK globally. Note: When a library is preferred other libraries may still be used if the preferred library doesn't implement the operation(s) called. This flag may achieve better performance if PyTorch's library selection is incorrect for your application's inputs. Nrr) r7r8_ROCmFABackendsr9_ROCmFABackends_strr$r%_set_rocm_fa_preferred_backend_ROCmFABackend ValueError_get_rocm_fa_preferred_backendrs r)rrs0 GS ! ! V / ) )K5HKKK  //0HIIII GUX4 5 5V //8888T>QTTTUUU 8 2 2 4 44r(ztorch.backends.cudarc>tjS)z .. warning:: This flag is beta and subject to change. Returns whether composable_kernel may be used as the backend for scaled-dot-product-attention. )r$r%_is_ck_sdpa_availabler'r(r)rrs 8 ) ) + ++r(c>tjS)z .. warning:: This flag is beta and subject to change. Returns whether flash scaled dot product attention is enabled or not. )r$r%_get_flash_sdp_enabledr'r(r)rr 8 * * , ,,r(enabledcDtj|dS)z| .. warning:: This flag is beta and subject to change. Enables or disables flash scaled dot product attention. N)r$r%_set_sdp_use_flashrs r)rr  H(((((r(c>tjS)z .. warning:: This flag is beta and subject to change. Returns whether memory efficient scaled dot product attention is enabled or not. )r$r%_get_mem_efficient_sdp_enabledr'r(r)rr s 8 2 2 4 44r(cDtj|dS)z .. warning:: This flag is beta and subject to change. Enables or disables memory efficient scaled dot product attention. N)r$r%_set_sdp_use_mem_efficientrs r)rrs  H''00000r(c>tjS)z .. warning:: This flag is beta and subject to change. Returns whether math scaled dot product attention is enabled or not. )r$r%_get_math_sdp_enabledr'r(r)rrs 8 ) ) + ++r(cDtj|dS)z{ .. warning:: This flag is beta and subject to change. Enables or disables math scaled dot product attention. N)r$r%_set_sdp_use_mathrs r)rr's  Hw'''''r(cDtj|dS)z .. warning:: This flag is beta and subject to change. Enables or disables fp16/bf16 reduction in math scaled dot product attention. N)r$r%'_set_math_sdp_allow_fp16_bf16_reductionrs r)rr0s  H44W=====r(c>tjS)z .. warning:: This flag is beta and subject to change. Returns whether fp16/bf16 reduction in math scaled dot product attention is enabled or not. )r$r%'_get_math_sdp_allow_fp16_bf16_reductionr'r(r)rr9s 8 ; ; = ==r(c>tjS)a5Check if PyTorch was built with FlashAttention for scaled_dot_product_attention. Returns: True if FlashAttention is built and available; otherwise, False. Note: This function is dependent on a CUDA-enabled build of PyTorch. It will return False in non-CUDA environments. )r$r%_is_flash_attention_availabler'r(r)rrBs 8 1 1 3 33r(FparamsdebugcBtj||S)aCheck if FlashAttention can be utilized in scaled_dot_product_attention. Args: params: An instance of SDPAParams containing the tensors for query, key, value, an optional attention mask, dropout rate, and a flag indicating if the attention is causal. debug: Whether to logging.warn debug information as to why FlashAttention could not be run. Defaults to False. Returns: True if FlashAttention can be used with the given parameters; otherwise, False. Note: This function is dependent on a CUDA-enabled build of PyTorch. It will return False in non-CUDA environments. )r$r%_can_use_flash_attentionrrs r)rrO" 8 , ,VU ; ;;r(cBtj||S)aCheck if efficient_attention can be utilized in scaled_dot_product_attention. Args: params: An instance of SDPAParams containing the tensors for query, key, value, an optional attention mask, dropout rate, and a flag indicating if the attention is causal. debug: Whether to logging.warn with information as to why efficient_attention could not be run. Defaults to False. Returns: True if efficient_attention can be used with the given parameters; otherwise, False. Note: This function is dependent on a CUDA-enabled build of PyTorch. It will return False in non-CUDA environments. )r$r% _can_use_mem_efficient_attentionrs r)r r cs" 8 4 4VU C CCr(cBtj||S)aCheck if cudnn_attention can be utilized in scaled_dot_product_attention. Args: params: An instance of SDPAParams containing the tensors for query, key, value, an optional attention mask, dropout rate, and a flag indicating if the attention is causal. debug: Whether to logging.warn with information as to why cuDNN attention could not be run. Defaults to False. Returns: True if cuDNN can be used with the given parameters; otherwise, False. Note: This function is dependent on a CUDA-enabled build of PyTorch. It will return False in non-CUDA environments. )r$r%_can_use_cudnn_attentionrs r)r!r!wrr(c>tjS)z .. warning:: This flag is beta and subject to change. Returns whether cuDNN scaled dot product attention is enabled or not. )r$r%_get_cudnn_sdp_enabledr'r(r)rrrr(cDtj|dS)z| .. warning:: This flag is beta and subject to change. Enables or disables cuDNN scaled dot product attention. N)r$r%_set_sdp_use_cudnnrs r)rrrr(z`torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature.)categoryT enable_flash enable_mathenable_mem_efficient enable_cudnnc#Kddlm}g}|r|tj|r|tj|r|tj|r|tj||5} |Vn#wxYw ddddS#1swxYwYdS)a .. warning:: This flag is beta and subject to change. This context manager can be used to temporarily enable or disable any of the three backends for scaled dot product attention. Upon exiting the context manager, the previous state of the flags will be restored. r) sdpa_kernelN)torch.nn.attentionrappend SDPBackendFLASH_ATTENTIONEFFICIENT_ATTENTIONMATHCUDNN_ATTENTION)rrrrr backend_listcontexts r)r"r"s',/.....L8J6777<J:;;;-JO,,,8J6777 \ " "g MMM DDDDD s*B4B! B4!B##B44B8;B8r,)NN)F)TTTT)C contextlibtypingrtyping_extensionsrr$__all__rrrrr r%rrCusolverMagmarjoinkeysrr8r rCublasrrrrr intr r rrAOTritonrrtorch._Crrrrrr=r<rorrrrrrrrrrrr r!rrcontextmanager FutureWarningr"rrr'r(r)rs((((((    D + + + + + + + + @@@@@@@@2#4#4#4#4#4#4#4#4LN:N:N:N:N:N:N:N:dx&.'0 X $ * ii 4 4 6 677599494 CZ%(1 194 X94949494zx$,h#*x$+%.&/ (  "  IIm00223337)2)2 CZ%(/ /)2 X)2)2)2)2X33s 3c3333&55$*55555(26>'>' *>' CZ%(/ />' >'>'>'>'Dx&.'0 ( ! $ ii 4 4 6 677JIIIIIII59%5%5 CZ%(1 1%5 X%5%5%5%5R. " ,d,,,,---)d))))5551d1111,,,(T((((>>>>>>>> 4d 4 4 4 4<)(** r(