Sj$UdZddlZddlmZddlmZddlmZddlZ ddl m Z ddlm Z mZmZgdZeeed <d Z ee _d e _d ed dfdZddddddZdefdZd"defdZd"deded dfdZejd"dee e zdefdZejde fdZd efdZ dd l!m"Z"e"j#Z#e"j$Z$e"j%Z%e"j&Z&e"j'Z'e"j(Z(e"j)Z)ee%_ee&_ee'_ee(_ee)_dd!l!m*Z*m+Z+dS)#zvThis module contains functions and classes that alter the behavior of torch.nn.functional.scaled_dot_product_attentionN)Iterable)Union)warn) _SDPBackend)can_use_efficient_attentioncan_use_flash_attention SDPAParams) SDPBackend sdpa_kernelWARN_FOR_UNFUSED_KERNELSregister_flash_attention_implactivate_flash_attention_impllist_flash_attention_implscurrent_flash_attention_implrestore_flash_attention_impl__all__Fr paramsreturnctrbt|s!tddt|dt|s%tddt|ddSdSdS)z If WARN_FOR_UNFUSED_KERNELS is set to True, this will raise warnings for all the reasons why the fused kernels can't be run. If using subclasses z*Efficient attention can't be used because:) stacklevelTz&Flash attention can't be used because:N)r rrr)rs IC:\Projects\DSCons\.venv\Lib\site-packages\torch\nn\attention\__init__.py_raise_kernel_warningsr;s  2*622 6 =! L L L L ' 5 5 5&v.. 2 9a H H H H #FD 1 1 1 1 1 22 2 2CUDNN_ATTENTIONFLASH_ATTENTIONEFFICIENT_ATTENTIONMATH OVERRIDEABLE)cudnnflash mem_efficientmath overrideablenamec,tt|SN)getattrr )r%s r_backend_from_stringr)Rs :t $ $$r with_prioritycNg}tD]S\}}ttjd|dr(|tt |T|r2tjt|fd}|S)N_get_ _sdp_enabledcHt|Sr')indexint)backend curr_prioritys rz+_cur_sdpa_kernel_backends..^s-*=*=c'll*K*Kr)key) _backend_namesitemsr(torch_Cappendr _get_sdp_priority_ordersorted)r*backendsr%valr2s @r_cur_sdpa_kernel_backendsr>VsH#))++66 c 87587T777 8 8 : : 6 OOGJ44 5 5 5 88::  KKKK    Orr< set_priorityctD]B\}}tt||v}ttjd||C|rvd|D}tj}|D](}||vr"|t|)tj |dSdS)N _set_sdp_use_c,g|]}t|S)r0).0r1s r z _sdpa_kernel..is>>>'W>>>r) r5r6r(r r7r8r:r9r0_set_sdp_priority_order)r<r?r%r=enabled user_priorityprevious_priorityr1s r _sdpa_kernelrJcs#))++;; c*c**h610$0011'::::8>>X>>> !H<<>>( 3 3Gm++$$S\\222 ((7777788rc#Kt|ttfs$tdt |jt|tr|g}tt |}t|} t||iVt||dS#t||wxYw)aP Context manager to select which backend to use for scaled dot product attention. .. warning:: This function is beta and subject to change. Args: backends (Union[List[SDPBackend], SDPBackend]): A backend or list of backends for scaled dot product attention. set_priority (bool=False): Whether the ordering of the backends is interpreted as their priority order. Example: .. code-block:: python from torch.nn.functional import scaled_dot_product_attention from torch.nn.attention import SDPBackend, sdpa_kernel # Only enable flash attention backend with sdpa_kernel(SDPBackend.FLASH_ATTENTION): scaled_dot_product_attention(...) # Enable the Math or Efficient attention backends with sdpa_kernel([SDPBackend.MATH, SDPBackend.EFFICIENT_ATTENTION]): scaled_dot_product_attention(...) # Enable the cuDNN or flash attention backends, and in that order with sdpa_kernel( [SDPBackend.CUDNN_ATTENTION, SDPBackend.FLASH_ATTENTION], set_priority=True ): scaled_dot_product_attention(...) This context manager can be used to select which backend to use for scaled dot product attention. Upon exiting the context manager, the previous state of the flags will be restored, enabling all backends. zQBackend must be an instance of SDPBackend or a list of SDPBackend instances, got )r*N) isinstancelistr AssertionErrortype__name__dictfromkeysr>rJ)r<r?previous_backendss rr r qsF hz 2 3 3  y`dem`n`n`w y y   (J'':DMM(++,,H1 MMM6X|,,,& 55555 & 5555s B99C c'|Ktt|5dVddddS#1swxYwYdSr')r rM)r<s r_sdpa_kernel_variadicrUs T(^^ $ $ s 155cdS)zEThis returns the closest matching tag for the flash attention backendz2.5.7rCrCrr_get_flash_versionrWs 7r) _registry)_fa3_fa4)F),__doc__ contextlibcollections.abcrtypingrwarningsrtorch.backends.cudar7torch._Crr rrr rrMstr__annotations__r rP __module__rr5r)boolr>rJcontextmanagerr rUrWrY_FlashAttentionImpl _RegisterFnr rrrrrZr[rCrrrks|||$$$$$$......   c   &!! "  2: 2$ 2 2 2 2 * " %s%%%%  T     8 88 84 8D 8 8 8 8 1616$z*Z716t16161616j Z C  3# ) G ) G&A(E(E+3(+3((0%*2'*2'r