˲j4 4UdZddlZddlmZmZmZddlZgZeee d<dej de fdZ dej d e dej fd Zd e d eedefd ZgdZdee ejfde fdZ ddej dej dej deej fdZdS)zCDefines utilities for interacting with scaled_dot_product_attentionN)ListOptionalUnion__all__tensorsreturnc4td|DS)z0Returns True if any of the tensors requires gradc3$K|] }|jV dS)N) requires_grad).0ts JC:\Projects\Historical\venv\Lib\site-packages\torch/nn/attention/_utils.py z'_input_requires_grad..s$001q000000)any)rs r_input_requires_gradr s 00000 0 00r inpt_tensorog_sizecP|d|kr |dd|fS|S)z'Handles the unpad of the last dimension.N)size)rrs r_postprocess_flash_outputrs4w&&3=)) r head_dim_sizescalec8||Sdtj|z S)z For FlashAttention we pad the head dimension to be a multiple of 8 so we need to scale the output by the original head size and not the padded. Ng?)mathsqrt)rrs r_calculate_scalers$   =)) ))r)  @iinc|tvS)z:Returns true if the head dim is supported by FlexAttention)_SUPPORTED_HEAD_DIMS)r's r_supported_head_dimr*%s $ $$rFquerykeyvalue attn_maskc T|j|jks|j|jkr(td|jd|jd|jd|j|jks|j|jkr(td|jd|jd|jd|dks0|dks|dkrOtd |d |d |ddS) NzLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: z instead.zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: rzUExpected query, key, and value to all be at least 2 dimensional, but got query.dim: z , key.dim: z and value.dim: )dtype ValueErrordevicedim)r,r-r.r/ dropout_p is_causalrs r_validate_sdpa_inputr7*so {ci5;%+#=#= 7$)K 7 7>Ai 7 7 %  7 7 7    |sz!!U\U\%A%A 9%*\ 9 9AD 9 9!& 9 9 9    yy{{Q#''))a--599;;?? Yyy{{ Y Y'*wwyy Y YBG))++ Y Y Y   ,;?r)Nr+FN)__doc__rtypingrrrtorchrstr__annotations__Tensorboolrintrfloatrr)SymIntr*r7rrrCsqIII (((((((((( c15<1D1111 5<#%,*C**E****BAA%5el!23%%%%%)-   <   <  %       r