mj"8UddlZddlZddlmZddlmZddlmZmZm Z ddl m Z m Z e j eZe rddlZerddlmZdZ d"d ed d ed d edzdedzdedeff dZ d#d ed d ed d edzdedzdededeff dZ d"d ed d ed d edzdedzdedeff dZ d$d d d ed d edzdedzdedeff dZ d$d d d ed d edzdedzdedeff dZ d$d d d ed d edzdedzdedeff dZeeeeeedZeeededefffe d<Gdde Z!GddZ"d%d e"d e#dzfd!Z$dS)&N)Callablewraps) TYPE_CHECKINGOptional TypedDict)is_torch_availablelogging)PreTrainedConfigcVddddtdfd }|S)ad Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. Ncbtj|dz}|#|j}|j}d}|jjd}n=|j|}t ||d}|d}|jj|d}||krkt||ds't|} | |j||dz|\} } | |d | d t||d | dS| |}| |d |d t||d |dS)zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r N original_max_position_embeddings_original_inv_freq__long_inv_freqseq_len layer_typeinv_freqF persistent long_inv_freqoriginal_inv_freq) torchmax rope_typerconfigrope_parametersgetattrhasattrROPE_INIT_FUNCTIONSregister_buffersetattrto) self position_idsdevicerrrrprefixr rope_init_fnrrs NC:\Projects\DSCons\.venv\Lib\site-packages\transformers\modeling_rope_utils.pylongrope_frequency_updatez6dynamic_rope_update..longrope_frequency_update/s)L))A-  I $ 6 F/3{/JKm/n , ,z2I '.O.O.O P P "%%%F/3{/J:/V20 , 5 5 54J!>!>!>?? 29= #/<K|j|}t ||d|j}t ||d}|d}||krXt |} | |j|||\} |_| |d| d t||d |||j krj||j kra| |}| |d|d t||d |t||d |j dSdSdS) a dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) r Nr_max_seq_len_cachedrrrrFrmax_seq_len_cachedr) rrrr1rr!r#rattention_scalingr$r%original_max_seq_lenr&) r'r(r)rrrr1rr*r+rs r,dynamic_frequency_updatez5dynamic_rope_update..dynamic_frequency_updateRs )L))A-  I!%!8  $ 6 FFz2I!(*/Q/Q/QSWSj!k!k  '.O.O.O P P "%%%F ' ' '.y9L/;| % 000 ,Hd,  F!4!4!4h5 Q Q Q DV777 A A A T. . .3EHa3a3a!2 4 4V < <   F!4!4!46GTY Z Z Z DV6668I J J J DV7779R S S S S S / .3a3ar.c||jn |j|}|d|ini}d|vr||fd|ji|n|dkr||fd|ji||||fi|S)Nrdynamicr)longrope)rr)) r'xr(rrkwargsr4r- rope_forwards r,wrapperz$dynamic_rope_update..wrapperxs&0&8DNNdnZ>X /9/E, ++2  ! ! $ $T< S S SF S S S S * $ $ % %dL T T TV T T T|D!\<> FW$v'EE&ST*U[^behibi[jk kDdu|AsAU[IIILLTZbgbmLnnqttuvH % %%rXcN | |j|n|}n#t$rYnwxYw|| |j|n|j}|d}|dd}t |d|j|jz}t||z}|d} |d} |d} |d } |d } | |j | z } dd }| 6| r)| r't|| | || | z } n || } |d pd}|dpd }dfd}d}|tj d|d |tj |z z}d|z }d| |zz }|jdd}|||||| |\}}d ||||dz |tj z }|d |z z||zz}|| fS)aD Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * rope_parameters (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied to the computed cos/sin. If None, the value is inferred from `factor`, `mscale`, and `mscale_all_dim` as available. * `beta_fast` (`float`, *optional*, defaults to 32): Parameter to set the boundary for extrapolation (only) in the linear ramp function. * `beta_slow` (`float`, *optional*, defaults to 1): Parameter to set the boundary for interpolation (only) in the linear ramp function. * `factor` (`float`, *optional*): The scaling factor applied when interpolating the position IDs to extend the possible context length. Additionally, if `attention_factor` is None, the log of this value is used to compute a value for `attention_factor`, possibly in conjunction with `mscale` and `mscale_all_dim`, if provided. * `mscale` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale` acts scalar augmenting `log(factor)` when computing the numerator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `mscale_all_dim` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale_all_dim` acts scalar augmenting `log(factor)` when computing the denominator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `original_max_position_embeddings` (`int`): The original max position embeddings used during pretraining. * `truncate` (`bool`, *optional*): Whether to truncate the correction range. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. NrArBrCrDr@rVmscalemscale_all_dimrr cL|dkrdSd|ztj|zdzS)Nr rCg?)mathlog)scalerls r, get_mscalez,_compute_yarn_parameters..get_mscales, A::3V|dhuoo-33r. beta_fast beta_slowc|tj||dztjzz zdtj|zz S)zPInverse dimension formula to find the dimension based on the number of rotationsrE)rorppi) num_rotationsrUrTres r,find_correction_dimz5_compute_yarn_parameters..find_correction_dimsAdh6-!:Kdg:UVWWW\]`d`him`n`n\noor.c||||}||||}|r(tj|}tj|}t|dt ||dz fS)z.Find dimension range bounds based on rotationsrr )rofloorceilrmin) low_rothigh_rotrUrTretruncatelowhighrys r,find_correction_rangez7_compute_yarn_parameters..find_correction_rangest!!'36MNN""8S$8OPP  #*S//C9T??D3{{CcAg....r.c||kr|dz }tj|tj|z ||z z }tj|dd}|S)NgMbP?rFrr )rrPr^clamp)r}rrU linear_func ramp_funcs r,linear_ramp_factorz4_compute_yarn_parameters..linear_ramp_factorsQ #:: 5LC|Cu}===Cc R K Q22 r.rrErHrT)r )rIrJrKr rLr!rMrNrOrerRrrPr&)rr)rrrSrTrBrDrUr@rVrlrmrrrrsrurr pos_freqsinv_freq_extrapolationinv_freq_interpolationrrrinv_freq_extrapolation_factorrrys @r,_compute_yarn_parametersrYst 8B8N(44TZ       ""$$$AKAW61*==]c]s  -D0445LcRRvz6+=A[+[\\H h.. / /C !( +F+//0BCC ! % %h / /F)--.>??N';<^'_$ ~/2RR4444   2n 2$ZZ%?%?**VUcBdBd%dee  )z&11 %((55;I$((55:Ippp/////aa003363UUX[[\I 9_ FY$67%))*d;;H%%iCGgiqrrIC%&(:(:3cQh(O(O(R(RZ`hmhs(R(t(t$t!!&C"CD #@ @ A  % %%s  ""c | |j|n|}n#t$rYnwxYw|| |j|n|j}|d}|dd}t |d|j|jz}t||z}|d} |d} |d} |d } |d } | |j | z } | G| dkrd} n>tj d tj | tj | z z} |r(|| kr"tj| tj| }n!tj| tj| }tjd |dtj| |z }d|||zzz }|| fS)a Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * original_max_position_embeddings (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, defaults to `max_position_embeddings`. * rope_parameters (`dict[str, float]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, inferred from the value of `factor`. * `factor` (`float`, *optional*): The scaling factor to apply to the RoPE embeddings. If both `max_position_embeddings` and `original_max_position_embeddings` are provided, this value will be overridden s the ratio between those values. * `long_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is provided and greater than `original_max_position_embeddings`. * `short_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. NrArBrCrD long_factor short_factorr@rVrr r[rrE)rIrJrKr rLr!rMrNrOrerosqrtrprrir^rPrQrR)rr)rrrSrTrBrDrUrrr@rVr ext_factorsinv_freq_shapers r,_compute_longrope_parametersrsd 8B8N(44TZ       ""$$$AKAW61*==]c]s  -D0445LcRRvz6+=A[+[\\H h.. / /C&}5K'7L ! % %h / /F+//0BCC';<^'_$ ~/2RR S=="  #yTXf-=-=Ii@j@j-j)jkk U7===l;emFSSS l?*+MNO&8'*::$'kH$G[+;!;X=NPXYYN$w.@EUXgEghM]*n