[jEdZddlZddlmZddlmZmZddlmZddlm Z ej e Z dZ Gd d eZejZejejZejejZd ejfd Z dd ejdejdejdejdejdejdzdejfdZGddejZddZdZGddeZGddeZdS)z"CompressedTensors integration fileN)nn) ConversionOps _IdentityOp)logging)is_torch_greater_or_equalct|j}|D]$}td|jDr|cS%|D]$}td|jDr|cS%|dS)a/Resolve which config group quantizes the MoE experts. Mixed configs quantize different layers with different schemes (e.g. Kimi: FP8 attention + INT4 experts), so the experts' scheme cannot be assumed global: the experts' group is the one whose `re:` targets mention the expert modules. c3HK|]}|dod|vVdS)re:expertsN startswith.0targets ZC:\Projects\DSCons\.venv\Lib\site-packages\transformers\integrations\compressed_tensors.py z%get_experts_scheme..#s:\\Fv  ''?I,?\\\\\\c3BK|]}|d VdS)r Nr rs rrz%get_experts_scheme..(s3HH6$$U+++HHHHHHrr)list config_groupsvaluesanytargets)quantization_configgroupsgroups rget_experts_schemers %3::<< = =F \\em\\\ \ \ LLL  HH%-HHH H H LLL  !9rc eZdZdZd dZ d deeejfde ede ededzdeeejff d Z e d d Z dS)DecompressExpertsa Dequantize MoE layers when they are in new layout, because they aren't `nn.Module` anymore! Takes packed weights and scales from the loaded state dict, creates a dummy Module to take advantage of higher-lvl API `decompress_module` and dequantizes all weights. Requires MoE conversion to be defined on conversion mapping, so that decompressed weights are stacked/merged for all experts. Nc"||_||_dSN) hf_quantizerscheme)selfr#r$s r__init__zDecompressExperts.__init__8s( r input_dictsource_patternstarget_patternsfull_layer_namereturnc ddlm}ddlm}|jjj}|jpt|} | j } | j dkrm| j dkrai} | D]G\} } d| vr | dd}||vr| | | <+t| tr| n| g}||}t|tr|n|g}d}t!t#||D]\}\}}|t&j|t&jzt&j}|7t'jt/|g|jR|j|j }|||~||| | <I| S| jp|t:j| }||}Gd d t:j }d | j j z}i} | D]i\} } d | vr | }|| d d}|!| d d}d}t!t#||D]\}\}}|dn||}||"r|}n/t'j#|jd|jd|zg}||||}| |_$|%||Ft'jt/|g|j&jR|j&j|j&j }|||j&~~~||| | <k| S)Nr)BaseCompressor)infer_module_formatfloat weight_scalez.weightz .weight_scale)dtypedeviceceZdZfdZxZS).DecompressExperts.convert..DummyModulecttj|d|_tj|d|_tj|d|_dS)NF) requires_grad)superr&r Parameter weight_packedr1 weight_shape)r%weightscaleshape __class__s rr&z7DecompressExperts.convert..DummyModule.__init__msd  """%'\&%N%N%N"$&Le$L$L$L!$&Le$L$L$L!!!r)__name__ __module__ __qualname__r& __classcell__r?s@r DummyModuler5lsA M M M M M M M M MrrE r:r;)'compressed_tensors.compressorsr-%compressed_tensors.compressors.formatr.r# compressorrr$rweightstypenum_bitsitemsreplace isinstancer enumerateziptotorchfloat32bfloat16emptylenr>r2r3copy_formatrLinearget_value_from_registryModulegetnumeltensorquantization_schemedecompress_moduler<)r%r'r(r)r*kwargsr-r.ct_quantization_configra weights_args processed_outkeyvalue scale_key quantizedscalesoutputiquantr=denserZrJrE pack_factorshapes stored_shaper>modules rconvertzDecompressExperts.convert>s BAAAAAMMMMMM!%!2!=!Q"kW-?@V-W-W*2   ' 'L,AQ,F,FM(..00 0 0 U!S((KK ?CC J..).M#&%/t%<%<IEE5' #I.#-fd#;#;I&)23y&3I3I)J)J%A~u"XXem44uxx 7N7NNRRSXSabbE~!&c)nn-Ku{-K-KSXS^glgs!t!t!t1IOOE***%)/M#& $+b/B/B29Na/b/b#;;FCC  M M M M M") M M M/7@@  $**,,( ,( ,JCc))I O^ L LMF^^CKK$P$PQQF F%.s9f/E/E%F%F  !>E5 (.~tt6!9 + 0B0B0D0D+(EE!L%+a.%+a.;:V)WXXE$UE599-@*,,V444>"[Y>&-*=>>$m1%}3F q  ...F6!&, c"rrctSr")rr%s r reverse_opzDecompressExperts.reverse_ops }}rr")r+r) r@rArB__doc__r&dictstrrTTensorrrtpropertyrwrrr r -s'+ ggel*+gcgc g t g c5< ggggRXrr xc|tj}|d}tj|dk|tj|}|tz }tj|| dz ttt}||fS)a:Quantize a 2D tensor to FP8 per-row using pure PyTorch (torch.compile compatible). Args: x: Input tensor of shape (num_rows, hidden_dim). Returns: Tuple of (x_fp8, scales) where x_fp8 is the quantized tensor and scales is a 1D tensor of shape (num_rows,) with per-row scales. )dimr)minmax) rSrTrUabsamaxwhere ones_like_FP8_MAXclamp unsqueeze_FP8_MIN _FP8_DTYPE)r~x_floatrow_maxsafe_maxrkx_fp8s rquantize_fp8_per_rowrsdd5=!!Gkkmm  R ((G{7Q;1I1IJJH  F K&"2"22"6"66H( S S S V VWa b bE &=rinputr<scale_ascale_b out_dtypebiasr+c tddrLtjjj}tjj||||j||j||Stj||||||S)a4Row-wise scaled FP8 matmul dispatcher that uses the public `torch.nn.functional.scaled_mm` (torch >= 2.10) if available, else falls back to `torch._scaled_mm`. Args: input (`torch.Tensor`): FP8 input of shape (M, K), scaled per-row. weight (`torch.Tensor`): FP8 weight of shape (K, N), scaled per-column. scale_a (`torch.Tensor`): Input scales of shape (M, 1), float32. scale_b (`torch.Tensor`): Weight scales of shape (1, N), float32. Returns: `torch.Tensor`: Output tensor of shape (M, N) in `out_dtype`. z2.10T) accept_dev)r output_dtyperrrr)rrTr functional ScalingType scaled_mmRowWise _scaled_mm)rr<rrrrrs r_scaled_mm_rowwisers(!D999  h)5 x",,        "-     E67GW`gk l l llrcVeZdZdZd dededeffd Zdejdejfd Z xZ S) CompressedTensorsFP8LinearaLinear layer for compressed-tensors FP8 models. Stores weights in FP8 format and uses a row-wise scaled matmul (`_scaled_mm_rowwise`) for FP8 matmul. Activations are always dynamically quantized per-row via ``quantize_fp8_per_row``, which works for both ``dynamic`` and ``static`` checkpoints (the checkpoint's static input scale, if any, is simply not needed). The weight scale (per-channel or per-tensor) is stored as ``weight_scale``. F in_features out_featureshas_biasct||||_tjt j||t|_tjt j d|ft j |_ |jr2tjt j|j |_ dS|dddS)N)r2rGr)r8r&rrr9rTrWrr<zerosrUr1rrregister_parameter)r%rrrr?s rr&z#CompressedTensorsFP8Linear.__init__s l333  l5;|[PZ#[#[#[\\ La5Fem)\)\)\]] = 2 U[1B%C%CDDDIII  # #FD 1 1 1 1 1rrr+cVg|jdddR}|d|jd}t|\}}t||j|d|j|j|j }||S)Nrr) r>reshaperrr<trr1r2r)r%r output_shaper~ x_quantizedx_scalerls rforwardz"CompressedTensorsFP8Linear.forwards/SbS).2.. MM"ek"o . .3A66 W#  KMMOO%%b))%k    ~~l+++r)F) r@rArBrxintboolr&rTr{rrCrDs@rrrs 2 2C 2s 2d 2 2 2 2 2 2,U\,el,,,,,,,,rrcddlm}ddlm}d}|d||||D}g}|D]\}} |||vr |||st | t jrNt| j | j | j du} | || | ||std |S) uReplace nn.Linear modules with CompressedTensorsFP8Linear for compressed-tensors FP8 loading. Quantization may only cover part of the model: `targets` / `ignore` are the compressed-tensors patterns (class names or `re:` regexes) from the config — only matching modules are replaced, everything else keeps its dense weights. `modules_to_not_convert` holds additional transformers-side exclusions (e.g. `_keep_in_fp32_modules`). r)match_named_modulesr)should_convert_moduleNch|]\}}|Sr}r})rname_s r z=replace_with_compressed_tensors_fp8_linear..!sZZZGD!4ZZZr)ignore)rrrzYou are loading your model using compressed-tensors FP8 but no linear modules were found. Please double check your model architecture.)compressed_tensors.utils.matchrquantizers.quantizers_utilsr named_modulesrPrr[rrrr set_submoduleappendloggerwarning) modelrrmodules_to_not_convertrrmatchedreplaced_names module_namers new_modules r*replace_with_compressed_tensors_fp8_linearrsGCBBBBBDCCCCCGZZ':':5'RX'Y'Y'YZZZN$2244// V  ;g#=#= $$[2HII   fbi ( ( /4".#0D0J    Z 8 8 8  ! !+ . . .   ;    rc||ddd}t|tr|ndS)zReturn the `CompressedTensorsFP8Linear` owning `full_layer_name` (a `weight_scale` key), or None if the key belongs to another kind of module (e.g. a non-FP8 group of a mixed config)..rGrN) get_submodulersplitrPr)rr*rss r_get_fp8_linearr=sF !7!7Q!?!?!B C CF(BCC M66Mrc6eZdZdZddZedefdZdS)ConvertFP8LinearScalea?Reshape the checkpoint ``weight_scale`` into the row-wise kernel layout at load time. compressed-tensors checkpoints store the dequantization scale as ``(out_features, 1)`` (channel strategy) or a single element (tensor strategy). The row-wise scaled matmul in :class:`CompressedTensorsFP8Linear` needs a contiguous float32 ``(1, out_features)`` scale; converting once at load time avoids a transpose + expand + copy on every forward call. Scales of modules that were not replaced with :class:`CompressedTensorsFP8Linear` pass through untouched. Nc tt|\}}t|tr|dn|}t ||}|||iS|tj dd}|j d|j kr-| d|j  }||iS)NrrGr)nextiterrNrPrrrSrTrUrr>rexpand contiguous)r%r'rr*rcrgr=rss rrtzConvertFP8LinearScale.convertOs$z//112233 U&ud33>a 88 >< ''//266 ;q>V0 0 0LLF$788CCEEEU|rr+ctSr")RevertFP8LinearScalervs rrwz ConvertFP8LinearScale.reverse_op\s#%%%rNNr@rArBrxrtr|rrwr}rrrrDsY    &M&&&X&&&rrc6eZdZdZddZedefdZdS)ra\Save-time inverse of :class:`ConvertFP8LinearScale`: restore the checkpoint layout, ``(out_features, 1)`` for the channel strategy and ``(1, 1)`` for the tensor strategy. The strategy is read from the config group targeting the module, so the saved checkpoint stays loadable by other compressed-tensors consumers (llm-compressor, vLLM).Nc nddlm}tt|\}}t |t r|dn|}t||}|||iS|ddd} |j j j } d} | j D]1} | j (|| || j| jpdr| j j} n2| dkr"||ddddfiS||iS)Nr)is_matchrrGr}r`)rrrrrNrPrrrconfigrrrrKrrstrategyclonerr) r%r'rr*rcrrgr=rsr ct_configrrs rrtzRevertFP8LinearScale.convertgsG;;;;;;$z//112233 U&ud33>a 88 >< %,,S!44Q7 L4H ,3355  E}(XXk65=ZcZjZpnp-q-q( =1 x  qqq"1"u++--. .UWWYY))++,,rr+ctSr")rrvs rrwzRevertFP8LinearScale.reverse_op|s$&&&rrrr}rrrras[TT ----*'M'''X'''rrr")NNN) rxrTrcore_model_loadingrrutilsrutils.import_utilsr get_loggerr@rrr float8_e4m3fnrfinforrrrr{rr2rr[rrrrrr}rrrs;%$ ;;;;;;;;::::::  H % %$||||| |||~ 5;z " " & 5;z " " &EL2!% m m < m L m\ m\ m { m ,  m \ m m m mF&,&,&,&,&,&,&,&,R****ZNNN&&&&&M&&&:'''''='''''r