+j%ddlmZddlmZddlZddlmZddlmcmZ ddl m Z ddl m Z dd ZGd d ejZGd dejZdS)) annotations)IterableN)"SparseMultipleNegativesRankingLoss) SparseEncoderreconstruction torch.Tensororiginal_inputreturnc|d}tj||tj|dddf|j|z }|S)z :param reconstruction: output of Autoencoder.decode (shape: [batch, n_inputs]) :param original_input: input of Autoencoder.encode (shape: [batch, n_inputs]) :return: normalized mean squared error (shape: [1]) r)dimN)meanFmse_loss broadcast_toshape)rr original_input_meanlosss ]C:\Projects\DSCons\.venv\Lib\site-packages\sentence_transformers\sparse_encoder\losses\csr.pynormalized_mean_squared_errorrsi )--!-44 :nn 5 5 D!!!G$11.2FGG99 D Kc8eZdZddfd Zdd ZddZdZxZS)CSRReconstructionLoss?modelrbetafloatr Nonecdt||_||_dS)a CSRReconstructionLoss implements the reconstruction loss component for Contrastive Sparse Representation (CSR) models. This loss ensures that the sparse encoding can accurately reconstruct the original model embeddings through three components: 1. A primary reconstruction loss (L_k) that measures the error between the original embedding and its reconstruction using the top-k sparse components. 2. A secondary reconstruction loss (L_4k) that measures the error using the top-4k sparse components. 3. An auxiliary loss (L_aux) that helps to learn residual information. Args: model: SparseEncoder model with autoencoder components beta: Weight for the auxiliary loss component (L_aux) References: - For more details, see the paper "Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation" https://huggingface.co/papers/2503.01776 Requirements: 1. The model must be configured to output the necessary reconstruction components 2. Used with SparseEncoder models that implement compositional sparse autoencoding Relations: - Used as a component within :class:`CSRLoss` combined with a contrastive loss Example: - This loss is never used standalone, but instead used within the :class:`CSRLoss` class. See that loss for more details. N)super__init__rr)selfrr __class__s rr zCSRReconstructionLoss.__init__s-<   rsentence_features!Iterable[dict[str, torch.Tensor]]dict[str, torch.Tensor]c td)Nz[CSRReconstructionLoss is not intended to be used standalone. Use it within CSRLoss instead.)NotImplementedError)r!r#s rforwardzCSRReconstructionLoss.forward?s! i   routputslist[dict[str, torch.Tensor]]cd}d}d}|D]}|d}|d}|d}|d} |d} tj||} tj||} t| || z } || z }|| z }|| z }t |}|dkr||z}||z}||z}||dz |j|zd S) a Compute the CSRReconstruction loss from embeddings. Args: outputs: List of dictionaries containing embeddings and their sparse representations Returns: total_loss: The total reconstruction loss value gsentence_embedding_backbonedecoded_embedding_kdecoded_embedding_4kdecoded_embedding_auxdecoded_embedding_k_pre_biasrg @)reconstruction_loss_kreconstruction_loss_4kreconstruction_loss_aux)rrrdetachlenr)r!r) total_L_k total_L_4k total_L_auxfeaturesxrecons_k recons_4k recons_auxreconsk_pre_biasL_kL_4kL_aux num_columnss rcompute_loss_from_embeddingsz2CSRReconstructionLoss.compute_loss_from_embeddingsDs!     ! !H67A 56H !78I!"9:J'(FG *Q))C:a++D2*aBRBYBYB[B[>[\\E  I $ J 5 KK'll ??  $I + %J ; &K&/&03&6'+y;'>   rcd|jiS) Get the configuration dictionary. Returns: Dictionary containing the configuration parameters r)rr!s rget_config_dictz%CSRReconstructionLoss.get_config_dictws ""r)r)rrrrr r)r#r$r r%)r)r*r r%)__name__ __module__ __qualname__r r(rCrG __classcell__r"s@rrrs{       D    1 1 1 1 f#######rrcLeZdZddfd Z dddZdZeddZxZS)CSRLossN皙?rrrrnn.Module | Nonerrgammact||_||_||_t |||_||nt||_dS)a9 CSRLoss implements a combined loss function for Contrastive Sparse Representation (CSR) models. This loss combines two components: 1. A reconstruction loss :class:`CSRReconstructionLoss` that ensures the sparse representation can faithfully reconstruct the original embedding. 2. A main loss, which in the paper is a :class:`SparseMultipleNegativesRankingLoss` that ensures semantically similar inputs have similar representations. The total loss is linear combination of the two losses. Args: model: SparseEncoder model loss: The principal loss function to use can be any of the SparseEncoder losses except flops loss and CSRReconstruction loss. If None, the default loss is used, which is the SparseMultipleNegativesRankingLoss. beta: Weight for the L_aux component in the reconstruction loss. Default is 0.1. gamma: Weight for the main loss component (MNRL a.k.a. InfoNCE by default). Default is 1.0. References: - For more details, see the paper "Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation" https://huggingface.co/papers/2503.01776 Requirements: 1. Input requirements depend on the chosen loss 2. Uses autoencoder components of the SparseEncoder model Relations: - Uses :class:`CSRReconstructionLoss` for the reconstruction component Example: :: from datasets import Dataset from sentence_transformers.sparse_encoder import SparseEncoder, SparseEncoderTrainer, losses model = SparseEncoder("sentence-transformers/all-MiniLM-L6-v2") train_dataset = Dataset.from_dict( { "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], "negative": ["It's quite rainy, sadly.", "She walked to the store."], } ) loss = losses.CSRLoss(model, beta=0.1, gamma=1.0) trainer = SparseEncoderTrainer(model=model, train_dataset=train_dataset, loss=loss) trainer.train() )rrNr) rr rrrQrreconstruction_lossrr)r!rrrrQr"s rr zCSRLoss.__init__sid    $9u4#P#P#P  ,DD2T[`2a2a2a rr#r$labelstorch.Tensor | Noner r%c:fd|D}d|D}j|}j||}t|tr(|D]\}}|jz||<n |jz|d<|S)Nc:g|]}|SrS).0sentence_featurer!s r z#CSRLoss.forward..s'ZZZ4D4::.//ZZZrcg|] }|d S)sentence_embeddingrY)rZoutputs rr\z#CSRLoss.forward..sQQQvf%9:QQQr base_loss)rTrCr isinstancedictitemsrQ) r!r#rUr)r^lossesr`keyvalues ` rr(zCSRLoss.forwards[ZZZHYZZZQQQQQ)FFwOOI::;MvVV i & & 9'oo// 1 1 U#dj0s  1#,dj"8F;  rcP|j|jt|jjdS)rE)rrQr)rrQtyperrHrFs rrGzCSRLoss.get_config_dicts# DJTY@XYYYrstrcdS)Na @misc{wen2025matryoshkarevisitingsparsecoding, title={Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation}, author={Tiansheng Wen and Yifei Wang and Zequn Zeng and Zhong Peng and Yudi Su and Xinyang Liu and Bo Chen and Hongwei Liu and Stefanie Jegelka and Chenyu You}, year={2025}, eprint={2503.01776}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2503.01776}, } rYrFs rcitationzCSRLoss.citations   r)NrOr)rrrrPrrrQr)N)r#r$rUrVr r%)r ri) rHrIrJr r(rGpropertyrkrKrLs@rrNrNs9b9b9b9b9b9b9bxcg$ZZZ   X     rrN)rrr rr r) __future__rcollections.abcrtorchtorch.nnnntorch.nn.functional functionalrMsentence_transformers.sparse_encoder.losses.sparse_multiple_negatives_rankingr*sentence_transformers.sparse_encoder.modelrrModulerrNrYrrrws""""""$$$$$$ EDDDDD    b#b#b#b#b#BIb#b#b#Jcccccbicccccr