*j~$UddlmZddlmZddlmZmZddlmZddl Z ddl m Z e j eZdZded <eGd d ZdS) ) annotations)Callable) dataclassfield)AnyN)logging)labellabelsscorescoresztuple[str, ...]DEFAULT_LABEL_COLUMNSc eZdZUdZded<edZded<eed Zd ed <eed Z d ed <eddZ ded<ee ddZ ded<d-dZ d.dZd/d Zd0d#Zd1d&Zd2d)Zd3d,ZdS)4BaseDataCollatoraBase data collator for Sentence Transformers models. Preprocesses text columns via ``preprocess_fn`` (typically ``model.preprocess``), producing ``{column}_input_ids``, ``{column}_attention_mask``, etc. Handles prompt resolution (per-column or per-dataset) and Router task mapping. It is important that the columns are in the expected order. For example, if your dataset has columns "answer", "question" in that order, then the MultipleNegativesRankingLoss will consider "answer" as the anchor and "question" as the positive, and it will (unexpectedly) optimize for "given the answer, what is the question?". r preprocess_fnc*ttS)N)listr VC:\Projects\DSCons\.venv\Lib\site-packages\sentence_transformers\base\data_collator.pyzBaseDataCollator. s4H]C^C^r)default_factory list[str]valid_label_columnsF)rreprz1dict[str, str] | dict[str, dict[str, str]] | Nonerouter_mappingz7str | dict[str, str] | dict[str, dict[str, str]] | NonepromptsN)defaultrzint | dict[str, int] | None max_length)rinitrzset[tuple[str, ...]]_warned_columnsbatchdict[str, Any]returndict[str, str]c|j}|rtt|tr_ttt |trd|vr|d|vr||dSiS|piS)zRResolve the router mapping for this batch, handling nested (per-dataset) mappings. dataset_name)r isinstancedictnextitervalues)selfr!rs r_resolve_router_mappingz(BaseDataCollator._resolve_router_mapping's,  >400 4^%:%:%<%< = =>>EE  &&5+@N+R+R%eN&;<<I##rstr | dict[str, str] | Nonecn|j}t|tr|s|Sd|v}|r|d|vr||dSttt |tr5|s1t dt|iS|S)zKResolve the prompts for this batch, handling nested (per-dataset) mappings.r&a=The prompts provided to the trainer are a nested dictionary. In this setting, the first level of the dictionary should map to dataset names and the second level to column names. However, as the provided dataset is a not a DatasetDict, no dataset names can be inferred. The keys to the provided prompts dictionary are ) rr'r(r)r*r+ ValueErrorrkeys)r,r!ris_multi_datasets r_resolve_promptsz!BaseDataCollator._resolve_prompts4s,'4((  N)U2  2n 5 @ @501 1 d4 0 01122D 9 9 #  `HLGLLNNG[G[`` Irstr | dict[str, str] column_namestr str | Nonecvt|tr|St|tr ||vr||SdS)z,Get the prompt string for a specific column.N)r'r6r()r,rr5s r_get_prompt_for_columnz'BaseDataCollator._get_prompt_for_columnJsE gs # # (N  & & (;'+A+A;' 'trcolumn_positionintc,||S)zWResolve the task a column is preprocessed with. Subclasses may add positional defaults.)get)r,r5r:rs r_get_task_for_columnz%BaseDataCollator._get_task_for_columnRs!!+...rtask int | Nonect|jtr|jSt|jtr||j|SdS)zwResolve the training max-length override for a column: an int applies to every column, a dict is keyed by task.N)r'rr;r(r=)r,r?s r_get_max_length_for_taskz)BaseDataCollator._get_max_length_for_taskXsT dos + + #? " dot , , -1A?&&t,, ,trfeatureslist[dict[str, Any]]c|siSt|d}i}d|vr&|d|dd|d<t||jvr|||jD]@|vr:tjfd|D|d<|nA| |}| |}d|D}|rt d|dt|D]\}|||}||} fd|D} ||} | || | || } n|| | | } | D]\} }||d | <|||d <|S) Nrr&c g|] }| Srr).0row label_columns r z-BaseDataCollator.__call__..ts.U.U.USs.{s.ggg3#++NeAfAf+3+++rz Column(s) z look like ID columns, but they will be tokenized and trained on as text. If they hold IDs, resolve them to their values first, e.g. with `sentence_transformers.util.resolve_ids`, or remove them from the dataset.c g|] }| Srr)rGrHr5s rrJz-BaseDataCollator.__call__..s;;;3c+&;;;r)promptr?r)rSr?__task)rr1removetupler maybe_warn_about_column_orderrtorchtensorr-r3logger warning_once enumerater>r9rBritems)r,rC column_namesr!rrid_liker:r?rSinputsr preprocessedkeyvaluer5rIs @@r__call__zBaseDataCollator.__call__as{ IHQK,,..//  \ ) )    / / /$,QK$?E. !   d&: : :  . .| < < <!4  L|++!&.U.U.U.UH.U.U.U!V!Vg##L111, 55e<<''..gg,ggg     ]W]]]    -6l,C,C 4 4 (O[,,[/>ZZD00+FFF;;;;(;;;F66t<rBrerXrrrrrse  %*U;^;^%_%_%_____HM^binHoHoHoNooooGLu]ahmGnGnGnGnnnn.3eDu.M.M.MJMMMM,1E#EX],^,^,^O^^^^ $ $ $ $,//// 3333j%6%6%6%6%6%6rr) __future__rcollections.abcr dataclassesrrtypingrrYtransformers.utilsrtransformers_logging get_loggerr{r[r rrrrrrs"""""""$$$$$$(((((((( >>>>>> )  ( 2 2*POOOO i6i6i6i6i6i6i6 i6i6i6r