+jmX UddlmZddlZddlmZmZddlmZddlm Z m Z m Z ddl Z ddlmZe rddlmZejeZdZd ed <d:dZd;dZddZd?dZGddZd@d$ZdAd*ZdBd3Z edd4fdCd9Z!dS)D) annotationsN)CallableMapping)partial) TYPE_CHECKINGAnyLiteral)logging)Dataset)labellabelsscorescorestuple[str, ...]DEFAULT_LABEL_COLUMNSdatasetr input_colstrreturnc d|jD}t|dkr-td|d|pddt|jd|dS) NcHg|]}|dks|d| S)id_id)endswith).0cols PC:\Projects\DSCons\.venv\Lib\site-packages\sentence_transformers\util\dataset.py z!_infer_id_col..s.\\\# UZH[H[#z&Cannot infer the ID column of lookups[zD]: expected exactly one column named 'id' or ending in '_id', found nonez among z+. Pass an explicit (dataset, id_col) tuple.r) column_nameslen ValueErrorsorted)rr candidatess r _infer_id_colr's\\!5\\\J :! _Y _ _PZPd^d _ _G011 _ _ _   a=rid_colc fd|jD}t|dkrtd|dd|pdd|dS) Nc g|] }|k| Sr+)rrr(s rrz$_infer_value_col..!sFFFv  rr z)Cannot infer the value column of lookups[z']: expected exactly one column besides z, found r!z6. Pass an explicit (dataset, id_col, value_col) tuple.r)r"r#r$)rr(r remainings ` r_infer_value_colr- sFFFF 4FFFI 9~~ D  D D39 D DENEXRX D D D   Q<rFpluralboolc|rdddddndddd}|D]4\}}||r|||zcS5|S)Niess)y_ids_idsr_idx)r5rr6)itemsr removesuffix)rr.suffixessuffix replacements r _output_colr<+sGMvBCCC\^gisuSvSvH'~~//@@   f % % @))&11K? ? ? ? @ rvaluerczt|to&|dS)N[) isinstancerlstrip startswith)r=s r_is_string_encoded_listrC4s- eS ! ! Dellnn&?&?&D&DDrcolumnlistcol_namec g}|D]} tj|}n:#ttf$r&}td|d|d|dddd}~wwxYwt |t t fs2td|dt|jd|dd| t ||S)NzColumn zF looks like it holds string-encoded lists, but a row failed to parse (z): dz? looks like it holds string-encoded lists, but a row parsed to z instead of a list: ) ast literal_evalr$ SyntaxErrorr@rEtupletype__name__append)rDrFparsedrowr=es r_parse_string_encoded_listsrS8s* F # # $S))EEK(   .(.... #DSD ..   %$// K(KK;;'KK=@#YKK   d5kk"""" MsA!AAc.eZdZdZdZd dZdd Zdd Zd S)_SortedIdIndexa``id -> row position`` lookup backed by sorted numpy arrays. A python dict retains a hash table plus every key object, and rebuilds that table in each DataLoader worker on spawn-start platforms. Sorted arrays hold the same lookup in a fraction of the memory and pickle as flat buffers, so they also unpickle far faster. Lookups are a searchsorted rather than a hash, so they take 6x to 16x as long per batch as the dict, which is immaterial next to the forward pass. Only used for homogeneous numeric ids and for strings narrow enough that the fixed-width array stays under the dict's footprint, since numpy sizes such an array by its longest entry. )keys positionsrV np.ndarrayrNonec"tj|d}||}tjt|t}t|dkr|dd|ddk|dd<|||_|||_dS)Nstable)kind)dtyper )npargsortonesr#r/rVrW)selfrVorder sorted_keyskeeps r__init__z_SortedIdIndex.__init__Zs 4h///5k ws;''t444 {  a  #ABB;ss+;;D"I% trintc*t|jSN)r#rV)rbs r__len__z_SortedIdIndex.__len__es49~~ridsct|jdkr |d}ntj|j|}tj|t|jdz }|j||k}|r |j|S|ttj|}t|j j dvr| nt|)zQReturn the row positions for ``ids``, raising ``KeyError(missing_id)`` on a miss.rr iuf)r#rVr_ searchsortedminimumallrWrgargminKeyErrorr]r\itemr)rbrkmissinginsertion_pointsclippedfounds rlookupz_SortedIdIndex.lookuphs ty>>Q  !fGG!ty#>> j!13ty>>A3EFFGIg&#-Eyy{{ /~g..#bi..//0G5)@)@w||~~~c'llSSSrN)rVrXrrY)rrg)rkrXrrX)rN __module__ __qualname____doc__ __slots__rfrjrxr+rrrUrULsg  &I % % % % T T T T T TrrUrkindex_SortedIdIndex | dict value_colc |sgS ttrtj|}|jjdks|jjjjjkrt|d| }nfd|D}n#t$rtd|ddt$rk}|j d}d} t|tr)| drd} td |d |d | dd}~wwxYw|||S) NOrc g|] }| Sr+r+)rkeyr}s rrz_take..s...3E#J...rzIDs from input column z are not hashable, which typically means the train dataset carries a format (e.g. with_format('numpy')). Call dataset.with_format(None) before applying the transform.r3r?zS The ID looks like a string-encoded list. Convert the column to native lists first.zID z from input column z% was not found in its lookup dataset.)r@rUr_asarrayr]r\rVrrrxtolist TypeErrorargsrrArB) rk value_viewr}rrqueriesrowsrRrthints ` r_takervs   e^ , , /jooG}!S((GM,>%*BRBW,W,Ws1v&&&<<((//11DD....#...D  -Y - - -     &) gs # # i(8(8(C(CC(H(H ihD h' h h h hbf h h     d I &&sB#B++'D=A&D88D=resolveddict[str, Any]namevaluesrYc>||vrtd|d|||<dS)NzOutput column z is produced more than once (e.g. two lookups entries resolving to the same name, or an expanded list column colliding with another entry). Rename the input columns to disambiguate.)r$)rrrs r_add_output_columnrsD x -T - - -   HTNNNrbatch resolutions;dict[str, tuple[Any, _SortedIdIndex | dict, str, str, str]] keep_columnsmax_list_length int | None output_formatLiteral['columns', 'lists']c t|ts%tdt|jdi}|D]Q\}\}}} } } ||} t| t s(td|dt| jdt| dkr%t| drt| |} t| dko!t| dt tf} | r|dkr҉fd| D} d | D}t|d kr#td |d t|d | }|dkrtd |dtd| D||| |}t|D]$}t!|| d|d z||d|%| rfd| D} td| D||| |}gd}}| D]A}||||t|z|t|z }Bt!|| |"t!|| tt | ||| |S|D]}||vr||}t|dkr%t|drt||}t|dko8t|dt$t&f ot)|dd}|rfd|D}t!||||S)Nz7resolve_ids expects batches as plain column dicts, got z: the train dataset carries a format (e.g. with_format('pandas')). Call dataset.with_format(None) before applying the transform.z Input column z holds a z rather than a list, which means the train dataset carries a format (e.g. with_format('numpy')). Call dataset.with_format(None) before applying the transform.rcolumnsc$g|] }|d Srir+rrQrs rrz&_resolve_ids_batch..$BBBC#../BBBrc,h|]}t|Sr+)r#)rrQs r z%_resolve_ids_batch..s222Cs3xx222rr zRows of input column z! hold lists of differing lengths a . List columns expand into one output column per position, which requires the same length on every row. Set max_list_length to cap all rows to a common length, or output_format="lists" to keep nested lists (the CrossEncoder listwise shape, which allows ragged rows).zl hold empty ID lists, so there are no positions to expand into columns. Filter such rows out of the dataset.cg|] }|D]}| Sr+r+rrQks rrz&_resolve_ids_batch..% B B Bsc B B B B B Br_c$g|] }|d Srir+rs rrz&_resolve_ids_batch..rrcg|] }|D]}| Sr+r+rs rrz&_resolve_ids_batch..rrrjc$g|] }|d Srir+rs rrz&_resolve_ids_batch..s$>>>c*?*+>>>r)r@rrrMrNr7rEr#rCrSrLr$r%poprrangerrOrbyteshasattr)rrrrrrrrr}r stripped_name plural_namerDis_listlengths num_positions flat_valuespositionnestedoffsetrQkeep_colris_row_sequences ` r_resolve_ids_batchrsI eW % %  -d5kkFZ - - -   "HQ\QbQbQdQd,v,vM MJy-y!&$'' K KKd6ll6KKKK  v;;??6vayAA?0CCFf++/JjT5M&J&J  v} 11*BBBB6BBB226222G7||q   SISSgSSS$KKMMM!! ]I]]] B Bf B B BJPUW`bkllK!-00 v v"8 -N-N1 -N-NP[\d\sfs\sPtuuuu v v*BBBB6BBB B Bf B B BJPUW`bkllKFF # # k&6CHH3D*DEFFF#c((" xf = = = = xd6llJX]_hjs8t8t u u u u! 7 7 5 x v;;??6vayAA?0BBFf++/w*VAYe 2U2U.UwZabhijbkmvZwZw  &? &>>>>v>>>F8Xv6666 OrrlookupsCdict[str, Dataset | tuple[Dataset, str] | tuple[Dataset, str, str]]list[str] | tuple[str, ...]*Callable[[dict[str, Any]], dict[str, Any]]c b ddlm}m}n"#t$r}td|d}~wwxYwt |t rt d|d||dkrtd|d|d vrtd |di}i}|D]\} } t | trqt| } t| d vr#td | d t| d| d} | d} t| dkr| dnd}n| dd}} } t | |rt d | dt | |s(t d | dt| j dt| }| d} | t!| | } n4| | jvr+td | d| dt%| jd|t'| | | }n4|| jvr+td | d|dt%| jd|| f}||vrg}| | gdD]}||| t/j|}|jdkr)|jjdvr|jdkrt;|}ndt=|D}t|| jkr9t@!d | d| d| jt|z d|||<| |g|||tE| tE| df|| <tGtH|t||| S)!aBuild a batched transform that resolves ID columns to their values via a join against lookup datasets. Useful for IR datasets that store query / document IDs alongside separate text (or image) datasets, e.g. `lightonai/ms-marco-en-bge `_. An input column may hold one ID per row (e.g. ``positive_id: str``) or a list of IDs per row (e.g. ``document_ids: list[str]``), detected from the row values. An ID that is missing from its lookup dataset raises a :class:`KeyError` when the batch is read. Pass the returned callable to :meth:`~datasets.Dataset.set_transform` (lazy, no caching) or :meth:`~datasets.Dataset.map` with ``batched=True`` (eager, cached). With ``map``, also pass ``remove_columns=list(lookups)`` so the raw ID columns are dropped from the result: unlike ``set_transform``, ``map`` merges the transform output with the untouched input columns. ``map`` also hands the transform batches in the dataset's current format, so remove any format first with ``with_format(None)``. Args: lookups: ``{input_col: lookup}`` for every input column to resolve, where ``lookup`` is: - a :class:`datasets.Dataset`: the ID column (the single column named ``id`` or ending in ``_id``) and the value column (the single remaining column) are inferred. Ambiguity raises at build time. - ``(dataset, id_col)``: explicit ID column, value column inferred. - ``(dataset, id_col, value_col)``: fully explicit. All other input columns are dropped (except ``keep_columns``), as the losses read batch columns positionally. Insertion order sets the output column order, so list the anchor (query) column first. keep_columns: Input columns to pass through unresolved. Defaults to the label columns the data collators recognize by default (``label``, ``labels``, ``score``, ``scores``). Mirror your collator's ``valid_label_columns`` if you customized it. Absent columns are ignored. max_list_length: Truncate every list-per-row column (ID lists and list-valued kept columns) to the first ``max_list_length`` entries. Must be a positive integer or ``None`` (default, keep all). Note that this takes the first N as stored, not top-N by teacher score. output_format: ``"columns"`` (default) expands each list column into numbered columns (``document_1``, ..., ``document_N``), requiring a uniform list length per row. ``"lists"`` keeps list columns nested under a plural name (``documents: list``), allowing ragged rows. Use ``"lists"`` for CrossEncoder listwise training. Returns: A picklable batched transform mapping a batch dict of ID columns to a batch dict of resolved columns, for :meth:`~datasets.Dataset.set_transform` or :meth:`~datasets.Dataset.map` with ``batched=True``. Example (KD, LightOn ms-marco-en-bge):: from datasets import load_dataset from sentence_transformers.util import resolve_ids train = load_dataset("lightonai/ms-marco-en-bge", "train", split="train") queries = load_dataset("lightonai/ms-marco-en-bge", "queries", split="train") documents = load_dataset("lightonai/ms-marco-en-bge", "documents", split="train") train.set_transform(resolve_ids({ "query_id": queries, "document_ids": documents, }, max_list_length=32)) # -> rows of {"query": str, "document_1": str, ..., "document_32": str, "scores": list[float]} Example (triplet with IDs):: train.set_transform(resolve_ids({ "query_id": queries, "positive_id": documents, "negative_id": documents, })) # -> rows of {"query": str, "positive": str, "negative": str} Example (CrossEncoder listwise, nested lists):: train.set_transform(resolve_ids({ "query_id": queries, "document_ids": documents, }, output_format="lists")) # -> rows of {"query": str, "documents": list[str], "scores": list[float]} Example (streaming):: from datasets import Features, Sequence, Value # Only the train split streams. The lookup datasets are random-access joins, so they # must stay regular (materialized) Datasets. train = load_dataset("lightonai/ms-marco-en-bge", "train", split="train", streaming=True) train = train.map( resolve_ids({"query_id": queries, "document_ids": documents}, max_list_length=32), batched=True, remove_columns=["query_id", "document_ids"], features=Features({ "query": Value("string"), **{f"document_{i}": Value("string") for i in range(1, 33)}, "scores": Sequence(Value("float32")), }), ) # Streaming map cannot infer the output schema, and the trainers require it, so pass # ``features=`` explicitly. r)r DatasetDictzTresolve_ids requires the `datasets` library. Install it with `pip install datasets`.NzEkeep_columns must be a list or tuple of column names, got the string .r z8max_list_length must be a positive integer or None, got )rlistsz0output_format must be "columns" or "lists", got )zlookups[z_] must be a Dataset, (dataset, id_col), or (dataset, id_col, value_col), got a tuple of length rrz] holds a `DatasetDict`, expected a `Dataset`. Pass a specific split (e.g. `dataset_dict['train']`), or load with `load_dataset(..., split='...')`.z] must hold a `Dataset`, got z ]: ID column z1 is not a column of the lookup dataset (columns: z).z]: value column iP) batch_sizeiufUSci|]\}}|| Sr+r+)rrr=s r zresolve_ids..sUUU_XuUUUrz] contains duplicate z values (zD rows shadowed). Each duplicated ID resolves to its last occurrence.T)r.)rrrr)%datasetsr r ImportErrorr@rrr$r7rLrEr#rMrNr with_formatr'r"r%r-select_columnsiterextendr_rndimr]r\itemsizerU enumeratenum_rowslogger warning_oncer<rr)rrrrr rrRrindicesr lookup_specspecrr(roriginal_dataset_id index_key id_valuesid_batchrVr}s r resolve_idsrsL111111111  b    ,$$sq`lqqqrrr"':':fTcfffggg000^M^^^___ PRK<>G")--//F F  ; k5 ) ) A $$D4yy&& XyXXKNt99XXX1gG!WF#&t99>>QtII)4dDYVG g{ + + g9ggg '7++ ljyjjQUV]Q^Q^Qgjjjkk k!kk%%d++ >"7I66FF 7/ / />9>>V>>#G$899>>>   (&)DDII g2 2 2F9FF FF%+G,@%A%AFFF  )&1 G # #!I#22F8<<AAVATT 3 3  &!12222:i((DyA~~$*/W"<"<RUAUAU&t,,VU )@T@TUUU5zzW---##'y''''(3u::5''' "'GI   " "I; / / I    " "  $ / / / " I <(('#    s *%*)rr rrrr)rr r(rrrrr)F)rrr.r/rr)r=rrr/)rDrErFrrrE) rkrEr}r~rrrrrrE)rrrrrrErrY) rrrrrrrrrrrr) rrrrrrrrrr)" __future__rrIcollections.abcrr functoolsrtypingrrr numpyr_transformers.utilsr transformers_loggingrr get_loggerrNrr__annotations__r'r-r<rCrSrUrrrrr+rrrs""""""" --------..........>>>>>>!       )  ( 2 2*POOOOEEEE('T'T'T'T'T'T'T'TT''''<NNNNf1F"&1: HHHHHHHr