ljA$dZddlmZddlZddlmZddlZd(d Zd)d Z d*dZ d+dZ d\Z Z d,dZd-dZd.dZd.dZee e e eeedZehdZd/dZejdZd0d!Zd1d#Zd2d&Zgd'ZdS)3zThis file contains the parsers used by chat response parsing. Each parser takes a chunk of captured text and parses it into a single key in the output message dictionary. Functions are generally boilerplate.) annotationsN)AnytextstrargsdictreturncZ|ddr|n|S)NstripT)getr rrs ]C:\Projects\DSCons\.venv\Lib\site-packages\transformers\utils\chat_parsing\content_parsers.py_textrs'88GT22 <4::<<<<intc<tt||SN)rrr s r_intrs uT4  ! !!rfloatc<tt||Sr)rrr s r_floatr!s tT"" # ##rboolcJt||dvS)N)true1)rlowerr s r_boolr%s# t   " " $ $ 55r)rc B |dg}|dd}|r!t|vs t|vrtd|}g |D]Y\}}t j|dzt j|z}d fd }t j|||tj }Z|rt jd d |}t D]=\} } | t| ttj | }> tj |S#tj $rd} |drt||cYd} ~ S||krtd|d| | td|d|d| | d} ~ wwxYw)aJSON parser with optional dialect knobs for LLM-emitted quirks. `args`: - `unquoted_keys` (bool): quote bare-identifier keys before parsing. - `string_delims` ([[open, close], ...]): strings delimited by these custom markers are pre-extracted, then restored as standard JSON strings. - `allow_non_json` (bool): return stripped text if parsing fails. string_delims unquoted_keysFzSjson: input contains reserved sentinel characters (\x01/\x02); cannot parse safely.z(.*?)mrr rc|dttdz tS)N)appendgroup _LAX_OPENlen _LAX_CLOSE)r#captureds r_capturez_json.._captureBs@ OOAGGAJJ ' ' '@X!2@J@@ @rflagsz(?<=[{,])(\w+):z"\1":allow_non_jsonNz5json parser could not parse region as JSON. Content: z Error: z:json: could not parse after dialect transforms. Original: z Transformed: )r#rr r)r r(r* ValueErrorreescapesubDOTALL enumeratereplacejsondumpsloadsJSONDecodeErrorr) rrr!r"workingopen_dclose_dpatternr,iser+s @r_jsonrB.s$HH_b11MHH_e44Mr)t++zT/A/ApqqqGH(FF)F##h.71C1CC A A A A A A&(G29EEE@&+Xw??(##PP1//Y"?"?:"?"?AOO z'"""   88$ % % %t$$ $ $ $ $ $ $ d??jVZjjghjjkkqr r H H H07 H HDE H H    s$D++F:$FF$5FFraw value_parser dict | Nonec|||St||dd|diS)Nnamerr) parse_contentr )rCrDs r _sub_parserI[sA l..vv>> @P@PQWY[@\@\ ] ]]rcR|d}|td|d}|dd}i}tj||tjD]}|}|d}|td |t |d d |} ||vrE|rCt||ts ||g||<|| | | ||<|S) zParse shallow XML-ish tags into a dict. `tag_pattern` regex must have named groups `key` and `value`. Optional `value_parser` recurses; `merge_duplicates` collects duplicate keys into a list. tag_patternNz1xml-inline: 'tag_pattern' content_arg is requiredrDmerge_duplicatesFr-keyz@xml-inline: tag_pattern must have a named group 'key'. Pattern: value) r r0r1finditerr4 groupdictrI isinstancelistr&) rrrKrDmergeoutr#groupsrMrNs r _xml_inlinerWas)((=))KLMMM88N++L HH' / /EC [d") < < <  jj ;m`kmmnn n6::gr22LAA #::%:c#h-- &H:C HOOE " " " "CHH Jrc|dd}|dd}|d}i}||D]e}t||}|r||vr||d\}}t||t||}}t||||<f|S)z7Parse line-delimited `keyvalue` pairs into a dict.line_sep kv_sep:rDr%)r splitrrI) rrrYr[rDrUlinekvs r _kv_linesra{sxx D))H XXh $ $F88N++LC 8$$--T4   vT)) zz&!$$1Q~~uQ~~1A|,,A Jr)rrrrr7z xml-inlinezkv-lines>rrrrrGc.t|||Sr)CONTENT_PARSERS)rrGrs rrHrHs 4 t , ,,rz \{(\w+)\} transformscopect|tr fd|DSt|trfd|DSt|ts|St |}|s|S|d}|vr"td|dt|S)ztRecursively walk a transform template, which is used to restructure parsed output into the actual shape we want.c8i|]\}}|t|S_apply_transform).0r_r`res r z$_apply_transform..s*LLL$!Q#Au--LLLrc0g|]}t|Srhri)rkr`res r z$_apply_transform..s$>>>q E**>>>rr%ztransform placeholder '{z}' is not defined. Available: ) rRritemsrSr _PLACEHOLDER fullmatchr'KeyErrorsorted)rdrewholerMs ` rrjrjs)T""MLLLL)//:K:KLLLL)T""?>>>>I>>>> i % %  " "9 - -E  ++a..C %f3ffW]^cWdWdffggg :rNonect|tr)|D]}t||dSt|tr|D]}t||dSt|t sdSt |r/t |st|d|ddSdS)a$Walk a transform template and reject any string that mixes a `{name}` placeholder with literal text. Only whole-string placeholders (e.g. `"{content}"`) and plain literals are supported. Called from the template loader so authors get a clear error at load time, not at parse time.Nz: transform string z mixes a {placeholder} with literal text. Use either a whole-string placeholder (e.g. "{content}") or a plain literal; string interpolation is not supported.) rRrvaluesvalidate_transform_stringsrSrrpsearchrqr0)rerdr`s rrxrxs )T""!!## 1 1A &ua 0 0 0 0)T"" 1 1A &ua 0 0 0 0 i % %9%% l.D.DY.O.O  5 5 5 5 5       rbodycapturesc t||j|j}|j|S|jrt |t s-td|jdt|j dg}|D]p}t |ts-td|jdt|j d| t|ji||q|St|ji|d|iS)a~Run `body` through the field's content parser, then optionally apply the transform template. When `transform_each` is set, the parsed content must be a list and the template is applied to each element (with the element's keys unpacked into the template scope, alongside any regex captures). `field` is a `spec.Field`; typed via duck-typing to avoid a cyclic import.NzField 'z@': transform_each requires the parsed content to be a list, got .z?': transform_each requires each list element to be a dict, got content)rHr~ content_argsrdtransform_eachrRrSr0rGtype__name__rr&rj)rzfieldr{rNrUitems r process_fieldrsD $ u/A B BE   %&& /%*//E{{+///  P PDdD))  2ej22::.222 JJ'9MH9M9MNN O O O O EO-K-K)U-K-K L LLr)rcSTREAMABLE_PARSERSrHrrx)rrrrr r)rrrrr r)rrrrr r)rrrrr r)rrrrr r)rCrrDrEr r)rrrrr r)rrrGrrrr r)rdrrerr r)rerrdrr ru)rzrr{rr r)__doc__ __future__rr7typingrregexr1rrrrr(r*rBrIrWrarc frozensetrrHcompilerprjrxr__all__rhrrrsjj#""""" ====""""$$$$6666 ' :****Z^^^^ 4$     Y???@@----rz,'' $    .MMMM8   r