1jjEddlmZddlZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Zddlmcm ZddlmZmZmZmZmZe rddlmZmZdKdZehdZeeZdZ dZ!dZ"GddZ#dLdZ$dMdZ%dNdZ&dOd"Z' dPdQd)Z( dRdSd0Z)dTd1Z*dUd3Z+dVd6Z,dWd;Z-dXd>Z.dVd?Z/dYdBZ0dVdCZ1dZdEZ2d[dFZ3d\dHZ4d]dJZ5dS)^) annotationsN)chain)IO TYPE_CHECKING)AttributeProto FunctionProto GraphProto ModelProto TensorProto)CallableIterablebase_dirstrlocation tensor_name read_onlyboolreturnintc0tj||||S)zlengthoffsetbasepathchecksumr diceZdZddZdS)ExternalDataInfotensorr rNonecd|_d|_d|_d|_d|_t }d}|jD]}|jtvrt||j|j ,|dz }t|tkrK|jdt}t|jtkr|dz }|||rht|}|t|z }t!|}|dkr |d|dz }t#jd|d|jd t(d |jCt+|j|_|jdkrt-d |jd|j|jCt+|j|_|jdkr!t-d |jd|jdSdS)Nrz...z and z morez&Ignoring unknown external data key(s) z for tensor z. Allowed keys: ) stacklevelz/External data offset must be non-negative, got z/External data length must be non-negative, got )rrrrrset external_datakey_ALLOWED_EXTERNAL_DATA_KEYSsetattrvaluelen_MAX_UNKNOWN_KEYS_IN_WARNING_MAX_KEY_DISPLAY_LENGTHaddsortedreprwarningswarnname_SORTED_ALLOWED_KEYSr ValueError) selfr$ unknown_keysunknown_key_countentry truncatedshownextrakey_lists r__init__zExternalDataInfo.__init__=s-     !$ ) 0 0Ey777ei5555!Q&!|$$'CCC % *B+B*B CI59~~(???!U*  $$Y///  <((E%E 2EE{{Hqyy0E0000 M888$k88!588      ; "dk**DK{Q 2dk22"(+22 ; "dk**DK{Q 2dk22"(+22 # "rN)r$r rr%)__name__ __module__ __qualname__rDrrr#r#<s(000000rr# data_file IO[bytes]infobytesc tj|j}|jB|j|krt d|jd|d|||j|jZ|j|jnd}||z }|j|kr t d|jd|d|d|||jS|S) a Validate offset/length against actual file size and read data. Layer 3 defense-in-depth (CWE-400): prevents memory exhaustion even if the model was crafted via direct protobuf APIs that bypass Python parsing. Returns the raw bytes read from the file. NExternal data offset (z) exceeds file size ( ) for tensor rzExternal data length (z) exceeds available data (z bytes from offset ) osfstatfilenost_sizerr;seekrread)rIrKr file_size read_start availables r#_validate_external_data_file_boundsrYps/))++,,4I { ; " "<<<<<,7<<  t{### {$(K$;T[[  * ; " ".....2<..)..  ~~dk*** >>  rr$r ct|}t||j|jd}t j|d5}t |||jcdddS#1swxYwYdS)aReads and returns the raw bytes stored in the external file for tensor. Unlike load_external_data_for_tensor, this does not mutate tensor: the tensor's data_location and external_data fields are left untouched, so the tensor still faithfully describes where its data lives on disk. Arguments: tensor: a TensorProto object. base_dir: directory that contains the external data. Returns: The raw bytes read from the external data file. TrbN)r#rrr9rPfdopenrY)r$rrKfdrIs r_read_external_data_bytesr^s F # #D $-d K KB 2t  Q 29dFKPPQQQQQQQQQQQQQQQQQQsA$$A(+A(r%cft|||_tj|_|jdd=dS)aLoads data from an external file for tensor and converts it to an in-memory tensor. Ideally TensorProto should not hold any raw data but if it does it will be ignored. After this call, tensor.raw_data holds the loaded bytes and tensor no longer describes external data: data_location is reset to DEFAULT and external_data is cleared. Arguments: tensor: a TensorProto object. base_dir: directory that contains the external data. N)r^raw_datar DEFAULT data_locationr,)r$rs rload_external_data_for_tensorrcs50AAFO&.FQQQrmodelr cht|D]!}t|rt||"dS)zLoads external tensors into model Arguments: model: ModelProto to load external data to base_dir: directory that contains external data N)_get_all_tensorsuses_external_datarc)rdrr$s rload_external_data_for_modelrhsF#5))<< f % % < )&( ; ; ;<= the size_threshold it will be converted to external data. To convert every tensor with raw data to external data set size_threshold=0. convert_attribute (bool): If true, convert all tensors to external data If false, convert only non-attribute tensors to external data Raise: ValueError: If location is not a relative path. FileExistsError: If a file already exists in location. z.datazDlocation must be a relative path that is relative to the model path.zExternal data file exists in .r`N)_get_initializer_tensorsrfruuiduuid1rPpathisabsr;existsFileExistsErrorrlsys getsizeofr`rqr9_is_valid_filename) rdrsrrtrutensors file_namer$tensor_locations rconvert_model_to_external_datars4'u--G*"5)); %%/  !w}}X&&  Zw~~h'' S%&Qh&Q&Q&QRRR I 5 5F ++ 5M&/22nDD!&)444  5 5 ; ;F ++ ;M&/22nDD"(+)/::8&)$*,,&7&7O!&/::: ; ;rct|D]P}t|r?|dstd|jdd=t j|_QdS)zCall to set all tensors which use external data as embedded data. save_model saves all the tensors data as embedded data after calling this function. Arguments: model (ModelProto): Model to be converted. r`raw_data field doesn't exist.N)rfrgrlr;r,r rarb)rdr$s r convert_model_from_external_datarst#5))77 f % % 7??:.. B !@AAA$QQQ'#.#6F 77r base_pathc 2t|}|dstjdt ||j|jd}tj|d5}| dd|j | }|j |z }d|cxkr tks5ntjd|j d |jd |d |tzd |dkr| d |z| |j | }| |jt||j|| |z ddddS#1swxYwYdS)aWrites tensor data to an external file according to information in the `external_data` field. The function checks the external is a valid name and located in folder `base_path`. Arguments: tensor (TensorProto): Tensor object to be serialized base_path: System path of a folder where tensor data is to be stored Raises: ValueError: If the external file is invalid. r`rFzr+brr)NrNrOz( must be between the current file size (z) and rw)r#rl onnx_checkerValidationErrorrrr9rPr\rTrtell_MAX_EXTERNAL_DATA_PADDINGwriter`rq)r$rrKr]rIrVpaddingrs rsave_external_datar*s F # #D ??: & &L*+JKKK  4=&+u M MB 2u  Tq! ; " "((IkI-G====#====="2AT[AAv{AA>GAA #==AAA {{000 NN4; ' ' '!!(((&$-9I9IF9RSSS-TTTTTTTTTTTTTTTTTTs*DF  FFonnx_model_protoIterable[TensorProto]cVtt|t|S)z=Scan an ONNX model for all tensors and return as an iterator.)rrx_get_attribute_tensorsrs rrfrfTs,  !122/00  r attributerfunc-Callable[[GraphProto], Iterable[TensorProto]]c#K|jtjkr||jEd{V|jtjkr|jD]}||Ed{VdSdS)zICreate an iterator through processing ONNX model attributes with functor.N)typerGRAPHgGRAPHSgraphs)rrgraphs r_recursive_attribute_processorr\s~---4 $$$$$$$$$~...% # #EtE{{ " " " " " " " "/. # #rrr c#K|jEd{V|jD]'}|jD]}t|tEd{V(dS)z@Create an iterator of initializer tensors from ONNX model graph.N) initializernoderr#_get_initializer_tensors_from_graph)rrrs rrrgs           I5>         rc#>Kt|jEd{VdS)z:Create an iterator of initializer tensors from ONNX model.N)rrrs rrxrxqs023C3IJJJJJJJJJJJrgraph_or_functionGraphProto | FunctionProtoc#K|jD]R}|jD]H}|dr |jV|jEd{Vt |t Ed{VISdS)zSCreate an iterator of tensors from node attributes of an ONNX model graph/function.tN)rrrlrrr!_get_attribute_tensors_from_graph)rrrs rrrvs"&  I!!#&& "k!!! ( ( ( ( ( ( ( (5<          rc#|Kt|jEd{V|jD]}t|Ed{VdS)zDCreate an iterator of tensors from node attributes of an ONNX model.N)rr functions)rfunctions rrrsi01A1GHHHHHHHHH$.??4X>>>>>>>>>>??rfilenamecrtjd}||}t|S)z8Utility to check whether the provided filename is valid.z^[^<>:;,?"*|/]+$)recompilematchr)rexprs rrrs/ *' ( (C IIh  E ;;rcV|do|jtjkS)z?Returns true if the tensor stores data in an external location.rb)rlrbr rm)r$s rrgrgs+ (( 9  K$8 8r field_keyc`t|jD]\}}|j|kr|j|=dS)aRemoves a field from a Tensor's external_data key-value store. Modifies tensor object in place. Arguments: tensor (TensorProto): Tensor object from which value will be removed field_key (string): The key of the field to be removed N) enumerater,r-)r$rifields rremove_external_data_fieldrsEf233((5 9 ! !$Q'((rfilepathct|D]K}t|r:|dr%t|||dL|S)aSerializes data for all the tensors which have data location set to TensorProto.External. Note: This function also strips basepath information from all tensors' external_data fields. Arguments: model (ModelProto): Model object which is the source of tensors to serialize. filepath: System path to the directory which should be treated as base path for external data. Returns: ModelProto: The modified model object. r`)rfrgrlr ClearField)rdrr$s rwrite_external_data_tensorsrsj#5))** f % % *&//**E*E * vx 0 0 0   j ) ) ) Lr) rrrrrrrrrr)rIrJrKr#rrrrL)r$r rrrrL)r$r rrrr%)rdr rrrr%)NNNN)r$r rrrrirrirrjrrjrr%)TNrrF) rdr rsrrrjrtrrurrr%)rdr rr%)r$r rrrr%)rr rr)rrrrrr)rr rr)rrrr)rrrr)r$r rr)r$r rrrr%)rdr rrrr )6 __future__rrPrrryr7 itertoolsrtypingrr onnx.checkercheckerronnx.onnx_cpp2py_export.checkeronnx_cpp2py_exportr onnx.onnx_pbrrr r r collections.abcr r r frozensetr.r5r:r2r3rr#rYr^rcrhrqrrrrfrrrxrrrrgrrrHrrrs1#""""" $$$$$$$$######333333333322222222UUUU*(i<<<v9::!'11111111h    FQQQQ(    " < < < < !!!!!<%)# 7;7;7;7;7;t 7 7 7 7 'T'T'T'TT####KKKK    ???? ( ( ( (r