1j 3ddlmZddlZddlZddlmZddlmZddlZ ddl Z ddl Z erddl m Z mZmZmZmZGddZ dddZddZd dZdS)!) annotationsN)deque) TYPE_CHECKING) FunctionProto ModelProto NodeProto TensorProtoValueInfoProtoc~eZdZd'dZed(dZed)d Zd*d Zd+dZd,dZ d-dZ d.dZ d/d$Z d0d%Z d&S)1 ExtractormodelrreturnNonec||_|jj|_||jj|_||jj|_|j||jj|j||jj | |j|_ dSN) r graph_build_name2obj_dict initializer initializers value_info value_infosupdateinputoutput_build_output_dictoutmap)selfr s ;C:\Projects\Historical\venv\Lib\site-packages\onnx/utils.py__init__zExtractor.__init__s Z% 484M4M J "5 5 7;6O6O J !7 7   9 9$*:J K KLLL  9 9$*:K L LMMM&*&=&=dj&I&I dictcd|DS)Nci|] }|j| S)name).0objs r z2Extractor._build_name2obj_dict..*s...##...r r$)objss rrzExtractor._build_name2obj_dict(s......r dict[str, int]cvi}t|jD]!\}}|jD]}|dkr ||vsJ|||<"|S)N) enumeratenoder)routput_to_indexindexr. output_names rrzExtractor._build_output_dict,sn*,$UZ00 5 5KE4#{ 5 5 "$$"/9999/4 ,,  5 r io_names_to_extract list[str]list[ValueInfoProto]cfd|D}|r%tdd|fd|DS)Nc&g|] }|jv |Sr$rr&r%rs r z-Extractor._collect_new_io..9s-   D@P4P4PD4P4P4Pr z3The following names were not found in value_infos: z, c*g|]}j|Sr$r7r8s rr9z-Extractor._collect_new_io..@s!GGG4 &GGGr ) ValueErrorjoin)rr2 missing_namess` r_collect_new_iozExtractor._collect_new_io7s}    0     `diiP]F^F^`` HGGG3FGGGGr node_output_namestrgraph_input_namesset[str] reachableset[int]c|g}|rp|}||vr||jvrJ|j|}||vr9|||d|jj|jDz }|ndSdS)a?Helper function to find nodes which are connected to an output Arguments: node_output_name (str): The name of the output graph_input_names (set of string): The names of all inputs of the graph reachable (set of int): The set of indexes to reachable nodes in `nodes` cg|] }|dk| S)r,r$)r& input_names rr9z9Extractor._dfs_search_reachable_nodes..[s-&%++#+++r N)popraddrr.r)rr?rArCstackcurrent_output_namer0s r_dfs_search_reachable_nodesz%Extractor._dfs_search_reachable_nodesBs"" "'))++ "&777"dk11 $78 ))MM%(((*.*/%*@*FE     r input_names output_nameslist[NodeProto]ct|}t}|D]}|||fdt|DS)Nc4g|]}jj|Sr$)rr.)r&r0rs rr9z6Extractor._collect_reachable_nodes..ks"FFF5 &FFFr )setrLsorted)rrMrN _input_namesrCr%s` r_collect_reachable_nodesz"Extractor._collect_reachable_nodesasi ;'' !ee   L LD  , ,T< K K K KFFFFF94E4EFFFFr nodeslist[FunctionProto]c\i}|jjD]}|||j|jf<g}t |}|rv|}|j|jf|vrP||j|jf}||| |j |v|Sr) r functionsr%domainrpopleftop_typerHappendextendr.)rrV function_mapfunctionreferred_local_functionsqueuer.s r!_collect_referred_local_functionsz+Extractor._collect_referred_local_functionsms>@  , F FH=EL(-9 : :8: e  ,==??D dk*l::'++T\4;,GHH(//999 X]+++  ,('r .tuple[list[TensorProto], list[ValueInfoProto]]ct|D]6}|j|j7fdjD}fdjD}t jj}|dkrtd|dt jj }|dkrtd|d||fS)Nc2g|]}|vj|Sr$)rr&tall_tensors_namesrs rr9z8Extractor._collect_reachable_tensors..s3   %&qDU?U?UD a ?U?U?Ur c2g|]}|vj|Sr$r7rgs rr9z8Extractor._collect_reachable_tensors..s3   $%QBS=S=SD Q =S=S=Sr rzlen_sparse_initializer is z, it must be 0.zlen_quantization_annotation is ) rRrrrrrlenrsparse_initializerr;quantization_annotation)rrVr.rrlen_sparse_initializerlen_quantization_annotationris` @r_collect_reachable_tensorsz$Extractor._collect_reachable_tensorss='*ee 2 2D  $ $TZ 0 0 0  $ $T[ 1 1 1 1     *.*;        )-)9   "%TZ%B!C!C !Q & &T-CTTT '*$**L&M&M# &! + +^2M^^^ J&&r inputsoutputsrlist[TensorProto]rlocal_functionscd|jjzdz}tj||||||}|jj|jjd|d} tjj|fi| S)NzExtracted from {})rrzonnx.utils.extract_model) ir_version opset_imports producer_namerY) rr%onnxhelper make_graphr rw opset_import make_model) rrVrqrrrrrtr%rmetas r _make_modelzExtractor._make_models"DJO3c9 && 4kj'  */!Z47(    {%e44t444r c||}||}|||}||\}}||}|||||||Sr)r>rUrprcr) rrMrNrqrrrVrrrts r extract_modelzExtractor.extract_models %%k22&&|44--k<HH"&"A"A%"H"H Z@@GG 67K_   r N)r rrr)rr!)rr*)r2r3rr4)r?r@rArBrCrDrr)rMr3rNr3rrO)rVrOrrW)rVrOrrd)rVrOrqr4rrr4rrsrr4rtrWrr)rMr3rNr3rr)__name__ __module__ __qualname__r staticmethodrrr>rLrUrcrprrr$r rr r s J J J J///\/\ H H H H> G G G G((((.''''45555*       r r T input_pathstr | os.PathLike output_pathrMr3rN check_modelbool infer_shapesrrcPtj|std||std|std|stdt |t t |krtdt |t t |krtd|rt j||rgtj |t jj kr5t j ||t j |}n|rjt j |d}t j |}tj|}t j||nt j |}t#|}|||} | t jj kr;tj|d z} t j| |d | nt j| ||r!t j|d Sd S) a Extracts sub-model from an ONNX model. The sub-model is defined by the names of the input and output tensors *exactly*. Note: For control-flow operators, e.g. If and Loop, the *boundary of sub-model*, which is defined by the input and output tensors, should not *cut through* the subgraph that is connected to the *main graph* as attributes of these operators. Note: When the extracted model size is larger than 2GB, the extra data will be saved in "output_path.data". Arguments: input_path (str | os.PathLike): The path to original ONNX model. output_path (str | os.PathLike): The path to save the extracted ONNX model. input_names (list of string): The names of the input tensors that to be extracted. output_names (list of string): The names of the output tensors that to be extracted. check_model (bool): Whether to run model checker on the original model and the extracted model. infer_shapes (bool): Whether to infer the shapes of the original model. zInvalid input model path: z%Output model path shall not be empty!z&Input tensor names shall not be empty!z'Output tensor names shall not be empty!z0Duplicate names found in the input tensor names.z1Duplicate names found in the output tensor names.F)load_external_dataz.dataT)save_as_external_datalocationN)ospathexistsr;rkrRrzcheckerrgetsizeMAXIMUM_PROTOBUFshape_inferenceinfer_shapes_pathloadrdirnameload_external_data_for_modelr rByteSizebasenamesave) rrrMrNrrr base_dire extractedrs rrrsg4 7>>* % %DBjBBCCC B@AAA CABBB DBCCC ;3s;//0000KLLL <CL 1 12222LMMM-   ,,, & 33dl6SSS ..z;GGG +&& & *???$11%887??:.. )%:::: *%%%A \::Idl;;;7##K007: )[xXXXXX )[))).   -----..r tartarfile.TarFilebaselist[tarfile.TarInfo]cg}tj|}|D]}tj||j}tj|} tj||g|k}n#t $rd}YnwxYw|std|d|s| rtd|d| ||S)zCheck that the content of ``tar`` will be extracted safely Args: tar: The tarball file base: The directory where the tarball will be extracted Returns: list of tarball members FzThe tarball member z^ in downloading model contains directory traversal sequence which may contain harmful payload.zP in downloading model contains symbolic links which may contain harmful payload.) rrabspathr<r% commonpathr; RuntimeErrorissymislnkr])rrresultabs_basemember member_path abs_memberis_within_bases r_tar_members_filterrs<Fwt$$Hgll455 W__[11  #W//:0FGG8SNN # # #"NNN # SkSSS  <<>> V\\^^ EkEEE   f Ms+%B B B model_tar_pathlocal_model_with_data_dir_pathctj|5}ttdr||dn%||t ||ddddS#1swxYwYdS)aQSafely extracts a tar file to a specified directory. This function ensures that the extraction process mitigates against directory traversal vulnerabilities by validating or sanitizing paths within the tar file. It also provides compatibility for different versions of the tarfile module by checking for the availability of certain attributes or methods before invoking them. Args: model_tar_path: The path to the tar file to be extracted. local_model_with_data_dir_path: The directory path where the tar file contents will be extracted to. data_filterdata)rfilter)rmembersN)tarfileopenhasattr extractallr)rrmodel_with_data_zippeds r_extract_model_safer(s n % % )? 7M * *  " - -3F .     # - -3+*,J .                     sAA55A9<A9)TT)rrrrrMr3rNr3rrrrrr)rrrrrr)rrrrrr) __future__rrr collectionsrtypingr onnx.checkerrz onnx.helperonnx.shape_inference onnx.onnx_pbrrrr r r rrrr$r rrsL#"""""  f f f f f f f f \ @.@.@.@.@.F    Fr