o Ďi/@sLddlmZddlZddlZddlZddlZddlmZddlm Z ddl m m Z ddlmZmZmZmZmZe rCddlmZmZGdddZdLddZdMddZ    dNdOddZ    dPdQd&d'ZdRd(d)ZdSd+d,ZdTd/d0ZdUd5d6ZdVd9d:Z dTd;d<Z!dVd=d>Z"dTd?d@Z#dWdBdCZ$dXdDdEZ%dYdGdHZ&dZdJdKZ'dS)[) annotationsN)chain) TYPE_CHECKING)AttributeProto FunctionProto GraphProto ModelProto TensorProto)CallableIterablec@seZdZdddZdS) ExternalDataInfotensorr returnNonecCsfd|_d|_d|_d|_d|_|jD] }t||j|jq|jr&t |j|_|jr1t |j|_dSdS)N) locationoffsetlengthchecksumbasepath external_datasetattrkeyvalueint)selfr entryrU/home/jeff/fluffinator/venv/lib/python3.10/site-packages/onnx/external_data_helper.py__init__s  zExternalDataInfo.__init__N)r r rr)__name__ __module__ __qualname__rrrrrr sr r r base_dirstrrrcCst|}t||j|j}t|d*}|jr||j|jr'| |j|_ n | |_ WddSWddS1s?wYdS)zLoads data from an external file for tensor. Ideally TensorProto should not hold any raw data but if it does it will be ignored. Arguments: tensor: a TensorProto object. base_dir: directory that contains the external data. rbN) r c_checker_resolve_external_data_locationrnameopenrseekrreadraw_data)r r#infoexternal_data_file_path data_filerrrload_external_data_for_tensor,s    "r0modelrcCs8t|D]}t|rt||tj|_|jdd=qdS)zLoads external tensors into model Arguments: model: ModelProto to load external data to base_dir: directory that contains external data N)_get_all_tensorsuses_external_datar0r DEFAULT data_locationr)r1r#r rrrload_external_data_for_modelBs   r6rr int | Nonerr str | Nonerc Cs|dstd|jd|jdd=tj|_||dur!t|nd|dur*t|nd||dD]\}}|durG|j }||_ t ||_ q2dS)Nr,zTensor zGdoes not have raw_data field. Cannot set external data for this tensor.)rrrrr) HasField ValueErrorr(rr EXTERNALr5ritemsaddrr$r) r rrrrrkvrrrrset_external_dataRs0     r@TFall_tensors_to_one_fileboolsize_thresholdrconvert_attributec Cst|}|r t|}|rIttd}|r0tj|r tdtj |r.t d|d|}|D]}| drFt |j|krFt||q2dS|D]!}| drlt |j|krl|j}t|sgtt}t||qKdS)aCall to set all tensors with raw data as external data. This call should precede 'save_model'. 'save_model' saves all the tensors data as external data after calling this function. Arguments: model (ModelProto): Model to be converted. all_tensors_to_one_file (bool): If true, save all tensors to one external file specified by location. If false, save each tensor to a file named with the tensor name. location: specify the external file relative to the model that all tensors to save to. Path is relative to the model path. If not specified, will use the model name. size_threshold: Threshold for size of data. Only when tensor's data is >= the size_threshold it will be converted to external data. To convert every tensor with raw data to external data set size_threshold=0. convert_attribute (bool): If true, convert all tensors to external data If false, convert only non-attribute tensors to external data Raise: ValueError: If location is not a relative path. FileExistsError: If a file already exists in location. z.datazDlocation must be a relative path that is relative to the model path.zExternal data file exists in .r,N)_get_initializer_tensorsr2r$uuiduuid1ospathisabsr:existsFileExistsErrorr9sys getsizeofr,r@r(_is_valid_filename) r1rBrrDrEtensors file_namer tensor_locationrrrconvert_model_to_external_dataps<     rUcCs@t|D]}t|r|dstd|jdd=tj|_qdS)zCall to set all tensors which use external data as embedded data. save_model saves all the tensors data as embedded data after calling this function. Arguments: model (ModelProto): Model to be converted. r,raw_data field doesn't exist.N)r2r3r9r:rr r4r5)r1r rrr convert_model_from_external_datas   rW base_pathcCst|}tj||j}|dstdtj|s0t|dWdn1s+wYt|dC}| dd|j durZ| }|j |krT| d|j || |j | }| |j t||j|| |WddS1s{wYdS) zWrites tensor data to an external file according to information in the `external_data` field. Arguments: tensor (TensorProto): Tensor object to be serialized base_path: System path of a folder where tensor data is to be stored r,rVabNzr+br)r rJrKjoinrr9r:isfiler)r*rtellwriter,r@)r rXr-r.r/ file_sizerrrrsave_external_datas&         "raonnx_model_protoIterable[TensorProto]cCstt|t|S)z=Scan an ONNX model for all tensors and return as an iterator.)rrG_get_attribute_tensors)rbrrrr2sr2 attributerfunc-Callable[[GraphProto], Iterable[TensorProto]]ccsL|jtjkr||jEdH|jtjkr"|jD] }||EdHqdSdS)zICreate an iterator through processing ONNX model attributes with functor.N)typerGRAPHgGRAPHSgraphs)rerfgraphrrr_recursive_attribute_processors   rngraph_or_functionGraphProto | FunctionProtoccsDt|tr |jEdH|jD]}|jD] }t|tEdHqqdS)zICreate an iterator of initializer tensors from ONNX model graph/function.N) isinstancer initializernoderern#_get_initializer_tensors_from_graphrorsrerrrrts     rtccs0t|jEdH|jD] }t|EdHq dS)z:Create an iterator of initializer tensors from ONNX model.N)rtrm functions!_get_attribute_tensors_from_graphrbfunctionrrrrG  rGccsL|jD]}|jD]}|dr|jV|jEdHt|tEdHq qdS)zSCreate an iterator of tensors from node attributes of an ONNX model graph/function.tN)rsrer9r{rRrnrwrurrrrws     rwccs0t|jEdH|jD] }t|EdHq dS)zDCreate an iterator of tensors from node attributes of an ONNX model.N)rwrmrvrxrrrrdrzrdfilenamecCstd}||}t|S)z8Utility to check whether the provided filename is valid.z^[^<>:;,?"*|/]+$)recompilematchrC)r|exprrrrrQs  rQcCs|do |jtjkS)z?Returns true if the tensor stores data in an external location.r5)r9r5r r;)r rrrr3s  r3 field_keycCs*t|jD] \}}|j|kr|j|=qdS)aRemoves a field from a Tensor's external_data key-value store. Modifies tensor object in place. Arguments: tensor (TensorProto): Tensor object from which value will be removed field_key (string): The key of the field to be removed N) enumeraterr)r rifieldrrrremove_external_data_field's  rfilepathcCs8t|D]}t|r|drt|||dq|S)aSerializes data for all the tensors which have data location set to TensorProto.External. Note: This function also strips basepath information from all tensors' external_data fields. Arguments: model (ModelProto): Model object which is the source of tensors to serialize. filepath: System path to the directory which should be treated as base path for external data. Returns: ModelProto: The modified model object. r,)r2r3r9ra ClearField)r1rr rrrwrite_external_data_tensors5s   r)r r r#r$rr)r1rr#r$rr)NNNN)r r rr$rr7rr7rr8rr8rr)TNrAF) r1rrBrCrr8rDrrErCrr)r1rrr)r r rXr$rr)rbrrrc)rerrfrgrrc)rorprrc)r|r$rrC)r r rrC)r r rr$rr)r1rrr$rr)( __future__rrJr}rOrH itertoolsrtypingronnx.onnx_cpp2py_export.checkeronnx_cpp2py_exportcheckerr& onnx.onnx_pbrrrrr collections.abcr r r r0r6r@rUrWrar2rnrtrGrwrdrQr3rrrrrrsH        :  "