o Ǝi @sddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZddlmZddl mZmZmZmZmZmZmZmZddlmZmZddl mZddlZddlmm Z!ddlm"Z"m#Z#m$Z$m%Z%dd l&m'Z'm(Z(dd l)m*Z*dd l+m,Z,dd l-m.Z.dd l/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>ddl?m@Z@ddlAmBZBmCZCddlDmEZEddlFmGZGddlHmIZImJZJddlKmLZLmMZMmNZNmOZOmPZPmQZQddlRmSZSddlTmUZUddlVmWZWmXZXmYZYer4ddlZm[Z[m\Z\m]Z]m^Z^ddl m_Z_ddl`maZadd lbmcZcdd!ldmeZemfZfegehZiejjkehd"Zlz ejjkehd#ZmWn$enymZozd$epeovraegehd%ZmneoWYdZo[ondZo[owweqZrej jsZted&ZuejvjwjxZxdZydazGd'd(d(Z{eGd)d*d*e|Z}eGd+d,d,e|Z~eGd-d.d.e|ZeGd/d0d0e|ZeGd1d2d2e|ZeGd3d4d4e|ZGd5d6d6e jZeZdd:d;Zejdd=d>ZejddBdCZddHdIZddMdNZddQdRZejddVdWZejddYdZZdd`daZddbdcZGdddedeZejddidjZejddkdlZddmdnZGdodpdpZGdqdrdrZGdsdtdte%ZeeBdudvfZeEeGdwdxdxZddydzZeEeGd{d|d|ZGd}d~d~ZeEeddGdddZeEeddGdddZeEeddGdddZereeefZeEeddGdddeZeEeddGdddZGdd@d@eJZeZdddZdddZdddZdddZGddde@ZdddZdddZdddZejwjjjeejwjxjjddejwjxjjddejwjxjjddiZeejwjxjjejwjxjjejwjxjjejwjjjZddlmZmZmZmZmZmZmZmZdddZejdddZdddZdddZdS)) annotationsN) defaultdict) dataclass)AnyCallablecastLiteralOptional TYPE_CHECKINGTypeVarUnion)Self TypeGuard) ReferenceType)SymBoolSymFloatSymIntTensor)is_functorch_wrapped_tensoris_legacy_batchedtensor)FakeScriptObject)MissingOpProfile)dtrace_structured)suggest_memory_format) assert_eqassert_metadata_eq is_sparse_anyis_sparse_compressed MetaConverter) render_call)immutable_dict)normalize_function)StorageWeakRef)TorchFunctionMode) IntLikeType py_sym_types)dataclass_slots) no_dispatch)is_traceable_wrapper_subclassTorchDispatchMode)KeyPathkeystrPyTreetree_map tree_map_TreeSpec)count)CapturedTraceback)_CacheKeyState_PySymInputStub_SymIntOutputStub) GeneratorIterableMappingSequence) TracebackType)Source) OpOverload)ShapeEnvSymbolicContexthierarchical_compilenot_implementedz 'not_implemented' not registeredz.not_implementedTc@s eZdZdddZdddZdS) IncrementRecursionCountreturnNonecCs td7adSNr2RECURSION_COUNTselfrJY/home/jeff/fluffinator/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py__init___ z IncrementRecursionCount.__init__cCs td8adSrErFrHrJrJrK__del__crMzIncrementRecursionCount.__del__NrCrD)__name__ __module__ __qualname__rLrNrJrJrJrKrB^s rBc@eZdZUded<dS)UnsupportedFakeTensorExceptionstrreasonNrPrQrR__annotations__rJrJrJrKrTh rTc@rS)DynamicOutputShapeExceptionr<funcNrWrJrJrJrKrZmrYrZc@rS)DataDependentOutputExceptionr<r[NrWrJrJrJrKr\rrYr\c@rS)UnsupportedOperatorExceptionr<r[NrWrJrJrJrKr]wrYr]c@rS)$UnsupportedMutationAliasingExceptionrUrVNrWrJrJrJrKr^|rYr^c@rS)MetadataMismatchErrorrUrVNrWrJrJrJrKr_rYr_c@s eZdZUded<dddZdS) FakeTensorTLSOptional[bool]allow_non_fake_inputs_overriderCrDcCs d|_dSN)rbrHrJrJrKrL zFakeTensorTLS.__init__NrO)rPrQrRrXrLrJrJrJrKr`s r`itemsrCdict[T, Literal[True]]cGs t|dSNT)dictfromkeys)rerJrJrK ordered_set rj2Generator[Optional[TorchDispatchMode], None, None]c csRtjtjjj}z|VW|durtj|dSdS|dur(tj|wwrc)torch_C_unset_dispatch_mode_TorchDispatchModeKeyFAKE_set_dispatch_mode)oldrJrJrKunset_fake_temporarilys rt fake_modeFakeTensorModeGenerator[None, None, None]ccs*|j}z d|_dVW||_dS||_w)NF) cache_enabled)ru old_valuerJrJrKdisable_fake_tensor_caches rzsubclassrout list[Union[Tensor, int, SymInt]]csV|g}|r)|ts|q\}}|fddt|D|s|S)Nc3s|]}t|VqdSrcgetattr).0keycurrrJrK z$get_plain_tensors..)popr(append__tensor_flatten__extendreversed)r{r|todo inner_keys_rJrrKget_plain_tensorss   rxobjectTypeGuard[Tensor]c sddlm}ttr dStr?t\}}fdd|D}tdd|D}tdd|D}||ks=Jd |St|rIt j Stt rdt rdt j}t jj|}t |Stt rxtrxt jj}t |Sd S) NrFunctionalTensorTcsg|]}t|qSrJr~)rattrrrJrK szis_fake..cs|]}t|VqdSrcis_fakerrrJrJrKrzis_fake..csrrcrrrJrJrKrrz got mixed fake and real tensors!F)#torch._subclasses.functional_tensorr isinstance FakeTensorr(typerallanyrelemrrm_is_functional_tensorrn$_functionalization_reapply_views_tls _functorch_unwrap_functional_tensorr get_unwrapped) rrattrsrflattened_tensorsall_fakeany_fake reapply_views unwrappedrJrrKrs(     rtOptional[FakeTensorMode]csddlm}ttrjStr4\}}fdd|D}|dtfdd|Ds2JSt|r>tj Stt rYt rYt j }t j j|}t|Stt rmtrmt j j}t|SdS)Nrrcsg|] }tt|qSrJ)maybe_get_fake_moder)rt_namerrJrKrsz'maybe_get_fake_mode..c3s|]}|uVqdSrcrJr)mrJrKrrz&maybe_get_fake_mode..)rrrrrur(rrrrrrmrrnrrrrr)rrinner_tensor_namesrmodesrrrJ)rrrKrs*       rr[r<torch._C._SchemaInfocCstj|jSrc)rmrn _SchemaInfo_schemar[rJrJrKget_schema_infosrboolcCs6ddlm}tjj}||jdo||jt|vS)Nrdecomposition_tablez torch._decomp) torch._decomprrm_decompdecompositionsrQ startswithrPdir)r[rrrJrJrKtorch_decomp_decompositionss  rtytype[T]treer,list[T]cst|}fdd|DS)Ncsg|] }t|r|qSrJ)rrrrrJrKrz%tree_flatten_only..)pytree tree_leaves)rr flat_valsrJrrKtree_flatten_onlys rcCs@t|tuo|jtjko|jp|jpt|pt|pt | Src) rrlayoutrmstrided is_sparse is_nestedrrrrrJrJrK_is_plain_tensors  rc@seZdZUed7ddZded<ded<d ed <d d d d8ddZd9ddZd:ddZd;ddZ dd-d.Z d?d@d5d6Z d S)AFakeTensorConverterrCweakref.WeakValueDictionarycCs|jjSrc)meta_converter tensor_memorHrJrJrKrszFakeTensorConverter.tensor_memorrz)dict[StorageWeakRef, list[ReferenceType]]constant_storage_mappingrexportF copy_datarrrDcCst|d|_||_i|_dS)N)r)rrrr)rIrrrJrJrKrL's  zFakeTensorConverter.__init__ fake_tensorrcCsTt|tr |jdus Jt|j}||jvrg|j|<|j|t|dSrc) rrconstantr"_typed_storagerrweakrefref)rIrweak_strJrJrKadd_constant_storage_mapping.s   z0FakeTensorConverter.add_constant_storage_mappingtensorrcCs`t|trJt|}||jvrdS|j|D]}|}|dur)|d|_q|j|=dSrc)rrr"rr _fix_weakrefr)rIrrweak_tensor_reftenrJrJrKinvalidate_constant_aliases<s   z/FakeTensorConverter.invalidate_constant_aliasesrOptional[FakeTensor]cCs(|jjj|}|durdS|j|Src)r describer lookup_tensorgetr)rIrtidrJrJrK _get_memoKs zFakeTensorConverter._get_memovcCs|jj|}||jj|<dSrc)rr get_tensor_idr)rIrrrrJrJrKset_tensor_memoQsz#FakeTensorConverter.set_tensor_memoNT)sourcesymbolic_contexttracerurv make_constant shape_envOptional[ShapeEnv]rOptional[Source]rOptional[SymbolicContext]rcs0|s(|s(|r(tjj}r(||jvr(|j|}ddlm} t|| s%J|j}| |} | dur3| S|j r:t dt |tj jurF|rFJ|rJ|nddfd d } |j||| |||d } | turgt d ddlm} d}|jst|r|dkr|jj dkr|jtjtjtjtjtjfvr|durt|| s|durddlm}m}ddlm}t |!}Wdn1swYt"#|st"$|st||r|j%}n||}|j&|||j'|d}|jtjkr|j(|||d| _)n|jtjkr|j*|||d| _)|r|+| | S)Nr)StatefulSymbolicContextzquantized nyi in meta tensors make_meta_tCallable[[], object]deviceUnion[torch.device, str]rCrcs<tt||dWdS1swYdS)Nr)r'r)rrrrurJrKmk_fake_tensors $z.mk_fake_tensor)rcallbackrrrzmeta converter nyi)RandomValueSourcecpu)CallMethodItemSourceFloatTensorSource) DimDynamic)r dynamic_dimr)hintr)rrrrrCr),rm_guardsTracingContexttry_gettensor_to_context%torch.fx.experimental.symbolic_shapesrr tensor_sourcer is_quantizedrTrnn ParameterrNotImplementedtorch._dynamo.sourcerrrdimrdtypeint64int32int16int8float64rrrr'itemmathisnanisinfbasecreate_unspecified_symbolDYNAMICcreate_symintnode item_memocreate_symfloatnoder)rIrurrrrrrtracing_contextr maybe_memorr|rvaluerrr item_sourcesymbolrJrrKfrom_real_tensor[s               z$FakeTensorConverter.from_real_tensorr torch.devicepytypeOptional[type[torch.Tensor]] dispatch_keysOptional[torch.DispatchKeySet]cCsX|jjdksJd|jjd||}|dur|St|||||d}||||S)Nmetaz$tensor's device must be `meta`, got z instead)r,r.)rrrrr)rIrurrr,r.r&r|rJrJrKfrom_meta_and_devices   z(FakeTensorConverter.from_meta_and_device)rCr)rrrrrCrD)rrrCrD)rrrCrD)rrrCr)rrrrrCrD)FN)rurvrrrrrrrrrrrrrCr)NN) rurvrrrr+r,r-r.r/rCr) rPrQrRpropertyrrXrLrrrrr*r1rJrJrJrKrs,      rrr+rDcCsDtjs tjr tjjdurtjd|dntjd|ddSdS)Nr2r)rmcuda is_availablexpuversionhipemptyzerosr3rJrJrKinit_gpu_contexts  r;c cs|j}tj}||ksJ|d|tj<d|_tjtjdzdVW||_n||_wWdn1sBwYWddSWddS1sZwYdS)N, T)in_kernel_invocationrmrn_meta_in_tls_dispatch_include_DisableTorchDispatch_PreserveDispatchKeyGuard!_set_meta_in_tls_dispatch_include)ruprev_in_kernel meta_in_tlsrJrJrKin_kernel_invocation_manager s    "rDcCs$tj|ddddS)Nz::.r)rmrn _should_allow_numbers_as_tensorsnamesplitrrJrJrKshould_allow_numbers_as_tensors$srJc@seZdZejdddkZdS)FakeTensorConfigTORCH_FAKE_TENSOR_DEBUG01N)rPrQrRosenvironrdebugrJrJrJrKrK*srKc@sneZdZUded<ded<ddd!d d Zd"ddZd#ddZd#ddZd#ddZ d$d%ddZ d&dd Z dS)'SymNumberMemoDescriptorrU_namer_is_nested_intF is_nested_intrVrCrDcC ||_dSrc)rT)rIrVrJrJrKrLCrdz SymNumberMemoDescriptor.__init__ownerrHcCs ||_dSrcrS)rIrXrHrJrJrK __set_name__Frdz$SymNumberMemoDescriptor.__set_name__objrcCs d|jS)NrrYrIr[rJrJrK_memoIrkzSymNumberMemoDescriptor._memocCd|jdS)Nr_vcrYr\rJrJrK_memo_vcLz SymNumberMemoDescriptor._memo_vccCr^)Nr_epochrYr\rJrJrK _memo_epochSraz#SymNumberMemoDescriptor._memo_epochNobjtypeOptional[type[FakeTensor]]-Optional[Union[torch.SymInt, torch.SymFloat]]cCst|||}durdSt|tjr|jjdur|S|js*t||||j ks9|jsDt|| ||j j krDt |||ddS|Src)rr]rrmrnoderrTr`_versionrcruepochsetattr)rIr[rdrrJrJrK__get__VszSymNumberMemoDescriptor.__get__r'cCs|dur!t|||dt|||dt|||ddS|r(|jrKt|||||js>t||||jt||||jjdSdSrc) rjr]r`rc is_inferencerTrhruri)rIr[r'rJrJrK__set__lszSymNumberMemoDescriptor.__set__)rVrrCrD)rXrUrHrUrCrD)r[rrCrUrc)r[rrdrerCrf)r[rr'rfrCrD) rPrQrRrXrLrZr]r`rcrlrnrJrJrJrKrR9s      rRcs$eZdZUdZded<ded<ded<ded<eZeZeZeZ ed d Z d ed <d ed<e j j jZed?ddZejd@ddZedAddZejdBddZe    dCdDddZdEfd#d$ ZedFd&d'Zeed(efdGd/d0ZedHd3d4Zd5d6dId:d;ZdJd=d>ZZS)Kral Meta tensors give you the ability to run PyTorch code without having to actually do computation through tensors allocated on a `meta` device. Because the device is `meta`, meta tensors do not model device propagation. FakeTensor extends MetaTensors to also carry an additional `fake_device` which tracks devices that would have been used. r+ fake_devicervruOptional[Tensor]r real_tensorTrUOptional[type[Tensor]]r,r/r.rCcCs|jjr tdS|jSNr0)rur=rmrrorHrJrJrKrs zFakeTensor.devicerrDcCtrcNotImplementedErrorrIrrJrJrKr list[str]cCstd)Nz+torch.compile doesn't support named tensors)rTrHrJrJrKnamesszFakeTensor.namescCrtrcrurwrJrJrKrzrxNrrrr c Csftj|||jd|d}|jstj|ntj||jj dks'J|jj t |tjr/|nt|}|j s>|j dks>J|j dvrGt ||j ddddtj fvr|jdur|j dkrxtt|j rxt|j d tt|j }n t|j d }||_||_||_||_||_t |trJ||_d|_d|_d|_d|_d|_tjrt !|_"|S) NT)dispatch_devicedevice_for_backend_keysr0)r4r6r4hpur6mps:z:0)#r_make_subclass requires_grad_allow_unsafe_data_ptr_accessrmrn_set_throw_on_mutable_data_ptr(_set_warn_deprecated_on_mutable_data_ptrrrr allow_metar;_get_privateuse1_backend_nameindexris_initializedcurrent_devicerorurr,r.rrq nonzero_memor# unique_memounique_consecutive_memonested_int_memorKrQr1extract _debug_trace) clsrurrrrqr,r.rIrJrJrK__new__sP     zFakeTensor.__new__argsrkwargscstdSrc)superrL)rIrr __class__rJrKrLrazFakeTensor.__init__rcCs ||Src) from_tensor)rrurJrJrKrs zFakeTensor.from_tensorrJr[r<typesSequence[type]Sequence[object]Mapping[str, object]c Cs(|tjjjjkr't|dkrt|dtsJ|djj r"tdS|dj St |}r2||Sdd|D}|rCt d|tSd}tj|i|D] }t|trY|j}nqM|dus`Jtjtjjj} | rtt d|| tS|j ryJ|||i|WdS1swYdS)Nr2rr0cSs"g|] }t|ts|tur|qSrJ) issubclassrrrrrJrJrKrBs z1FakeTensor.__torch_dispatch__..z(FakeTensor unrecognized subclass(es): %sz(FakeTensor mode already active: %s in %s)rmopsprimrdefaultlenrrrur=ro_DISPATCH_META_HANDLERSrnot_implemented_logrQrrarg_tree_leavesrn_get_dispatch_moderprq) rr[rrrhandlerunrecognized_typesruargmaybe_cur_fake_moderJrJrK__torch_dispatch__sH          $zFakeTensor.__torch_dispatch__ flat_argstuple[torch.device, bool]csdd}dttjjttjjddddfd d dfdd }|D]}||q-trCdurCd}tddusNJd|fS)NFrr+rCrcSs |jdkS)Nr)rr3rJrJrKcheck_cpu_devicerdz8FakeTensor._find_common_device..check_cpu_devicerrcs|jo |dkSNr)rrr)rrJrK cpu_zero_dimz4FakeTensor._find_common_device..cpu_zero_dimrrDcst|tsdSdur|j|dS|}|jkr#r!|dSv}|r-|s-dSr8|s8|j|dSvrHtt|jfrHdStjjj}|durm|j v}||jj v}|sg|rg|j|dS|rm|smdSt ddd|j)Nz,Unhandled FakeTensor Device Propagation for z, found two different devices r<) rrrrmaprmrconfigfake_tensor_prefer_device_typer RuntimeError)rt_is_cpu_zero_dim&is_bypass_zero_dim_cpu_tensor_check_opprefer_device_typecommon_has_preferredt_has_preferred$bypass_zero_dim_cpu_tensor_check_opsr common_devicerr[is_cpu_zero_dimmixed_device_fnsrJrK merge_devicessF     z5FakeTensor._find_common_device..merge_devicesTrz!Could not find common device for )rr+rCrrrrCr)rrrCrD)rjaten _foreach_copyr nextafterrJrmr)r[rhas_scalar_only_inputsrrrJrrK_find_common_devicejs&  =  zFakeTensor._find_common_devicer2)coeffrUnion[int, torch.SymInt] torch.SymIntcCs6|jdur |jjdd|_t|jtjsJ|j|S)N nt_tensor_id)rrucreate_symbolic_nested_intrrmr)rIrrJrJrKget_nested_ints  zFakeTensor.get_nested_intrcCs<|dkr |S|dkrdd|DSdd|DS)Nrr2cSg|]}|qSrJ)rrrJrJrKrz%FakeTensor.tolist..cSrrJ)tolistrrJrJrKrr)rrrHrJrJrKrs  zFakeTensor.tolist)rCr+)rr+rCrD)rCry)rryrCrD)NNNN)rurvrrrr+rrprqrpr,rrr.r/rCr )rrrrrCrD)rrrurvrCr r[r<rrrrrrrCr)r[r<rrrCr)rrrCr)rCr) rPrQrR__doc__rXrRrr#rrrrmrnrprq _mode_keyr2rsetterrz staticmethodrrLr classmethodr0r rrrr __classcell__rJrJrrKrzsR       S I k rr4r5c@seZdZUdZded<ded<ded<ded<d ed <d ed <d ed<ded<ded<ded<ded<ded<ded<ded<ded<ded<ded<d(d%d&Zd'S))TensorMetadatazK The Tensor metadata relevant to hashing FakeTensors when caching. z torch.dtyperztuple[_MetadataIntLike, ...]shapestrider+rz torch.layoutrzOptional[torch.memory_format] memory_format_MetadataIntLikestorage_offsetzOptional[_MetadataIntLike] storage_bytesrrris_conjis_negrmrra is_coalesced Optional[int] dense_dim sparse_dimresult list[object]modervstater3rCrDcCsrt|D]1}t||j}t|tttjfr%g}| ||||| qt|t r1| ||q| |qdSrc) dataclassesfieldsrrHrtuplelistrmSize_prep_args_for_hashclearrconvert_sym_intr)rIrrrfieldr'id_hashed_objectsrJrJrK _flatten_intos    zTensorMetadata._flatten_intoN)rrrrvrr3rCrD)rPrQrRrrXrrJrJrJrKrs( rcCst|}|jst|s|j|dsd}|}t|j|j|jt j kr&| nd|j |j||t|s7| nd|j|j||||j|jrN|ndt|rW|ndt|ra|SdS)z1 Extract the TensorMetadata of a tensor. )rNrJ)r_has_symbolic_sizes_stridesr is_contiguousrrrrrrmrrruntyped_storagenbytesrrrrrmrrrr)rrrrJrJrKextract_tensor_metadatas< rc@sJeZdZUdZded<ded<dd d ZdddZdddZdddZdS)_DispatchCacheKeyz0 Key for the FakeTensor dispatch cache. tuple[object, ...]rint hashvaluetuprCrDcCs||_t||_dSrc)rhashr)rIrrJrJrKrLKsz_DispatchCacheKey.__init__otherrrcCst|to |j|jkSrc)rrr)rIrrJrJrK__eq__Orz_DispatchCacheKey.__eq__cCs|jSrc)rrHrJrJrK__hash__Rsz_DispatchCacheKey.__hash__cCs"|jD] }t|tr|qdSrc)rrr4strip_shape_env)rIrrJrJrKrUs  z!_DispatchCacheKey.strip_shape_envN)rrrCrD)rrrCr)rCrrO) rPrQrRrrXrLrrrrJrJrJrKrAs    rc@s eZdZdS)SingletonConstantN)rPrQrRrJrJrJrKr_srT)frozenc@s6eZdZUdZded<ded<ded<eZded<d S) _DispatchCacheEntryOutputInfoa Entry type for the FakeTensor dispatch cache for an output. Accounts for three possibilities: 1) The op is inplace, and a hit means we need to alias the argument at a given index. 2) We need to synthesize a new FakeTensor given tensor metadata. For view ops, we further capture the index of the arg to alias. 3) if the tensor related fields are None, then it is a constant value (e.g. None or integer) r inplace_idxzOptional[TensorMetadata]metadataview_idxz Optional[Any]constant_valueN)rPrQrRrrXrrrJrJrJrKrcs  rc@s&eZdZUdZded<dZded<dS)_DispatchCacheValidEntryz Entry type for the FakeTensor dispatch cache. It supports two types of outputs 1) tensor 2) tuple of tensors is_output_tuple flag helps in differentiating the return type z$tuple[_DispatchCacheEntryOutputInfo] output_infosFris_output_tupleN)rPrQrRrrXr rJrJrJrKr ws r c@eZdZUdZded<dS)_DispatchCacheBypassEntryz0 Entry type for a negative cache entry. rUrVNrPrQrRrrXrJrJrJrKr   r c@r )_BypassDispatchCachez< Signals cases that should skip FakeTensor caching. rUrVNrrJrJrJrKrrrc@s2eZdZUdZded<ded<ded<ded<dS) DispatchCacheInfozG Information about the state of the FakeTensor dispatch cache. rhitsmissesdict[str, int]bypassessizeNrrJrJrJrKrs  rc seZdZUiZded<dZded<dZded<eeZ ded<dZ ded <d Z d ed <d ed <ded<ded<d ed<dZ ded<dZ ded<dd ddd ddfdd Zdd!d"Zdd&d'Zedd(d)Zedd+d,Zed-efdd6d7Zdfd9d: ZdfdAdB ZeddCdDZeddFdGZeddHdIZddJdKZddOdPZddQdRZddWdXZ dd\d]Z!dd`daZ"ddcddZ#ddfdgZ$ddidjZ%ddkdlZ&d-efddmdnZ'ddtduZ(dd{d|Z)dd}d~Z*e+ddddddddZ,dddZ-dddZ.dddZ/dddddZ0e+e1j2j3e1j4j5e1j6j5e1j7j5e1j8j5e1j9j5e1j:j5e1j;j5e1je1j?j5 Z@e+e1jAj5e1jBj5ZCdddZDe+e1jEj5e1jFj5ZGdddZHdddZIddddddddZJZKS)rv,dict[_DispatchCacheKey, _DispatchCacheEntry]cacherr cache_hits cache_missesrcache_bypassesriFrr= static_shapesrrz Optional[str]_stackrrEnt_tensor_id_counternt_tensor_id_initial_countTN)allow_fallback_kernelsallow_non_fake_inputsrrrr r!rarrCrDcstdt|t||_ddl}ddl}|jj j |_ t |j |d|_ |dur.||_n|du|_d|_|jj j|_|jj j|_|jj joJ|j |_|jj j|_||_d|_g|_||_t|_d|_ |j!j"j#|_$ddl%}|j&j'j(j)|_*|j*|_+dS)Nzcreate_mode 0x%xrrF),logrQidrrLr torch._dynamo.configtorch._functorch.configrr"fake_tensor_propagate_real_tensorspropagate_real_tensorsrfake_tensor_converterrallow_scalar_outputs(fake_tensor_allow_unsafe_data_ptr_accessrfake_tensor_allow_metar_dynamofake_tensor_cache_enabledrx$fake_tensor_cache_crosscheck_enabledcache_crosscheck_enabledr!r= enter_stackr traceback extract_stack _stack_tracerrnrprqr$torch.nested._internal.nested_tensornested _internal nested_tensor_tensor_id_counterrr)rIr r!rrrrmrrJrKrLsF          zFakeTensorMode.__init__cCs |j|_dSrc)rrrHrJrJrKreset_nt_tensor_id_counter+rkz)FakeTensorMode.reset_nt_tensor_id_counterrrTypeGuard[FakeTensor]cCst|to |j|uSrc)rrrurIrrJrJrK is_our_fake:szFakeTensorMode.is_our_fakecCsDtjrtjr Jtj Stjp ttdo tj S)Nr})rmr6 _is_compiledr4r5hasattrr}rHrJrJrKavoid_device_initBs   z FakeTensorMode.avoid_device_initrUcCs$|jdurdt|j|_|jS)N)rjoinr1 format_listr3rHrJrJrKstackMs zFakeTensorMode.stackrJr[r<rrrrrrcCsLtjtjjjdusJ|z |||||WSty%tdw)Nzfake tensor raised TypeError) rmrnrrprqdispatch TypeErrorr" exception)rIr[rrrrJrJrKrSs   z!FakeTensorMode.__torch_dispatch__r cs|ddl}d}|jr|j}|jd|j|j}||ur-|jd||ft S|j ||jdd|f|S)NrTF) r4r?rn_only_lift_cpu_tensors_set_only_lift_cpu_tensorsrorr0rr __enter__rr)rIrmprev_only_lift_cpu_tensorsmaybe_prev_fake_moderrJrKrIfs    zFakeTensorMode.__enter__aOptional[type[BaseException]]bOptional[BaseException]cOptional[TracebackType]csX|j\}}}|r(t||||durtj||dur*tj|dSdSdSrc)r0rr__exit__rmrnrrrH)rIrLrNrPliverK maybe_prev_only_lift_cpu_tensorsrrJrKrRzs  zFakeTensorMode.__exit__cCsdSrgrJrrJrJrK is_infra_moderxzFakeTensorMode.is_infra_modercCsttjtjttjttjS)z8 Query the state of the dispatch cache. )rrvrrrhrrrrUrJrJrK cache_infos zFakeTensorMode.cache_infocCs$d|_d|_|j|jdS)z+ Clear the dispatch cache. rN)rrrrrrUrJrJrK cache_clears zFakeTensorMode.cache_clearc Csd}d}zt|j}|||||}Wn3tyF}z't|tjjr2|dkr2t d|d|j t j |j d7<WYd}~nd}~ww|durS|||||S|dusYJ|rk|jdusdJ|jj}t} nt j}t} ||d} | durt| trt j | j d7<|||||S||| |||} t jd7_|jrt||| ||||Wd| S1swY| S|||||} z ||||Wn?ty}z2t|tjjr|dkrt d|d|j t j |j d7<| ||t|j | WYd}~Sd}~wwz ||||||| } Wn(tyI}zt j |j d7<| ||t|j | WYd}~Sd}~ww| ||| t jd7_| S)z Lookup a cache entry for the given arguments. If none exists, dispatch and cache the result (if the result is eligible for caching). Ninvoke_subgraphz6Fake tensor cache failed: identifier = %s, reason = %sr2)r3r _cache_keyrrrm_opsHigherOrderOperatorrHhc_logrQrVrvr_dispatch_implcache_on_shape_envfake_tensor_cache_set_cache_key_for_shape_envr_set_cache_keyrr _output_from_cache_entryrr/rz_crosscheck_cache_output_validate_cache_key_make_cache_entryr) rIr[rrrrrer set_cache_keyentryoutputrJrJrK_cached_dispatch_impls              z$FakeTensorMode._cached_dispatch_implrr3rc Cs|ttjt|jr|jjndg}|jr||j g}|r*| |||||r4| ||||t t |}|D] }t |tjt|dq<||S)z Create a cache key given the dispatch args. Raises _BypassDispatchCache for any situation that precludes caching. Nr)rmget_default_dtypern_get_default_deviceis_inference_mode_enabledrsettings known_symbolsrrirrrrfinalize functoolspartialevict_fake_tensor_cache_keyr) rIrr[rr key_valuesrr id_hashed_objrJrJrKrZ s(   zFakeTensorMode._cache_keycCs>ddlm}t|tjjr||vrdStjj|jvrt dtjj |jvrU|t j j urQt|||dd\}}|dD]}t|tj rN|jtjtjfvrNt dq9dSt dtjj|jvr`t d |t jjkrjt d ||jvrst d |d kr}t d tjj|st d |jrtj|tjjjrt ddSdS)za Validate that the cache key generated by _cache_key will be reasonable. rregistered_hop_fake_fnsNzdata dependent outputTrrnormalize_to_only_use_kwargsindiceszdynamic output shapez inplace viewz unsafe viewliftzinductor::resize_storage_bytes_z non-builtinCompositeImplicitAutograd)torch._higher_order_ops.utilsryrrmr[r\Tagdata_dependent_outputtagsrdynamic_output_shaperrrr!rrr inplace_view _unsafe_viewrlift_fnsrH_libraryutils is_builtinis_viewrn%_dispatch_has_kernel_for_dispatch_key DispatchKeyr~)rIr[rrryr new_kwargsrrJrJrKre8sN       z"FakeTensorMode._validate_cache_keyrr?Union[Mapping[str, object], Sequence[object], Iterable[object]]rc Cs ddlm}ddlm}t|tttfr%|t ||dt |t|tr@| || ||| || ||dS|D]}t|trs||sRtd|jdur[tdt|rgt|jdt|}||||qBt|tr|td t|tr|||qBt|ttfrtd t|tttfr| ||||qBt|tjrtd t|tjjr|t ||t |||qBt||r|t!|||j"qBt||r|t ||t!|||j#qB|t |||qBdS) a7 Translate the provided args into a form suitable for caching at FakeTensor dispatch, i.e., convert unhashable types like lists & dicts into tuples and convert FakeTensors into metadata. Raises _BypassDispatchCache to signal unsupported cases that should bypass caching. r)FunctionalCallableWithEpilogue)FunctionalizeCtxWrapperlength_Nz not our fakeconstant attributez tensorznon-fake tensorzsymbolic shapezfunction argument)$*torch._higher_order_ops.auto_functionalizerrrrrrrhrrrrkeysvaluesrr<rrrrrrrrrrrr FunctionTypermfx GraphModuler#rsubgraph orig_callable) rIrrrrrrrrrJrJrKr~sV            z"FakeTensorMode._prep_args_for_hashrrjrcCst|ttdfr dSt||rtdt|tstd|jdur&td|jr-tdt|r5td| D]}t |t |krGtdq9dS)Nzunrepresented symbol in outputznon-FakeTensor outputrz sparse outputzsparse compressed outputzkwarg aliases output) rrr_has_unrepresented_symbolsrrrrrrr#)rIrrr[rrrjkvalrJrJrK _validate_output_for_cache_entrys"     z/FakeTensorMode._validate_output_for_cache_entryrrcst|ttjtdfrtddd|dStt|D]}t||t|kr.t|dddSqd}t|tj j rP|j rPddt |D} t| dksLJ| d}t |} tfdd| jD| _tfd d| jD| _| j| _| jdur|dn| j| _td| |d} t| fd d } dd lm} z || |||}Wn | ytd dwtj|}tj|}||krtd| S)Nrrrr)rrrcSsg|] \}}t|tr|qSrJrr)rirrJrJrKr szCFakeTensorMode._get_output_info_for_cache_entry..r2rc3|]}|VqdSrcconvert_outputrrrrJrKrrzBFakeTensorMode._get_output_info_for_cache_entry..c3rrcrrrrJrKrrFr r )GuardOnDataDependentSymNodezdata dependent symnodezdispatch_key_set mismatch)rrrmrrrrangerr#r[r<r enumeraterrrrrrrr r rrcrrn_dispatch_key_set)rIrrr[rrrjidxridxsrrientry_for_synth_outputr synth_output synth_key_setkey_setrJrrK _get_output_info_for_cache_entrysX          z/FakeTensorMode._get_output_info_for_cache_entryr c s0ddlm}ddlmttjjr3|vr3t|tsJt fdd|D}|r3t ddt|t tj t dfrMtddd|d } t| fd d St|trb|D] } | qTn |t|trfd d |D} tt| dd S|} t| fd d S)z Make a cache entry object for the given 'output' Tensor. Raises _BypassDispatchCache if the output tensor has characteristics that prevent caching it. rrxhas_free_unbacked_symbolsc3s*|]}t|tjtjfo|VqdSrc)rrmrr)rorrJrKrVs  z3FakeTensorMode._make_cache_entry..zunbacked symbol in HOP z outputNrFrc s g|] }|qSrJ)r)rout_elem)rr[rrrIrrJrKrqs  z4FakeTensorMode._make_cache_entry..T)rryr rrrmr[r\rrrrrrrr rr) rIrrr[rrrjry non_cacheable output_info out_elementr rJ)rr[rrrrIrrKrf;sP         z FakeTensorMode._make_cache_entryric s0|jdur|jdur|jdur|jtusJ|jS|jdur,||j}t|ts*J|S|j}|dur5dSt|r;Jdfdd tfd d |j D}tfd d |j D} |j } |j durm|j t j} |jdury|jj} t|)| tj|| |j|jd |jd } Wdn1swYWdn1swY|jrtj| d|jrtj| dt|tjjr|jr|tt |j} t| tsJ| !}t|#| | "|| || Wdn1swYWdn 1s wYt|| |j#S)Nr'rrr3rCUnion[IntLikeType]cs8t|tr|jdus J||jSt|trJ|Src)rr5rrr4)r'rrlrJrK check_values zGFakeTensorMode._get_output_tensor_from_cache_entry..check_valuec3|]}|VqdSrcrJrrrrJrKrrzEFakeTensorMode._get_output_tensor_from_cache_entry..c3rrcrJrrrJrKrrr0)rrrrT)r'rrr3rCr)$rrrrrrrrrrrrr contextlib nullcontextrsuppress_guardsrDrm empty_stridedrrrrrn _set_conjr_set_negr[r<rrrrset_r)rIrrirr[r inplace_argrrrrmaybe_suppressr9view_argstoragerJ)rrrrK#_get_output_tensor_from_cache_entrys`           z2FakeTensorMode._get_output_tensor_from_cache_entry=Union[Optional[FakeTensor], tuple[Optional[FakeTensor], ...]]csB|jrfdd|jD}t|S|jdS)z? Create a new FakeTensor from the cache entry. c sg|] }|qSrJ)r)rrrr[rrIrrJrKrs  z;FakeTensorMode._output_from_cache_entry..r)r r rr)rIrrirr[routputsrJrrKrcs z'FakeTensorMode._output_from_cache_entryc sd fdd z |||||}Wnty.}ztd|d |d |d ||d }~wwz||Wd StyQ}ztd|d |d ||d }~ww)z Helper to validate that the output synthesized from the cache matches the output created by normal dispatch. rLrrNrCrDcst|tr't|ts Jt|t|ksJt||D] \}}||qdSt|tr9t|tr5||ks7JdS|durE|dusCJdSt|tr\t|t|krX|j|jusZJdSt|tj rrt|tj sjJt t ||dSt dt|)NzUnsupported type ) rrrziprr%rrgrmrrrr)rLrNlrk assert_helperrJrKrs    $ z>FakeTensorMode._crosscheck_cache_output..assert_helperz*FakeTensor cache crosscheck failure: func=z, args=z , kwargs=z: Dispatch raised=N)rLrrNrrCrD)r^ Exceptionr)rIrjr[rrr true_outputrgrJrrKrds> z'FakeTensorMode._crosscheck_cache_outputcCs|pi}ttd|||Wdn1swY|tvr)t||Sttjkrtd|d|ddSdSt|tttfrW||krYtd|d|ddSdS)NT) compute_hintmismatch between fake value  and real value r)rrrrrgexpr free_symbols var_to_valrunbacked_var_to_val_maybe_evaluate_staticEqStruer_rfloatr)rr)rIsympyrJrK_check_fake_real_vals@s6   z?FakeTensorMode._maybe_infer_fake.._check_fake_real_valsReal tensor propagation foundFT)contextsizesstridesrrmismatched_fake_kernelctjdSNoprVrUrVrJexcr[rJrKjz2FakeTensorMode._maybe_infer_fake.. metadata_fnzFReal tensor propagation found a metadata mismatch between fake tensor z and real tensor z , at outputz , for func: crrrrJrrJrKr}rzIReal tensor propagation found an output size mismatch between fake shape z and real shape z , at outputz.size(z ), for func: cstdddS)NrrrrUrJ)rr[rrJrKrszQReal tensor propagation found an output value mismatch between fake output value z and real output value )rrrrrCrD)rtorch._subclasses.fake_utilsrrrmrr_rr*generate_fake_kernels_from_real_mismatchesr_infer_fake_from_real_tensorr+rrr) rIr[rrrrrjs_fakes_realrJ)rrr[rrIrrK_maybe_infer_fake4s     "   '  z FakeTensorMode._maybe_infer_fakefake_inreal_infake_outreal_outOptional[object]c sddlm}djrtjjdfdd }t|\}} t|\} } z |d||||Wn=tynz1t j j j r\t d fd d d |tfd d |WYdStd|d|ddwwtfddt|| D\} } t| rr|t| | S)z Helper to cross-check fake/real output properties & values, and create new fake vals if mismatched, but at the kernel level. Means this handles pytree outputs & checks aliasing. r)_check_alias_infoNrCrDcsttjjj_dSrc)rsetrpending_fresh_unbacked_symbols differencerJ)pending_unbackedrIrJrK_clear_pending_unbackeds   zXFakeTensorMode._maybe_infer_fake_kernel_from_pytree_out.._clear_pending_unbackedrrcstdjdS)Nz>Mismatched aliasing spec between fake kernel and real kernel: rrrJrrJrKrs zIFakeTensorMode._maybe_infer_fake_kernel_from_pytree_out..rcs t|Srcrrr[rIrJrKrs zGReal tensor propagation found an aliasing mismatch between fake output z and real output z , for func: cs$g|]\\}}}|||qSrJ)r)r _fake_path _fake_out _real_outrrJrKrs zKFakeTensorMode._maybe_infer_fake_kernel_from_pytree_out..rO)rrrrr rtree_flatten_with_path tree_flattenr_rmrrrrr-rrtree_unflatten)rIr[rrrrrr fake_paths_leaves fake_spec real_leavesr fake_leaves overridesrJ)rr[r rIrK(_maybe_infer_fake_kernel_from_pytree_outs^         z7FakeTensorMode._maybe_infer_fake_kernel_from_pytree_outc( s ddlm}t f\t}|r$ddD}td|tSfddD}tdd|Dp>td dDj j v} t j j jjuo^tdt jo^djjd k} | rc|rmtrs|s| std d|Ds}Jd fd dD} t| \} } | i| }t|tur|rt |}Wdn1swYj|ddS| rt dkrtdksJd tdturdSd}jrt j j jjkrd vr ddkrd}t j jjjkrd}\} tdd|D}tt j j!rt j"j#j$vrt j"j%j$vr|rt|dkrs|st j&jurfddD} t| \} } t| i| }Wdn 1sywYt'|}dd|D}tfdd|D}|rt(tfdd|S|D]})|qt\ tt j j*r|vrj+durt,j-nj+j.},||i WdWdS1swYWdn 1swY/| d<fdd dd!l0m1m2t3 j4rtd"d|Drtfd#dDst5d$ fd%dD}t|\ t67}|sft68|}z  i Wnt9y}z t5d&|WYd}~nd}~ww|s|:t6;j<nj4rt5d'|j+rj+j=ndd= f d)d* }rt>?}|dur||gRi Sdd+l@mA}|vrZBsZrjCvsZdd,l@mD}|vr/stEr/td-d|Dr/||i WdS1s*wY jFi }|turJ||WdSWdn 1sUwYd.jGjHvrtId/rtJs|jKi WdS1swYt jLjMjN}|dur|jOvr|jPgRi Sj4rՈ urt67sՈj+durt6QstR}tSd0fd1dd2||St jTjUjVWHjXjY} | rozLt jTjXZt jTjX[fd3d.| i }||WdWdWS1swYWdn 1s*wYWn>t\yn}!z1j4rh urht67shj+durhtR}tSd0fd4dd2||WYd}!~!S|!d}!~!wwt]D]!\}"}#|"r|#gRi }$|$tur||$Sqq d>d?fd8d9 }%t^s|%}&||&Szt_i }Wdn 1swYWn%t`y}'z |%|'WYd}'~'Sd}'~'wtayt5bd:w|jc| ?dd;S)@NrrxcSsg|] }t|rt|qSrJ)_check_for_subclass_argrrrJrJrKr s z1FakeTensorMode._dispatch_impl..z,FakeTensorMode unrecognized subclass(es): %scsg|] }|r|qSrJ)r<rrHrJrKr rcss|]}|jVqdSrc)r)rrrJrJrKr s z0FakeTensorMode._dispatch_impl..css|]}t|tVqdSrc)rrrrLrJrJrKr rr0cs|]}|jduVqdSrcrrrJrJrKr' rz. should not have fake inputs without constantsc g|] }|r |jn|qSrJr<rrrHrJrKr* Trr2rFrrcsrrcrrrgrJrJrKrc rcrrJr rrHrJrKrn r!cSsg|] }t|tr|qSrJrrrJrJrKry rc3rrc)may_turn_constrrHrJrKrz rcsj|ddS)NTr")r*r) converterrIrJrKr sz/FakeTensorMode._dispatch_impl..rrArC1Optional[Union[T, Tensor, torch._C.ScriptObject]]cs`t|tr|jSt|tr&jdusJ|j|jjjj jj St|t r.|j S|Src) rrrqr%rrgr,rxreplacerrrreal_objrrHrJrKmaybe_to_real_tensor s   z;FakeTensorMode._dispatch_impl..maybe_to_real_tensor)compute_unbacked_bindingsfree_unbacked_symbolscsrrc)rqr#rJrJrKr rc3sF|]}t|to|ojduotfddDVqdS)Nc3s|] }|jjvVqdSrc)rrrsrHrJrKr z:FakeTensorMode._dispatch_impl...)rr%rrr)r+rIsymsrJrKr s  zpropagate_real_tensors %scg|]}|qSrJrJr)r)rJrKr rz9real-tensor fallback failed for %s: %s; silently ignoringz,SKIPPED propagate_real_tensors %s(%s, %s) %srcsddltdd  fd d uritjjjs+ f f| n f f| }t|t s[t t s[t |t kr[t t t |t t  nt |  j|d d |S)Nrzmaybe_propagate_real_tensors %srrreal_trrCrDcs2t|trDtdt|t|||_t||D] \}}||qt||D] \}}||q/| | dSt|t r|rt|j j j rgjdus\Jj|j j |dSt|j j }jrt|jj r|jdkrjdusJj|t|dSdSdSdSdSdS)Nz%maybe_propagate_real_tensors %s -> %sr2)rrr"rQr#rqrrrrr%rgrSymbolrset_unbacked_var_to_valrlhsrhsr)rr1r-real_s)r+gorIrrJrKr7 s0     zOFakeTensorMode._dispatch_impl..maybe_propagate_real_tensors..goT)peek)rrr1rrCrD)rr"rQrmrrrrrrrr.rrrr)r) rr*r+r[rnil real_args real_kwargsrrI)r7rrKmaybe_propagate_real_tensors s@       zCFakeTensorMode._dispatch_impl..maybe_propagate_real_tensors) meta_tablercss|]}t| VqdSrc)rr#rJrJrKrJ rzprims::prim_meta_implmissing_fake_kernelc dtiSNrrrJrrJrKry rcsSrcrJrJ)ctxrJrKr scr@rArrJrrJrKr rBerrorOptional[RuntimeError]rcsHtjjr dSsst|durt}t|Src)rmrrcan_generate_trivial_fake_implcan_run_unsafe_fallbackr]run_fallback_kernel)rD) args_specrr[has_symbolic_sizesrIrJrKmaybe_run_unsafe_fallback sz@FakeTensorMode._dispatch_impl..maybe_run_unsafe_fallbackz*failed while attempting to run meta for %sr3)rrArCr&)rrArCrArc)rDrErCr)drryrr_check_for_subclassrrQrrr(rrmrr_to_copyrrrrrrJrrr$r'cloner*rr?prims device_put%validate_and_convert_non_fake_tensorsr[r<rnondeterministic_seededrr_nested_tensor_from_tensor_listr tree_map_onlyrr\rrrignore_fresh_unbacked_symbolsinvalidate_written_to_constantsr r*r+rr'r" library_utilsrMutationCheckerZeroDivisionErrorcheckcheck_aliasing_constraintrSrget_fast_op_implsrrr=cpp_meta_supports_symint_view_fake_tensor_impl_opsrr decomposerrHr>stride_incorrect_opr>r,r_custom_ops_profiledatageneric_fake_kernelhas_fake_kernel"inferred_fake_kernel_from_real_outrrsimple_registry singletonfind fake_implkernel FakeImplCtxset_ctx_getterrop_implementations_checkshas_metarDrvrrF+wrap_meta_outputs_with_default_device_logic)(rIr[rrrryhas_unrecognized_typesrflat_arg_fake_tensors is_lift_func!device_conversion_skip_const_propconst_flat_args const_args const_kwargsr|avoiding_device_init all_constantflat_outflat_out_tensorsr#maybe_ignore_fresh_unbacked_symbolsreal_flat_argsrmutation_checkerrr< fast_implr=rrkprofilesrmaybe_fake_implrgrun_impl_checkop_impl op_impl_outrKfallbacknot_implemented_errorrJ)rrIr*r%rCrr+r[rJrr)r9r:r;rrIr/rKr^s     *        P    I        "  "         R           zFakeTensorMode._dispatch_impl debugprimsrOrxlavision torchtext torchaudio quantizedcCs"|jsdS|j|jvp|dkS)NFz fbgemm::gmm)r  namespace+_can_run_unsafe_fallback_allowed_namespacesrHrIr[rJrJrKrG s   z&FakeTensorMode.can_run_unsafe_fallbackr%rrrIr/%tuple[list[object], list[FakeTensor]]cs6gd fdd fddD}|fS) z Checks if the list of tensors are fake tensors. If not, try to convert them to fake tensors. Returns the original args, kwargs, and a flattened list of (args, kwargs) that are fake tensors. rrArCUnion[T, FakeTensor]cst|ts|S|s`tdr+tjjjvr+t \}}t dt ||t j dur3jnt j }|sYt|trF|jurFt dt \}}t dt |||}n|}||S)NrzECan't call metadata mutating ops on non-Fake Tensor inputs. Found in zMixing fake modes NYIzuPlease convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in )rrr<r>rmrrrrrAssertionErrorrfake_tensor_tlsrbr!rrur*r)rrrr!r|)rIr%rqrr[rIrJrKvalidate s2     zFFakeTensorMode.validate_and_convert_non_fake_tensors..validatecr0rJrJr)rrJrKr rzHFakeTensorMode.validate_and_convert_non_fake_tensors..N)rrArCrrJ)rIr[r%rrIvalidated_argsrJ)rIr%rqrr[rIrrKrQ s  z4FakeTensorMode.validate_and_convert_non_fake_tensorsrkrr+r,cs2jdddfdd }t||S) NFrgrArCrcsttsSdurt\}|r-tjkfddtt SdurBr9 S p@SS)NcsdjdS)Nz-FakeTensor is wrapped to wrong device, found z , expected r3rJ)rrgrJrKr: zZFakeTensorMode.wrap_meta_outputs_with_default_device_logic..wrap..) rrrrr<rm_checkrrrAr*r1)rgr<rr%rrr[rrIrgrKwrap) s*       zHFakeTensorMode.wrap_meta_outputs_with_default_device_logic..wraprgrArCr)r(r-)rIrkr[rrrrJrrKro s  !z:FakeTensorMode.wrap_meta_outputs_with_default_device_logicrrrrcCsddl}ddlm}|dur|j}|jsJd|jd7_|||d}|jjd}|j dus5J|j j |j j ||d||d}|S)Nr) NestedIntNodez1should only called while FakeTensorMode is activer2intermediate_offsets_or_lengths)valr)symrr) r4!torch.nested._internal.nested_intrrr0rr,rEphemeralSourcerr" create_symbol)rIrrmrrsrcretrJrJrKrL s$ z)FakeTensorMode.create_symbolic_nested_intcCstjj|jvr dS||jvSrg)rmr view_copyr_cpp_meta_supports_symintrrJrJrKr]x s z'FakeTensorMode.cpp_meta_supports_symintrcCs0|tkot| o|| o|jjdk Srs)numelCONSTANT_NUMEL_LIMITrr<rrr;rJrJrKr$ s   zFakeTensorMode.may_turn_constrqSequence[FakeTensor]c Cstdd|D}t|}|rI|rKt|||dd\}}|D]+\} } | dks.|| r0| nd} || rH|| rH| jdurH|j | jq!dSdSdS)Ncsrrcrr#rJrJrKr rzAFakeTensorMode.invalidate_written_to_constants..TrzinputrI) rr is_mutabler!re has_argumentr<rr(r) rIr[rqrr any_constant schema_inforrkrrJrJrKrV s(   z.FakeTensorMode.invalidate_written_to_constants)rrrrrrrrrrcCsD|j}|dur |j}|r|dusJdd}|jj||||||dS)Nz2cannot set both static_shapes and symbolic_context)rrrr)rrr(r*)rIrrrrrrrJrJrKr s   zFakeTensorMode.from_tensor) r rr!rrrrrarrrCrDrO)rrrCr:)rCr)rCrUr)rCr )rLrMrNrOrPrQrCrD)rCr) rr3r[r<rrrrrCr)r[r<rrrrrCrD) rrrrrr3rrrCrD)rr3rrr[r<rrrrrjrrCrD)rr3rrr[r<rrrrrjrrCr)rr3rrr[r<rrrrrjrrCr ) rr3rirrrr[r<rrrCr) rr3rir rrr[r<rrrCr) rjrr[r<rrrrrrrCrD) r[r<rr*rrrrrCr) r[r<rrrrrrrrrCr) r[r<rrrrrrrCrr[r<rCr) r[r<r%rrrrIr/rCr) rkrr[r<rrrr+rCr,)rrrCrr) r[r<rqrrrrrrCrD) rrrrarrrrrrrCr)LrPrQrRrrXrrrrrrir=rrrLr9r<r2r?rCr0r rrIrRrrVrWrXrkrZrerrrrfrrcrdrDrrr^rjrrGrQrorrr9rrras_strided_scatter as_strided as_strided_r:detach view_as_realview_as_complexrsource_Storage_storage_offset(_sparse_coo_tensor_with_dims_and_tensorsrviewrr^r] lift_freshlift_fresh_copyrr$rVrrrJrJrrKrvs        _       a / F G , J H I 2 ! n Lj 11    rr3rjrcCs<ddlm}||D]}|jD] }||jvrdSqq dS)Nr)_iterate_exprsTF)r rrrq)rrjrr-r)rJrJrKr s    rrrrIorig_not_implemented_exceptionrc stjj|jvr it(d fdd fdd|D}t||\}}||i|}Wdn1s9wYt|D]}t|t rVt |sV | j qCd fd d } t| |S)NrgrArCUnion[T, Tensor]csB|rtj||jd}|jr|||t|<|S|S)Nr3)r<rm zeros_likeror _coalesced_rr#)rgr|)ru inp_implsrJrKto_real_tensor s  z+run_fallback_kernel..to_real_tensorcr0rJrJr)rrJrKr rz'run_fallback_kernel..rcsdt|vrt|trt|s|jvrt|tr0t|vr)t|Sj|S|Src)r#rrrr_cdatar(r*r)rurrstoragesrJrKmap_out s    z$run_fallback_kernel..map_out)rgrArCrr)rmrrrr'rrr rrraddrrr-) rur[rrIrrrrkrgrrJ)rurrrrrKrH s"    rHrrrri_DispatchCacheEntrycCs||||<dSrc)rrrrirJrJrKra s racCs |||<dSrcrJrrJrJrKrb s rbc@s&eZdZdddZ  ddddZdS) FakeCopyModerurvrCrDcCrWrc)ru)rIrurJrJrKrL rdzFakeCopyMode.__init__rJNr[r<rrrrrOptional[Mapping[str, object]]rcCs |r|ni}|tjjjkr&t|dtsJ||jj|dddfi|S|tjkrgt |dkr7t |dks9Jt t|d}t t t t f|d}t||vrW|t|S|jj|dd}||t|<|Stj||i|WdS1s~wYdS)NrT)rr2)rmrn TensorBaserNrrrur __deepcopy__rrrhrrr#DisableTorchFunctionSubclass)rIr[rrrrmemor|rJrJrK__torch_function__# s(       $zFakeCopyMode.__torch_function__)rurvrCrD)rJN) r[r<rrrrrrrCr)rPrQrRrLrrJrJrJrKr s  rrcCs>t|dkr t|dtsJ|djjrtdS|djS)Nr2rr0)rrrrur=rmrrorrJrJrK_device_handlerB s   rcCstdd|DS)Ncsrrc)rrrJrJrKr[ rz&_check_for_subclass..)r)rrJrJrKrLZ srLcCs2t|t ot|tot|tuot|tjjuSrc)rrrrrmrrrrJrJrKr^ s  rcC tddtt|dDS)Ncsrrcrr,rJrJrKri  ..r)rrrrrrJrJrKri rcCr)Ncsrrcrr,rJrJrKrl rrr)rrrrrrJrJrKrl rcCsttt|dSr)rrrrrrJrJrKro s)_device_not_kwarg_ops_is_tensor_constructor_like_tensor_constructorscontains_tensor_typesr\rnrmr`cCs|tjvr tj|dSdSrc)rvrrrlrJrJrKru s rucCstdtdtjtdtjtj}|rBtdtdd|D}t|ddd D]\}}td |d |d |q0dSdS) NzFakeTensor cache stats:z cache_hits: %sz cache_misses: %sz cache_bypasses:csrrc)r)rrrJrJrKr rz#dump_cache_stats..cSs |d SrErJ)rrJrJrKr s z"dump_cache_stats..rlz %-*s %sr2r) r"inforvrrrmaxsortedre)rwidthrrrJrJrKdump_cache_stats s  rrrtorch._ops.OpOverloadr torch.Tensorc sdfdd }|dkr|d|fd d t|D}d g|}d d t|D}|jd ddd}|}|D]'\} } | |krY|d|jd|d||| <||j| }||| }qCtj|||j |j |j dWdS1swYdS)NrVrUrCrDcstdjd|d)NzQpropagate_real_tensors: we cannot infer a Fake kernel (meta kernel) for operator z because z>. Please use torch.library.register_fake to add a Fake kernel.)rrS)rV)rrJrK unsupported sz1_infer_fake_from_real_tensor..unsupportedrz'a return has a non-zero storage offset csg|] }tjjjqSrJ)rmrri allocate_sizer)rr)rrJrKr sz0_infer_fake_from_real_tensor..rEcSsg|]\}}||fqSrJrJ)rrr-rJrJrKr cSs|d|d fS)Nrr2rJrrJrJrKr rz._infer_fake_from_real_tensor..rlr2z(a return was not dense in memory (sizes z strides ))rrr)rVrUrCrD) rrrrrsortrrmrrrr) rrrr fake_shape fake_stridesrexpected fake_strider-rrJrrrKr s<     $rrcs^jdusJt|\}}tdd|Dstdjfdd|D}t||S)Ncss|] }t|tjVqdSrc)rrmrrrJrJrKr r.z5inferred_fake_kernel_from_real_out..zPpropagate_real_tensors: we don't support operators that return non-Tensors. Got csg|]}t|qSrJrrrrJrKr rz6inferred_fake_kernel_from_real_out..)rrrrrrr)rrr real_flat_outspec fake_flat_outrJrrKre s re)rerArCrf)rCrl)rurvrCrw)r{rr|r}rCr})rrrCr)rrrCr)r[r<rCrr)rrrr,rCr)rrrCr)rr+rCrD)rrrCr)rr3rjrrCr) rurvr[r<rrrIr,rrrCr)rrrrrirrCrD)rrrCr+)rrrCr)rrrCr)rrrCrDrO)rrvrrrrrCr)rrvrrrrrCr) __future__ratexitrrrsrrrO threadingr1rtypingr collectionsrrrrrrr r r r typing_extensionsr rrrmtorch._library.utilsrrrWrrrrtorch._C._functorchrr"torch._library.fake_class_registryrtorch._library.fake_profilertorch._loggingrtorch._prims_commonrtorch._subclasses.meta_utilsrrrrr torch._utilsrtorch.fx.immutable_collectionsr torch.fx.operator_schemasr! torch.multiprocessing.reductionsr"torch.overridesr# torch.typesr$r%torch.utils._backport_slotsr&torch.utils._mode_utilsr'torch.utils._python_dispatchr(r)torch.utils._pytreer*r+r,r-r.r/torch.utils._statsr0torch.utils._tracebackr1_fake_tensor_utilsr3r4r5collections.abcr6r7r8r9r: torch._guardsr; torch._opsr<r r=r> getLoggerrPr"_logginggetArtifactLoggerr]r ValueErrorrgrUrDimList_pytreerrAr[rrrrGrBrrTrZr\r]r^r_localr`rrjcontextmanagerrtrzrrrrrrrrrr;rDrJrKrRrrrrrrrr r rrrrrvr_StoragePointerrrHrarbrrrLrrrrrrrrrrrprofiler_record_function_exit_RecordFunctionrtorch._subclasses.fake_implsrrrrr\rnrmr`ruregisterrrrerJrJrJrKsV   (                              i   Aq -&     @  #       (   <