o ĎiM@sdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl m Z e ZGdd d e jZd d Zd d ZddZddZddZddZddZdS)z;Bridge from event multiplexer storage to generic data APIs.N)errors) summary_pb2)provider) tb_logging) tensor_utilc@seZdZddZddZddZddZd d Zd d Zd dZ d.ddZ d.ddZ d.ddZ d.ddddZ d.dddddZ d.ddddZ d.ddddZ d.dddd d!Zd"d#Zd$d%Zd&d'Z d.ddd(d)Z d.dddd*d+Zd.d,d-ZdS)/MultiplexerDataProvidercCs||_||_dS)a6Trivial initializer. Args: multiplexer: A `plugin_event_multiplexer.EventMultiplexer` (note: not a boring old `event_multiplexer.EventMultiplexer`). logdir: The log directory from which data is being read. Only used cosmetically. Should be a `str`. N) _multiplexer_logdir)self multiplexerlogdirr n/home/jeff/fluffinator/venv/lib/python3.10/site-packages/tensorboard/backend/event_processing/data_provider.py__init__!s z MultiplexerDataProvider.__init__cCs d|jS)Nz"MultiplexerDataProvider(logdir=%r))r r r r r__str__-s zMultiplexerDataProvider.__str__cCs t|jdkrtd|fdS)NRequestContextz%ctx must be a RequestContext; got: %r)type__name__ TypeError)r ctxr r r_validate_context0sz)MultiplexerDataProvider._validate_contextcCs$t|tstdtt||fdS)Nz(experiment_id must be %r, but got %r: %r) isinstancestrrr)r experiment_idr r r_validate_experiment_id4s  z/MultiplexerDataProvider._validate_experiment_idcCs2|durtdt|trdStdt||f)Nz#`downsample` required but not givenz+`downsample` must be an int, but got %r: %r)rrintr)r downsampler r r_validate_downsample>s  z,MultiplexerDataProvider._validate_downsamplecCs8|j}|dur ||vr dS|j}|dur||vrdSdS)NFTrunstags)r run_tag_filterruntagr r!r r r _test_run_tagHsz%MultiplexerDataProvider._test_run_tagc Cs6z|j|WSty}zWYd}~dSd}~wwN)rFirstEventTimestamp ValueError)r run_nameer r r_get_first_event_timestampQs z2MultiplexerDataProvider._get_first_event_timestampNcCs"||||tj|jdS)N) data_location)rrrExperimentMetadatar r rrr r rexperiment_metadataWs  z+MultiplexerDataProvider.experiment_metadatacCs|||||jSr&)rrr ActivePluginsr.r r r list_plugins\s   z$MultiplexerDataProvider.list_pluginscs,||fddjDS)Ncs"g|] }tj|||dqS))run_idr) start_time)rRunr+).0r#rr r isz5MultiplexerDataProvider.list_runs..)rrrRunsr.r rr list_runsfs   z!MultiplexerDataProvider.list_runs)r"cC2|||||||tj}|tj|Sr&)rr_indexrDATA_CLASS_SCALAR_listrScalarTimeSeriesr rr plugin_namer"indexr r r list_scalarsr  z$MultiplexerDataProvider.list_scalars)rr"cC<|||||||||tj}|t||Sr&)rrrr:rr;_read_convert_scalar_eventr rrr?rr"r@r r r read_scalars|  z$MultiplexerDataProvider.read_scalarsc Cs||||||||tj}tt}|D] \}}|D]\} } |j || } | r:t | d||| <q#q|S)N) rrr:rr; collections defaultdictdictitemsrTensorsrE) r rrr?r"r@run_tag_to_last_scalar_datumr# tags_for_runr$metadataeventsr r rread_last_scalarss     z)MultiplexerDataProvider.read_last_scalarscCr9r&)rrr:rDATA_CLASS_TENSORr<rTensorTimeSeriesr>r r r list_tensorsrBz$MultiplexerDataProvider.list_tensorscCrCr&)rrrr:rrTrD_convert_tensor_eventrFr r r read_tensorsrHz$MultiplexerDataProvider.read_tensorsc Cs|dur tjddd}|j}|j}|rCt|dkrC|rCt|dkrC|\}|\}z |j||}Wn ty;iYSw|||ii} n|j} i} | D]6\}} |dur[||vr[qNi} | D]"\}}|durn||vrnqa|j |krtqa|j j |kr{qa| | |<|| |<qaqN| S)aList time series and metadata matching the given filters. This is like `_list`, but doesn't traverse `Tensors(...)` to compute metadata that's not always needed. Args: plugin_name: A string plugin name filter (required). run_tag_filter: An `provider.RunTagFilter`, or `None`. data_class_filter: A `summary_pb2.DataClass` filter (required). Returns: A nested dict `d` such that `d[run][tag]` is a `SummaryMetadata` proto. Nr) r RunTagFilterr r!lenrSummaryMetadataKeyErrorAllSummaryMetadatarM data_class plugin_datar?) r r?r"data_class_filterr r!r#r$rQ all_metadataresulttag_to_metadataresult_for_runr r rr:s<       zMultiplexerDataProvider._indexc Csi}|D]O\}}i}|||<|D]@\}}d} d} |j||D]} | dus.| | jkr1| j} | dus:| | jkr=| j} q#|j||}|| | |jj|j|j d||<qq|S)acHelper to list scalar or tensor time series. Args: construct_time_series: `ScalarTimeSeries` or `TensorTimeSeries`. index: The result of `self._index(...)`. Returns: A list of objects of type given by `construct_time_series`, suitable to be returned from `list_scalars` or `list_tensors`. N)max_step max_wall_timeplugin_content description display_name) rMrrNstep wall_timer\r`contentsummary_descriptionrj) r construct_time_seriesr@rcr#rdrer$summary_metadatarfrgeventr r rr<s.  zMultiplexerDataProvider._listc sfi}|D]*\}}i}|||<|D]\}} |j||} fdd| D} t| |||<qq|S)aHelper to read scalar or tensor data from the multiplexer. Args: convert_event: Takes `plugin_event_accumulator.TensorEvent` to either `provider.ScalarDatum` or `provider.TensorDatum`. index: The result of `self._index(...)`. downsample: Non-negative `int`; how many samples to return per time series. Returns: A dict of dicts of values returned by `convert_event` calls, suitable to be returned from `read_scalars` or `read_tensors`. csg|]}|qSr r r5r* convert_eventr rr6&z1MultiplexerDataProvider._read..)rMrrN _downsample) r rtr@rrcr#rPrer$rQrRdatar rsrrDszMultiplexerDataProvider._readc Cs|||||||tj}i}|D][\}}i} | ||<|D]L\} } d} d} d}|j|| D])}| dusB| |jkrE|j} | dusN| |j krQ|j } t |j }|dus^||kr`|}q7t j | | || jj| j| jd| | <q&q|S)N)rfrg max_lengthrhrirj)rrr:rDATA_CLASS_BLOB_SEQUENCErMrrNrkrl _tensor_size tensor_protorBlobSequenceTimeSeriesr`rmrnrj)r rrr?r"r@rcr#rdrer$rQrfrgrxrqlengthr r rlist_blob_sequences*s@    z+MultiplexerDataProvider.list_blob_sequencesc Cs|||||||||tj}i}|D]@\}} i} | ||<| D]3} |j|| } i} | D]}|j | vr>q6t |||| || |j <q6ddt | D}t ||| | <q)q|S)NcSsg|]\}}|qSr r )r5rkdatumr r rr6hruz?MultiplexerDataProvider.read_blob_sequences..) rrrr:rryrMrrNrk_convert_blob_sequence_eventsortedrv)r rrr?rr"r@rcr#r!rer$rR data_by_steprqrwr r rread_blob_sequencesLs.      z+MultiplexerDataProvider.read_blob_sequencesc s||t|\}}}}}|j||}|jtjkr!t||j ||} t fdd| Dd} | s?td|ft | j } | |S)Nc3s|] }|jkr|VqdSr&rkrrrr r |sz4MultiplexerDataProvider.read_blob..z%s: no such step %r)r_decode_blob_keyrr\r_rryr NotFoundErrorrNnextr make_ndarrayr{) r rblob_keyunused_experiment_idr?r#r$r@rp tensor_events matching_steptensorr rr read_blobls$    z!MultiplexerDataProvider.read_blobr&)r __module__ __qualname__rrrrrr%r+r/r1r8rArGrSrVrXr:r<rDr~rrr r r rr s\              1! $  rc Cs>tj||||||fdd}|d}t|}|ddS)aGenerate a blob key: a short, URL-safe string identifying a blob. A blob can be located using a set of integer and string fields; here we serialize these to allow passing the data through a URL. Specifically, we 1) construct a tuple of the arguments in order; 2) represent that as an ascii-encoded JSON string (without whitespace); and 3) take the URL-safe base64 encoding of that, with no padding. For example: 1) Tuple: ("some_id", "graphs", "train", "graph_def", 2, 0) 2) JSON: ["some_id","graphs","train","graph_def",2,0] 3) base64: WyJzb21lX2lkIiwiZ3JhcGhzIiwidHJhaW4iLCJncmFwaF9kZWYiLDIsMF0K Args: experiment_id: a string ID identifying an experiment. plugin_name: string run: string tag: string step: int index: int Returns: A URL-safe base64-encoded string representing the provided arguments. ),:) separatorsascii=)jsondumpsencodebase64urlsafe_b64encodedecoderstrip) rr?r#r$rkr@ stringified bytesifiedencodedr r r_encode_blob_keys  rc Cs>t|d}|d}t|\}}}}}}||||||fS)a'Decode a blob key produced by `_encode_blob_key` into component fields. Args: key: a blob key, as generated by `_encode_blob_key`. Returns: A tuple of `(experiment_id, plugin_name, run, tag, step, index)`, with types matching the arguments of `_encode_blob_key`. z==r)rurlsafe_b64decoderrloads) keydecodedrrr?r#r$rkr@r r rrs  rcCs tj|j|jt|jdS)zHelper for `read_scalars`.)rkrlvalue)r ScalarDatumrkrlrrr{itemrqr r rrEs rEcCstj|j|jt|jdS)zHelper for `read_tensors`.)rkrlnumpy)r TensorDatumrkrlrrr{rr r rrWs  rWcs@tj}tfddt|D}tjjj|dS)z!Helper for `read_blob_sequences`.c 3s*|]}ttj|VqdSr&)r BlobReferencerrk)r5idxrqrr?r#r$r rrs  z/_convert_blob_sequence_event..)rlrkvalues)rzr{tuplerangerBlobSequenceDatumrlrk)rr?r#r$rq num_blobsrr rrrs   rcCs d}|jjD]}||j9}q|S)zCompute the number of elements in a tensor. This does not deserialize the full tensor contents. Args: tensor_proto: A `tensorboard.compat.proto.tensor_pb2.TensorProto`. Returns: A non-negative `int`. rY) tensor_shapedimsize)r{rcrr r rrzs  rzcsn|tkr tS|dkrgStdttd|d}||tdg7}fdd|DS)avDownsample `xs` to at most `k` elements. If `k` is larger than `xs`, then the contents of `xs` itself will be returned. If `k` is smaller than `xs`, the last element of `xs` will always be included (unless `k` is `0`) and the preceding elements will be selected uniformly at random. This differs from `random.sample` in that it returns a subsequence (i.e., order is preserved) and that it permits `k > len(xs)`. The random number generator will always be `random.Random(0)`, so this function is deterministic (within a Python process). Args: xs: A sequence (`collections.abc.Sequence`). k: A non-negative integer. Returns: A new list whose elements are a subsequence of `xs` of length `min(k, len(xs))` and that is guaranteed to include the last element of `xs`, uniformly selected among such subsequences. rrYcsg|]}|qSr r )r5ixsr rr6ruz_downsample..)r[listrandomRandomsamplersort)rkindicesr rrrvs "rv)__doc__rrJrr tensorboardrtensorboard.compat.protortensorboard.datartensorboard.utilrr get_loggerlogger DataProviderrrrrErWrrzrvr r r rs*     f)