o ĎiXm@sdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z m Z mZmZmZmZmZmZddlZddlmZdd lmZmZmZmZmZmZmZmZdd l m!Z!m"Z"d d l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*e*d dZ+eddGdddZ,e-dZ.hdZ/hdZ0hdZ1e-dZ2dddddZ3iZ4e5e1D]Z6e6e4e6<e6de4e6d<e6de4e6d<qe de4e7e8d Z9eddGd!d"d"Z:eeeefZ;eeed#eeeZHd?e7d$eIee;geIe d@ffeAffdAdBZJd)edCeKe7d$e,fdDdEZLe+jMgdFdGddHde$jNdHddfdIe ee7ejOdJdfdKe eAejOdLdfdMe eeKe7ejOdNdOdPdfdQe e$ejOdRdfdSe eAejOdTdUdVdfdWe ee9ejOdXdfdYe eePejOdZdfd$dfd[d\ZQe+jMgd]dG  H Hd{dCe eKe7ejRd^dfdIe ee7ejOdJdfd_e eAejOd`dadbdfdce eAejOdddfd$df dedfZSe+jMdgdhgdG  H Hd{dIe ee7ejOdJdfd_e eAejOd`dadbdfdce eAejOdddfd$dfdidjZTe+jMgdkdGe&jUddddHdHdfdle%dme&dne'dIe ee7ejOdodfdpe ee7ejOdqdfdre eAejOdsdtdfdue eAejOdvdwdfdxe(d$dfdydzZVdS)|zHContains the 'hf cache' command group with cache management subcommands.N) defaultdict) dataclass)Enum) AnnotatedAnyCallableDictListMappingOptionalTuple)CLIError)ANSICachedRepoInfoCachedRevisionInfo CacheNotFound HFCacheInfo _format_sizescan_cache_dirtabulate)parse_duration parse_size) OutputFormat RepoIdArg RepoTypeOpt RevisionOptTokenOpt get_hf_api typer_factoryzManage local cache directory.)helpT)frozenc@s>eZdZUeeed<eeeefed<e edfed<dS)_DeletionResolution revisionsselected.missingN) __name__ __module__ __qualname__ frozensetstr__annotations__dictrrtupler/r/U/home/jeff/fluffinator/venv/lib/python3.10/site-packages/huggingface_hub/cli/cache.pyr#:s r#zA^(?P[a-zA-Z_]+)\s*(?P==|!=|>=|<=|>|<|=)\s*(?P.+)$><=>!=<=>=>refssizetypeaccessedmodified>namer8r:r;z.^(?P[a-zA-Z_]+)(?::(?Pasc|desc))?$descasc)r:r;r8r<z:asc_ascz:desc_desc SortOptions)r9modulec@s*eZdZUdZeed<eed<eed<dS)CacheDeletionCountsz>Simple counters summarizing cache deletions for CLI messaging. repo_countpartial_revision_counttotal_revision_countN)r'r(r)__doc__intr,r/r/r/r0rCYs  rCselected_by_reporeturncCsdd}d}d}|D]\}}|t|7}t|t|jkr'|d7}|t|7}q ||}t|||S)z(Summarize deletions across repositories.rr)itemslenr$rC)rIrDtotal_revisionsrevisions_in_full_reposrepor$rEr/r/r0summarize_deletionsfs   rPc Cst|dddD]M}|jd|j}t||ddd}t|t|jkr1td|dq td|d|D]}d t|jpFd }td |j d |d |j q;q dS)zBPretty-print selected cache revisions during confirmation prompts.cS|j|jfSN repo_typerepo_idlowerrOr/r/r0zz0print_cache_selected_revisions..key/cS|jSrR commit_hashrevr/r/r0rX| - z (entire repo): z (detached)z z [z] N) sortedkeysrTrUrLr$printjoinr7r_size_on_disk_str)rIrOrepo_keyr$revisionr7r/r/r0print_cache_selected_revisionsxs rm hf_cache_infocCsLi}i}|jD]}|j}|||<|jD] }||f||j<qq||fS)zPCreate lookup tables so CLI commands can resolve repo ids and revisions quickly.)reposcache_idrVr$r_)rn repo_lookuprevision_lookuprOrkrlr/r/r0build_cache_indexs   rsinclude_revisionscCsg}i}t|jddd}|D]+}tdd|jD||<|r3t|jdddD] }|||fq(q||dfq|rI|jddd||fS|jd dd||fS) z;Flatten cache metadata into rows consumed by `hf cache ls`.cSrQrRrSrWr/r/r0rXrYz'collect_cache_entries..rZcSsh|] }|jD]}|qqSr/)r7).0rlrefr/r/r0 sz(collect_cache_entries..cSr]rRr^r`r/r/r0rXrbNcSs&|dj|ddur|djfSdfS)Nrr)rpr_entryr/r/r0rXs cSs |djS)Nr)rpryr/r/r0rXs )rfror*r$appendsort)rnrtentries repo_refs_map sorted_reposrOrlr/r/r0collect_cache_entriess"rexprr~cst|}|std|d|d|d|dtvr8tdd|dttd tvrLtd d |dttd d kr[t fd dSdvrzt dt dt t dtdtffdd }|Sdkrdkrtdddt dt t dtdtffdd }|Sdkrtdd dt dt t dtdtffdd }|S)zqConvert a `hf cache ls` filter expression into the yes/no test we apply to each cache entry before displaying it.zInvalid filter expression: 'z'.r[opvaluezUnsupported operator 'z ' in filter ''. Must be one of .zUnsupported filter key '' in 'r8cs"t|dur |jS|jSrR)_compare_numeric size_on_diskrOrl_)rsize_thresholdr/r0rXs z&compile_cache_filter..>r:r;rOrlnowrJcs>dkr|jn |dur|jn|j}|durdSt||S)Nr:F) last_accessed last_modifiedr)rOrlr timestamp)r[rsecondsr/r0 _time_filtersz*compile_cache_filter.._time_filterr9r2z/Only '=' is supported for 'type' filters. Got 'rcs|jkSrR)rTrVr)expectedr/r0 _type_filtersz*compile_cache_filter.._type_filterz.Only '=' is supported for 'refs' filters. Got cs2|dur|jn|t}dd|DvS)NcSsg|]}|qSr/)rV)rurvr/r/r0 z>compile_cache_filter.._refs_filter..)r7getr*rV)rOrlrr7)r~ value_rawr/r0 _refs_filtersz*compile_cache_filter.._refs_filter)_FILTER_PATTERNmatchstrip ValueErrorgrouprV_ALLOWED_OPERATORSlist _FILTER_KEYSrrrr rfloatbool)rr~rrrrr/)rr[rr~rrrr0compile_cache_filters6    & "$rr}c Csg}|D]=\}}|r'|durq|j|j|jt|j|j|j|jt|j d}n|j|j|j|j|jt| |t d}| |q|S)zGNormalize cache entries into serializable records for JSON/CSV exports.N)rUrTrl snapshot_pathrrrr7)rUrTrrrr7) rUrTr_r+rrrrrfr7rr*r{)r}rtr~payloadrOrlrecordr/r/r0_build_cache_export_payloads.   rc s|s|rdnd}t|dS|rgd}dd|D}n gd}fdd|D}tt||d d d |D}t|}|rQtd d |D}tdd |D} ntdd |D}tdd |D} d|d|dt| d} tt| dS)zBRender cache entries as a table and show a human-readable summary.zNo cached revisions found.zNo cached repositories found.N)IDREVISIONSIZE LAST_MODIFIEDREFSc Ss@g|]\}}|dur|j|j|jd|jdt|jgqS)Nre)rpr_rjrjustlast_modified_strrirfr7)rurOrlr/r/r0rs z-print_cache_entries_table..)rr LAST_ACCESSEDrrc sDg|]\}}|j|jd|jpd|jdt|tgqS)rrxre) rprjrlast_accessed_strrrirfrr*rurOrr~r/r0r+s )headerscSsh|]\}}|qSr/r/rr/r/r0rw8rz,print_cache_entries_table..css |] \}}|durdVqdS)Nrr/rurrlr/r/r0 ;sz,print_cache_entries_table..css"|] \}}|dur|jVqdSrRrrr/r/r0r< css|]}t|jVqdSrR)rLr$rurOr/r/r0r>scs|]}|jVqdSrRrrr/r/r0r?z Found z repo(s) for a total of z revision(s) and z on disk.)rhrrLsumrrbold) r}rtr~messager table_rows unique_reposrDrevision_count total_sizesummaryr/rr0print_cache_entries_tables.   rcCs0t|||d}tj|tjddtjddS)z7Dump cache entries as JSON for scripting or automation.rtr~r)indent N)rjsondumpsysstdoutwrite)r}rtr~rr/r/r0print_cache_entries_jsonEsrleftrrightcCsT|durdS||k||k||k||k||k||kd}||vr&td|||S)z)Evaluate numeric comparisons for filters.NF)r2r4r3r1r6r5z)Unsupported numeric comparison operator: )r)rrr comparisonsr/r/r0rNs r sort_expr.cst|}|std|d|d|d}tvr3tdd|dttd|r7|nt}|d k}d t d t t d fffd d }||fS)zConvert a `hf cache ls` sort expression into a key function for sorting entries. Returns: A tuple of (key_function, reverse_flag) where reverse_flag indicates whether to sort in descending order (True) or ascending order (False). zInvalid sort expression: 'z5'. Expected format: 'key' or 'key:asc' or 'key:desc'.r[orderzUnsupported sort key 'rrrr=rzrJ.cs|\}}dkr|j}|fSdkr$|dur|j}|fS|j}|fSdkr8|jdur3|j}|fSd}|fSdkr]|durP|jdurK|j}|fSd}|fS|jdurX|jnd}|fStd)Nr<r8r:gr;zUnsupported sort key: )rprVrrrr)rzrOrlrrZr/r0 _sort_keyws. z%compile_cache_sort.._sort_key) _SORT_PATTERNrrrVrr _SORT_KEYSr_SORT_DEFAULT_ORDER CacheEntryr.r)rrexplicit_orderrreverserr/rZr0compile_cache_sortbs  rtargetscCst|\}}tt}t}g}|D]V}|}|sq|} td| rE|| } | dur3||q| \} } ||  | | | j q|| } | durT||q| j D]} ||  | | | j qWqdd| D}t t||t|dS)z8Resolve the deletion targets into a deletion resolution.z[0-9a-fA-F]{40}NcSsi|] \}}|t|qSr/)r*)rurOrevsr/r/r0 sz-_resolve_deletion_targets..r$r%r&)rsrsetrrVre fullmatchrr{addr_r$rKr#r*r.)rnrrqrrr%r$r& raw_targettargetloweredrrOrl matched_repofrozen_selectedr/r/r0_resolve_deletion_targetss@        r)z hf cache lszhf cache ls --revisionsz*hf cache ls --filter "size>1GB" --limit 20zhf cache ls --format json)examplesF cache_dirz9Cache directory to scan (defaults to Hugging Face cache).r$zCInclude revisions in the output instead of aggregated repositories.filterz-fz--filterzZFilter entries (e.g. 'size>1GB', 'type=model', 'accessed>7d'). Can be used multiple times.formatzOutput format.quietz-qz--quietz-Print only IDs (repo IDs or revision hashes).r|aSort entries by key. Supported keys: 'accessed', 'modified', 'name', 'size'. Append ':asc' or ':desc' to explicitly set the order (e.g., 'modified:asc'). Defaults: 'accessed', 'modified', 'size' default to 'desc' (newest/biggest first); 'name' defaults to 'asc' (alphabetical).limitzSLimit the number of results returned. Returns only the top N entries after sorting.c szt|}Wnty}z td|j|d}~ww|pg} t||d\} z fdd| D} WntyG}ztt||d}~wwt | D] fdd| D} qN|rzt |j \} } | j | | dWnty}ztt||d}~ww|dur|dkrtd |d | d|} |r| D]\}}t |dur|jn|jqdStjttjti}||| |d S) z&List cached repositories or revisions.Cache directory not found: N)rtcsg|]}t|qSr/)r)rurrr/r0r szls..cs$g|]}|d|dr|qS)rrr/)rurz)fnrr/r0rs$)r[rrz&Limit must be a positive integer, got rr)rrr rrrtyper BadParameterr+timerrr|rhr_rprtablerrr)rr$rrrr|rrnexcfiltersr} filter_fns sort_key_fnrrOrl formattersr/)rrr~r0lssJ<   r)zhf cache rm model/gpt2zhf cache rm z hf cache rm model/gpt2 --dry-runzhf cache rm model/gpt2 --yeszQOne or more repo IDs (e.g. model/bert-base-uncased) or revision hashes to delete.yesz-yz--yeszSkip confirmation prompt.dry_runz,Preview deletions without removing anything.c Cszt|}Wnty}z td|j|d}~wwt||}|jr5td|jD] }td|q+t|jdkrFtdt j dd|j t |j}t |j} g} | jra| | jd| jrm| | jd | sx| | jd d | } td | d |jd t|j|rtddS|st jdddstddS|t |j} td| jd| jd|jd dS)z(Remove cached repositories or revisions.rNz2Could not find the following targets in the cache:rcrzNothing to delete.codez repo(s)z revision(s)z and About to delete z totalling rDry run: no files were deleted.zProceed with deletion?F)defaultzDeletion cancelled.Deleted z repo(s) and z revision(s); freed )rrr rrr&rhrLr$rExitdelete_revisionsrfrPr%rDr{rErFriexpected_freed_size_strrmconfirmexecute) rrrrrnr resolutionrzstrategycounts summary_parts summary_textr/r/r0rm+sJ%        r zhf cache prunezhf cache prune --dry-runc Cs8zt|}Wnty}z td|j|d}~wwi}t}|jD]}tdd|jD}|s3q$|||<|dd|Dq$t |dkrNt ddSt t||dd } |j t | j} t|} t d | jd | jd t||r|t d dS|stdst ddS| t d| jd| jddS)z)Remove detached revisions from the cache.rNcss"|] }t|jdkr|VqdS)rN)rLr7rurlr/r/r0rrzprune..csrrRr^rr/r/r0rrrz2No unreferenced revisions found. Nothing to prune.r/rrz unreferenced revision(s) (z total).rzProceed?zPruning cancelled.rz! unreferenced revision(s); freed r)rrr rrror*r$updaterLrhr#rrfrPrFrrmrrr) rrrrnrr%r$rOdetachedrr r r/r/r0prune~sH   r)zhf cache verify gpt2z)hf cache verify gpt2 --revision refs/pr/1z.hf cache verify my-dataset --repo-type datasetrUrTrlzXCache directory to use when verifying files from cache (defaults to Hugging Face cache). local_dirz?If set, verify files under this directory instead of the cache.fail_on_missing_filesz--fail-on-missing-filesz?Fail if some files exist on the remote but are missing locally.fail_on_extra_filesz--fail-on-extra-fileszLFail if some files exist locally but are not present on the remote revision.tokenc Cs|dur|durtdtjddt|d}|j|t|dr"|jnt|||||d} d} t| j } | rYtd | j D]} td | d d | d d| dd| dqsX   ( ( $         A   2  ,6,  #,2 _  K B