Xj] 8dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZd d lmZejeZejd Zejd d eejdZdZdZdZdZdZdZ ejej!edZ"de#e zde fdZ$de#e zde#de fdZ%de#e zde#de fdZ&de de'fdZ(de de'fdZ)de d e*dzde'fd!Z+de#e zde'fd"Z,d#e de'fd$Z-de#e zde'fd%Z.de#e zde#de dzfd&Z/de#e zde*fd'Z0de#e zde*fd(Z1de de#e zddfd)Z2d*e d+e de#e zddfd,Z3de#e zde fd-Z4d*e#e zde#e zde#dzfd.Z5d/d0d*e#e zde#e zd1e'de dzfd2Z6d*e#e zde#e zde dzfd3Z7de de fd4Z8de de fd5Z9ede de d6fd7Z:d8e d9e#ddfd:Z;d*e de de fd;Zde#de#e zd e*dzde'fd>Z?d/d?d*e#de#de#e zd e*dzd@e'de'f dAZ@de de#e zdeAe dzfdBZBde dCeAe de#e zde*fdDZCde#e zdeAe fdEZDde#e zdeEe e*ffdFZFd8e dGeAe de#e zddfdHZGde de#e zde*fdIZHdS)JajCache-wide shared blob store, deduplicating Xet files across cached repos. Store entries at `/blobs/<2-hex-prefix>/` are regular files. Per-repo `blobs/` entries are relative symlinks to them. A companion `.refs` manifest lists those symlinks so cache deletion only needs to inspect the blobs it touches instead of scanning the entire store. The manifest is a hint, not trusted metadata: every entry is validated against the filesystem before use, and any missing or unreadable manifest makes garbage collection leak the shared blob rather than risk deleting referenced content. N)contextmanager)Path) Generator)FileLock SoftFileLock) constants)loggingz [0-9a-f]{64}z(?:|z)--.+blobsz.huggingface-shared-blobs1z.refsz.lock z\.[0-9a-f]{8}\.tmp cache_dirreturnc0t|tz S)zBReturn the path of the shared blob store inside a cache directory.)rSHARED_BLOBS_DIR_NAME)rs TC:\Projects\Historical\venv\Lib\site-packages\huggingface_hub/utils/_shared_blobs.pyshared_blobs_dirr7s  ??2 22xet_hashct|td|dt||ddz |z S)zuReturn the store path for a given Xet file hash. Raises `ValueError` if `xet_hash` is not a valid Xet hash. NzInvalid Xet file hash: ''.r)_XET_HASH_REGEX fullmatch ValueErrorrrrs rshared_blob_pathr<sR   **2@H@@@AAA I & &"1" 5 @@rcZt|||tS)z=Return the reference manifest path for a given Xet file hash.)r with_name_MANIFEST_SUFFIXrs rshared_blob_manifest_pathr"Fs, Ix 0 0 : :h;ZHX;Z;Z [ [[rpathc| tj|jS#t$rYdSwxYwNF)statS_ISREGlstatst_modeOSErrorr#s r_is_regular_filer,KD|DJJLL0111 uu *- ;;c| tj|jS#t$rYdSwxYwr%)r&S_ISDIRr(r)r*r+s r _is_directoryr1Sr-r. store_path expected_sizec ||dS |}n#t$rYdSwxYwtj|jsdS|j|kr+t d|d|jd|ddStj |tj s t d|ddSdS) zPReturn whether `store_path` is a regular, readable payload of the expected size.NFz Shared blob 'z' has an unexpected size (z instead of z). Not using it.z ' is not readable. Not using it.T) r(r*r&r'r)st_sizeloggerwarningosaccessR_OK)r2r3 store_stats r_is_usable_store_entryr<[su%%'' uu < * + +u]** J  *BT  bo      u 9Z ) )SzSSSTTTu 4s  ))ct|}|tz }t|rt|sdS |t dkS#t $rYdSwxYw)z?Return whether `path` is an owned, supported shared blob store.F )rSHARED_BLOBS_MARKER_NAMEr1r, read_textSHARED_BLOBS_LAYOUT_VERSIONr*)r# store_dir marker_paths ris_shared_blobs_dirrDrsT I66K  # #+;K+H+Hu$$&&-H*L*L*LLL uusA A)(A)rBctd}t|}|D]w}t|jt |sdS ||sdS_#ttf$rYdSwxYw|D]}| dt| S)zRemove leftover marker temporaries. Returns whether the directory is empty afterwards, i.e. safe to mark. Any foreign content returns False. r>NFT missing_ok) rAlistiterdir_MARKER_TMP_REGEXrnamer, startswithr@r* UnicodeErrorunlinkany)rBexpected_contententriesentrys r_cleanup_abandoned_marker_tempsrS~s 69999$$&&''G  & &uz 2 2 :BRSXBYBY :55 #..u/@/@AA uu &   555 &&  %%%%9$$&&'' ''s"'B  B#"B#c t|}|tz } |dt|rdSt |rt |s t d|ddSt||| |j dtj dd} | td |t!|t#j|||d n#|d wxYwt|S#t($r+}t d |d |Yd }~dSd }~wwxYw)zJCreate and mark the shared store, refusing to adopt an unmarked directory.Texist_okz0Refusing to use unmarked shared blob directory 'rF..tmpr>rFz,Could not initialize shared blob directory '': N)rr?mkdirrDr1rSr6debug_repair_shared_directory_moder rKsecrets token_hex write_textrAchmod_shared_blob_moder8replacerNr*)rrBrC tmp_markeres r_ensure_shared_blobs_dirrfs ++I66K&&& y ) ) 4Y'' /Ny/Y/Y  LLYIYYY Z Z Z5%i;;; **k.>+[+[ARSTAUAU+[+[+[\\  /  ! !%@"D"D"D E E E   .y99 : : : Jz; / / /     . . . .J    . . . ."9---  UIUURSUUVVVuuuuus=%E:X\XdKdBk%%jk lls8A AAc tjt|j}n#t$rYdSwxYw|dtjztjzzS)zQMirror cache-root access and inheritance bits on newly created store directories.ii)r&rprr)r*S_ISGIDS_ISVTXrss r_shared_directory_moderxsh\$y//"6"6"8"8"@AA uu - < ==s8; A A ct|}tj|j|kr||dSdSN)rxr&rpr(r)ra)r#r expected_modes rr]r]sO*955M |DJJLL())]:: =!!!!!;:r blob_pathrict|}tj||tjdkst tdsdS|j}|j|krdS tj|d|dS#t$r&|t j zr|t j zsYdSYdSwxYw)zKMake a payload immutable and readable according to the shared cache policy.rlchownN) rbr8rarKhasattrr&st_gidr~r*S_IRGRPS_IROTH)r|rir blob_mode target_gids r _prepare_shared_blob_permissionsrs!),,IHY """ w$gb'22"")J~~*,, B +++++  t| # I ,D        sB)C  C ctj|}|dddkrd|dd}n|dr |dd}t tj|S)zCReturn an absolute path without the Windows extended-length prefix.Nz\\?\unc\z\\z\\?\rX)r8fspathlowerrLrr#abspath)r#path_strs r_path_for_comparisonrsyH|~--((122,((   Y ' ' ABB< )) * **rct|}t|} ||}n#t$rYdSwxYwt|jdksCt |jd|jddks |jdsdS|}d|vsd|vrdn|S)Nrr r rr> )r relative_torlenparts_REPO_DIR_REGEXras_posix)r|r relative_path relative_strs r_relative_blob_pathrs$Y//I$Y//I!--i88 tt M   A%%  $ $]%8%; < < D  q !W , ,"1% -t ))++L<''4<+?+?44\Qs6 AAFraise_on_io_errorrct|}t||dS tj|jsdStt j|}n #t$rYdSt$r|rYdSwxYwt|}| r|n |j |z }t|}tt|} ||}n#t$rYdSwxYwt!|jdksIt$|jd$|jd|jdddkrdSt||z S)zMReturn the canonical store target if `blob_path` is a managed shared symlink.Nrr r)rrr&S_ISLNKr(r)r8readlinkFileNotFoundErrorr*r is_absoluterhrrrrrrr)r|rr link_targetcomparable_blob_pathtargetrBrelative_targets r_shared_blob_targetrsYI9i008t |IOO--566 42;y1122 tt    tt 0 ::'3355 d[[;O;VYd;dF !& ) )F$%5i%@%@AAI ,,Y77 tt O !""a''  $ $_%:1%= > > F   #'z*Could not append a complete reference to 'N)r8rO_APPENDrrrrr&r'rr)r*rencodewriterfsyncr)rrrrlines r_append_manifest_referencer`sR K"+ % 2Er<    u - -B |BHRLL011 ][-[[[\\ \ 2x    "e$$$$    $(((//11 8B  T * *X}XXXYY Y     s2"AE6C E CECA'EE,c|d|jdtjdd}tjt|t|j}t j |||S)NrWrXz.shared)start) r rKr^r_r8r#relpathrrhsymlink)r|r2tmp_linkrs r_make_temporary_symlinkrvsz""#Uy~#U#U8I!8L8L#U#U#UVVHgoo&::&F&FNbclNmNmNtouuOJ))) Orc6tj o tj S)z1Return whether the shared blob store may be used.)r HF_HUB_DISABLE_SHARED_BLOBSHF_HUB_DISABLE_XETrrshared_blobs_enabledr}s4 4 YY=Y9YYrcpt|tt|sdSt ||}t ||}|dSd} t |5t||s( ddd||ddSdStt||}tt||tj||dddn #1swxYwYnR#t$rE}t d|d|Yd}~||ddSdSd}~wwxYw||dn#||dwwxYwt d|ddS) zMaterialize `blobs/` as a symlink to an existing store entry, if any. The reference manifest is flushed before the symlink becomes visible. Failures are best-effort misses and leave the regular download path untouched. NFTrFzCould not symlink 'z' from shared blob store: Blob 'z5' reused from shared blob store (no download needed).)rrrDrrrrr<rNrrrrr8rcr*r6r\)r|rrr3r2rrres rtry_link_from_shared_storersM  **2:MN^_hNiNi:j:j2u!)X66J,Y BB!u H - z * * , ,)*mDD  , , , , , ,   OOtO , , , , , /tI KKH &'DZ'P'PRd e e e Jx + + +  , , , , , , , , , , , , , , ,  S9SSPQSSTTTuuu   OOtO , , , , ,     OOtO , , ,   OOtO , , , ,  LLZ)ZZZ[[[ 4sa D/D D(AD7 DDD D DE: E E9E:EE::Fctr6t|tt |sdSt t |||S)zGReturn whether a usable store entry exists, without touching the cache.NF)rrrrDrr<r)rrr3s rhas_shared_blobrsa ! " "  $ $X . . 6"#3I#>#>?? 7u !"29h"G"G W WWr)replace_existingrc|t|dSt||}t||}||dSt |}t ||}t |} d} d} t|5t||} | ot||} t| || s(t|||tj ||d} tj | |n(#t$r| rtj||d} wxYw dddn #1swxYwYnh#t$r[} t"d|d| | rtd|d| Yd} ~ | | ddSdSd} ~ wwxYw| | dn#| | dwwxYwt"d |d dS) zMove a fresh Xet download into the store and replace its repo blob with a symlink. Best-effort: on failure the repo blob remains (or is restored as) a regular local file. Returns whether the repo blob was successfully shared. NFTzCould not publish 'z' to shared blob store: zCould not restore repo blob 'z' after shared-store failurerFrz!' published to shared blob store.)rrrjrrrrrrr<rrr8rcr*shutilcopyfiler6r\rN)r|rrr3rrir blob_path_objr2rr blob_movedstore_is_usableres rpublish_blob_to_shared_storers 9 9( C C Ku#Ix88J,Y BB/7uOOM!)X66J1*==M HJ- z * *  .}jIIH"22h7MjZg7h7hO &}6H I I I" "0 IVVV =*555!   8]3333   'OJ >>>!&J  4                 Q9QQaQQRRR  ja)aaabbhi iuuu   OOtO , , , , ,     OOtO , , ,   OOtO , , , ,  LLF)FFFGGG 4sm8D=AD1&C<;D1<%D!!D1% D=1D55D=8D59D=<F>= F"6F=F>F""F>>GcLt|}t|sdS |}n#tt f$rYdSwxYwt }ttj |}|D]}t|}| s d|j vr/||z }t||E t||d}n#t$rYdSwxYw||kr|||S)Nz..Tr)rr,r@ splitlinesr*rMsetrr8r#rrrrradd) r2rrlines referencesrr candidaters r_read_valid_manifest_referencesrsT1*==M M * *t''))4466 \ "tt EEJRW__Y//00I & &T  $ $ & & $-2E*E*E  - y) 4 4 <  (IQUVVVFF   444  Z   NN9 % % % s#&A AA#C66 DDblob_paths_to_deletect||}||r||sdS t|r|jndS#t $rYdSwxYw)zHPredict bytes freed if all known references are part of a deletion plan.Nr)rissubsetr,r(r5r*)r2rrrs rexpected_shared_blob_freed_sizers1YGGJ:3F3FG[3\3\q-=j-I-IPz!!))qP qqs)A A%$A%cvt|}t|stSt} t|}n#t $r|cYSwxYw|D]}t jd|jt|s, t|}n#t $rY[wxYw|D]d}t|jC|j |jr$t|r| |e|S)zBReturn all valid canonical payload paths in a marked shared store.z [0-9a-f]{2})rrDrrHrIr*rerrKr1rrLr,r)rrBpaths prefix_dirsrirQrRs rshared_store_blob_pathsrs[ ++I y ) )uu uuE9,,..//  ! ! ! <  8 8 @ V`HaHa @  :--//00GG    H  ! !E))%*55AJ))*/::B$U++B %    ! Ls#!A A-,A-!C C Cci}t|D]D}t||}|0|s. |j||<4#t$rY@wxYwE|S)zReturn store payloads without any valid manifest reference, with their sizes. Payloads with a missing or unreadable manifest are not listed, consistent with `sweep_shared_blob`. )rrr(r5r*)r unreferencedr2rs runreferenced_shared_blobsr*s %'L-i88 4ZKK  !* ! +5+;+;+=+=+E Z((     sA AArc||jdtjdd}t t j dfdt|D}| || d| d5}t j |dddn #1swxYwYt j|||d dS#|d wxYw) NrWrXrYc3lK|].}|dV/dS)r>N)rr).0r#rs r z$_rewrite_manifest..>sFggDT--i88AACCGGGggggggrrmrbTrF)r rKr^r_rr8r#rjoinsortedr`rarrfilenorcrN)rrrtmp_pathcontentfiles ` r_rewrite_manifestr:sn&&-*<'Y'Yw?PQR?S?S'Y'Y'YZZHRW__Y//00I)''ggggTZ[eTfTfgggggG$$$u ]]4  $D HT[[]] # # # $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ 8]+++4(((((4((((s1!A-D9'D5 D9DD9D D99Ecfttj|}ttj|}ttjt |} ||}n#t $rYdSwxYwt|rTt|j dksy.I.IJJKK(445GHH qq 2 3 3 "( ) )Q . .  $ $Z_ 5 5 =   !Z_RaR%8 8 8q z * *  8YOOJ!        :*EEM !-YGGG        $J//         ##%%-D        D 1 1 1                    IzIIaIIJJJqqqqqssB)) B76B7 G;/G. G;#G.3 G;G. G;AG.! G;.G22G;5G26G;; H0 H++H0)I__doc__r8rr^rr& contextlibrpathlibrtypingrfilelockrrrr r get_logger__name__r6compilerrrREPO_TYPES_MAPPINGrrr?rAr!rrescaperJstrrrr"boolr,r1intr<rDrSrfrjrbrxr]rrrrrrrrrrrrrrrrrrdictrrrrrrrs    %%%%%%++++++++  H % %"*_--"*YCHHVVI4P-Q-Q$R$RYYYZZ6! BJ929-E#F#F\\\]]3d 3t3333 Ad AcAdAAAA\t\s\t\\\\ 4DtC$J4. cDj T    (t((((((d t4#*t ( mt m m m m m>cDj>S>>>>""t""""" $SVY]S]bf"+sTz+d++++R3:R#*RtRRRR&PU 9 9 9Tz 9&)Dj 9HL 9 D[ 9 9 9 9FO#*OtOt OOOOHdHtHHHHD$D4DDDD $ 95E+F    FdPT,t$ZdZZZZ  !$ 14t LORVJ      FXXtXCRVJX[_XXXX"# 44444Tz 4 : 4  4 4444ntPSTXPY\`P`6  /24y EH4Z     sTzc$i6 t T3Y     )T )s4y )SSWZ )\` ) ) ) )&$&cDj&S&&&&&&r