XjI,dZddlZddlmZddlZddlmZmZmZm Z ddl m Z m Z ddl mZmZmZmZmZmZmZmZmZmZmZddlmZmZdd lmZmZed Zed Z ee!ed fZ"ee!dzed fZ#ee!dzed fZ$dddddddddd Z%ee!dzedee&e%dfZ'ee!dzed fZ(ee)dzed fZ*ee)dzedd fZ+ee)dzed d! fZ,e-d"d#d$g% dd&e$d'ed(dfd)Z.d*d+hZ/d,ed(e0fd-Z1e-d.d/d0g% dd&e$d2ee!dzed3 fd4ee!dzed5 fd6ee!dzed7 fd8ee!dzed9 fd:ee0ed;dZ2e-d?d@dAgBdddddddddddddddddddddddddCdDe"dEee!edF fdGee!edH fd6ee!edI fdJee!edK fd8ee!edL fd4ee!edM fd2ee!edN fd&e$dOee!dzedP fd'edQee)edR fdSee)edT fdUee)dzedV fdWeedzedX fdYee3dzedZ fd[ed\ee!dzed]d^ fd_e'd`e(dae*dbee!dzedc fddee!dzede fdfe+dge,dhee!dzedidj fdkee!dzedldm fdnedoedpedqedree!dzedseedtfd(dfBduZ4e -d?dvdwgB ddEee!dzedx fdyee!dzedz fdDe#d6ee!dzeegd{d|fd}ee!dzed~ fd&e$d'ed(dfdZ5 dd6ee!dzeegd{dfd_ee!dzeegddfdee!dzed fdOee!dzed fdedee)dzed fd'ed(dfdZ6e -d"ddgBe6e-dde6e7e de-dg% ddDe"d&e$d'ed(dfdZ8e-gd% ddDe"d&e$dEee!dzedF fd6ee!dzedI fdJee!dzedK fd8ee!dzedL fdGee!dzed fd[ee!dzed fdOee!dzedP fd\ee!dzed]d fd_e'd`e(dae*dfe+dge,dhee!dzedid fdkee!dzedld fdQee)dzedR fdSee)dzedT fdUee)dzedV fdWeedzedX fdYee3dzedZ fd'ed(df0dZ9e-dg% ddDe"d&e$dee0edd fd'ed(df dZ:e-dg% ddDe"d&e$d'ed(dfdZ;e-dg% ddDe"d&e$dee0edd fd'ed(df dZdzfdZ?dS)z2CLI commands for Hugging Face Inference Endpoints.N) Annotated)InferenceEndpointHardwareInferenceEndpointScalingMetricInferenceEndpointType_set_parallelism_in_image)CLIErrorHfHubHTTPError) EnvFileOptEnvOpt RevisionOpt SearchOptSecretsFileOpt SecretsOpt SoftChoiceTokenOpt get_hf_api parse_env_map typer_factory)ArgumentOption)_dataclass_to_dictoutz(Manage Hugging Face Inference Endpoints.)helpz.Interact with the Inference Endpoints catalog.zEndpoint name.z_The namespace associated with the Inference Endpoint. Defaults to the current user's namespace.customhfServellamacppsGLangteitgi tgiNeuronvLLM vLLMNeuron) rzhf-serversglangrr z tgi-neuronvllmz vllm-neuron--enginezbManaged engine image to run --custom-image with (e.g. 'vllm'). Defaults to an arbitrary container.) click_typerzVHealth check route exposed by the container (e.g. '/health'). Requires --custom-image.zDPort the container listens on (e.g. 30000). Requires --custom-image.--tensor-parallel-sizezZNumber of accelerators to shard a single model copy across (vLLM and SGLang engines only).--data-parallel-sizezFNumber of model copies to run, one per accelerator (vLLM engine only).z list | lszhf endpoints lsz"hf endpoints ls --namespace my-org)examples namespacetokenreturnct|} |||}nT#t$rG}tjd|t j|jj |d}~wwxYwg}|D]}|j }| di}| di} | di} | d i} | | d d t| tr| d d nd t|tr| d d nd t| tr| dd nd t| tr| dd nd t| tr| dd nd t| tr| dd nd t| tr| dd nd dtj|d dS)z6Lists all Inference Endpoints for the given namespace.r,r+r,zListing failed: codeNstatusmodelcomputeprovidername repositorystatetask framework instanceTypevendorregion)r7r4r3r;r<instancer>r?id_key)rlist_inference_endpointsr rerrorclick exceptionsExitresponse status_coderawgetappend isinstancedicttable) r+r,api endpointsrDresultsendpointrJr3r4r5r6s XC:\Projects\Historical\venv\Lib\site-packages\huggingface_hub/cli/inference_endpoints.pylsrUlsW 5 ! ! !CP009E0RR PPP ,U,,---##)C#DD%OPG  l2&&$$'')R((77:r**++8B5$8O8OW<444UW5?5M5MU&**Wb111SU1;E41H1HP &"---b;EeT;R;RZUYY{B777XZ?I'SW?X?X`GKK;;;^`8B8T8R8RZ(,,x444XZ8B8T8R8RZ(,,x444XZ   Igf%%%%%%s* A;AA66A; deprecated not_availablehwcN|jtvo|j|jz |jkS)zhWhether the namespace can deploy one replica on this hardware right now: usable status and enough quota.)r3_UNDEPLOYABLE_STATUSESmax_acceleratorsused_acceleratorsnum_acceleratorsrXs rT_is_deployabler_s- //uB4G"J^4^bdbu4uhardwarezhf endpoints hardwarez4hf endpoints hardware --vendor aws --accelerator gpuFr>z>Only show hardware hosted by this cloud provider (e.g. 'aws').r?zEOnly show hardware available in this cloud region (e.g. 'us-east-1'). acceleratorzGOnly show hardware with this accelerator (e.g. 'cpu', 'gpu', 'neuron'). instance_typeztjd | jd| jd| j d| j d| j d S| rtjdd S| rtjdd Sddd|Dfdd|Dfdd|Dfdd|DffD} tj| rdd | dnd d S)!aList the hardware available to deploy an Inference Endpoint on. Only the hardware the namespace can deploy on right now is listed: a usable status, and enough accelerator quota left for one replica. Use `--all` to list every combination the API returns. Quota is per namespace, so pass the same `--namespace` you will pass to `hf endpoints deploy`. Prices are in USD, per replica per hour. r/r0cg|]}/|jkr/|jkrd/|jkr3/|jk|SN)r>lowerr?rbrc).0rXrbrcr?r>s rT zhardware..s  Nbioo//6<<>>AA ^ry00FLLNNBB  BN$8$8$:$:k>O>O>Q>Q$Q$Q  "b&6&<&<&>&>-BUBUBWBW&W&W 'X&W&Wr`c6g|]}st||Sr_)rirXrds rTrjzhardware..s+GGGbGN24F4FGrGGGr`cB|j|j|j|j|jfSrg)r>r?rbrcr]r^s rTzhardware..sBIr~rGWY[Yl mr`)keycTg|]%}t|d|jd|jiz&S)quota/)rr\r[rirXs rTrjzhardware..s@ s s seg  # #w23G0_0_"J]0_0_&` ` s s sr`) r>r?rbrc instance_size memory_gb gpu_memory_gbprice_per_hourrrr3id)headersrBc38K|]}t||VdSrgrmrts rT zhardware..s/??2N2,>,>?B??????r`NzmDeploy on one of these, e.g.: hf endpoints deploy my-endpoint --repo --framework --vendor z --region z --accelerator z --instance-type z --instance-size zMNone of these can be deployed on right now, see the QUOTA and STATUS columns.zGUse '--all' to also show hardware that cannot be deployed on right now.c g|]S\}}}||d|Dv(|d|ddt|dTS)Nc6h|]}|Srl)rh)rivs rT z&hardware...s :T:T:T17799:T:T:Tr`z 'z ' (valid: , ))rhjoinsorted)riflagvaluevalids rTrjzhardware..s    "eU  U[[]]:T:Te:T:T:T%T%T C Cu C C &--(@(@ C C C&U%T%Tr`z--vendorch|] }|j Srl)r>rts rTrzhardware..%H%H%HBbi%H%H%Hr`z--regionch|] }|j Srl)r?rts rTrzhardware..rr`z --acceleratorch|] }|j Srl)rbrts rTrzhardware..s/W/W/W2/W/W/Wr`z--instance-typech|] }|j Srl)rcrts rTrzhardware..s3]3]3]B4D3]3]3]r`zNo such hardware: z; .zCNo hardware matches all of these filters at once. Try dropping one.) r!list_inference_endpoints_hardwaresortrrOnexthintr>r?rbrcrur)r+r>r?rbrcrdr,rP hardware_listmatchingvisibleitemsrXunknowns ````` rTrarasV 5 ! ! !C99IUZ9[[MHHGGGHGGGG LLmmLnnn t skr s s sEI     $?????FFFS  U  U U-/Y U UGI~ U U!/ U UBDBR U U       `aaaaa   Z[[[[[   V%H%H-%H%H%HIV%H%H-%H%H%HI +/W/W/W/W/WX"M3]3]}3]3]3]^ '      W 67!3!3 6 6 6 6W      r`deployzChf endpoints deploy my-endpoint --repo gpt2 --framework pytorch ...zhf endpoints deploy my-endpoint --repo openai/gpt-oss-120b --framework custom --engine vllm --custom-image vllm/vllm-openai:v0.23.0 --tensor-parallel-size 8 ...)r7r*)r+r;r, min_replica max_replicascale_to_zero_timeoutscaling_metricscaling_thresholdrevision custom_imageengine health_routeportcontainer_registry_usernamecontainer_registry_passwordtensor_parallel_sizedata_parallel_sizecontainer_commandcontainer_argsenvenv_filesecrets secrets_file endpoint_typer7repozeThe name of the model repository associated with the Inference Endpoint (e.g. 'openai/gpt-oss-120b').r<z@The machine learning framework used for the model (e.g. 'vllm').z?The hardware accelerator to be used for inference (e.g. 'cpu').ruzNThe size or type of the instance to be used for hosting the model (e.g. 'x4').zYThe cloud instance type where the Inference Endpoint will be deployed (e.g. 'intel-icl').zTThe cloud region in which the Inference Endpoint will be created (e.g. 'us-east-1').zVThe cloud provider or vendor where the Inference Endpoint will be hosted (e.g. 'aws').r;zCThe task on which to deploy the model (e.g. 'text-classification').rzVThe minimum number of replicas (instances) to keep running for the Inference Endpoint.rzRThe maximum number of replicas (instances) to scale to for the Inference Endpoint.rzFThe duration in minutes before an inactive endpoint is scaled to zero.rz!The metric reference for scaling.rzeThe scaling metric threshold used to trigger a scale up. Ignored when scaling metric is not provided.rrz--custom-imagezhDocker image URL for the container to run (e.g. 'nexagi/sglang:v0.5.12'). Requires '--framework custom'.rrrrzGUsername used to authenticate with the registry hosting --custom-image.rzPOptional password used to authenticate with the registry hosting --custom-image.rrrz--container-commandzpOverride the container entrypoint, as a quoted string split into tokens (e.g. "python -m sglang.launch_server").rz--container-argsz~Arguments appended to the container entrypoint, as a quoted string split into tokens (e.g. "--tp 8 --reasoning-parser qwen3").rrrrrz--typezTEndpoint access type. Defaults to 'authenticated' (token-gated, publicly reachable).c t||||||||} dt||D}!dt||D}"i}#|||#d<| | |#d<|||#d<|||#d<|rtj||#d <|rtj||#d <|!r|!|#d <|"r|"|#d <t | }$|$jd!id|d|d|d|d|d|d|d|d|d| d| d| d| d|d|d| d||#}%tj|%j tj d|d dS)"zDeploy an Inference Endpoint from a Hub repository. Run `hf endpoints hardware` to list the valid `--vendor`, `--region`, `--accelerator`, `--instance-type` and `--instance-size` combinations. rrrrrrrci|] \}}||pd Sr8rlrirprs rT zdeploy..s"WWWJCsEKRWWWr`ci|] \}}||pd Srrlrs rTrzdeploy..s"ccc U3 cccr`Ntyperrrrrrrr/r7r9r<rbrurcr?r>r+r;r,rrrrrrUse 'hf endpoints describe !' to check the deployment status.rl) _build_custom_imagerrshlexsplitrcreate_inference_endpointrrNrJr)&r7rr<rbrurcr?r>r+r;r,rrrrrrrrrrrrrrrrrrrrrcustom_image_dictenv_map secrets_mapparamsrPrSs& rTrrs_t,! $?$?1-   XW-X2N2N2T2T2V2VWWWGccmG\6Z6Z6`6`6b6bcccKF &v$!2~".0K,-".0K,-E&+k2C&D&D"#?#(;~#>#>  u ('y 5 ! ! !C,s, T4) K  $m  $m vv)Te K K&~,+ 43!" %H(HX\H R4 R R RSSSSSr`zChf endpoints catalog deploy --repo meta-llama/Llama-3.2-1B-Instructz>hf endpoints catalog deploy --recipe sizzling-biryani-g4xsi1aczThe name of the model repository associated with the Inference Endpoint (e.g. 'openai/gpt-oss-120b'). Deploys its default recipe. Mutually exclusive with --recipe.recipezoThe id of the catalog recipe to deploy, as listed by 'hf endpoints catalog ls'. Mutually exclusive with --repo.)cpugpuneuronzDThe hardware accelerator to be used for inference. Only with --repo. gguf_filezUThe GGUF file to deploy, for models that have one recipe per quant. Only with --repo.c |du|dukrtdt|} ||||||||}nT#t$rG} t jd| t j| j j | d} ~ wwxYwt j |j t j d|jddS) awDeploy an Inference Endpoint from the Model Catalog. Catalog models are deployed through a recipe: a hardware and engine combination that has been tested for them. Pass --repo to deploy the default recipe of a model, optionally narrowed down with --accelerator and --gguf-file, or pass --recipe to deploy an exact recipe listed by `hf endpoints catalog ls`. NzWProvide exactly one of --repo or --recipe. List recipes with 'hf endpoints catalog ls'.r/)repo_id recipe_idr7rbrr+r,zDeployment failed: r1rr)rr&create_inference_endpoint_from_catalogr rrDrErFrGrHrIrNrJrr7) rrr7rbrr+r,rPrSrDs rTdeploy_from_catalogrs\  &D.))pqqq 5 ! ! !C P==#>   PPP ///000##)C#DD%OPHX\H [8= [ [ [\\\\\sA BABBz.Only show recipes running on this accelerator.)rr$rr%z0Only show recipes running this inference engine.licensez8Only show models under this license (e.g. 'Apache 2.0').z8Only show models for this task (e.g. 'text-generation').searchlimitz%Limit the number of models to return.c t|} ||||||||}nT#t$rG} tjd| t j| jj | d} ~ wwxYwd|D} tj | d| r%tj d | d ddSdS) zList the models available in the Model Catalog. One row per recipe, i.e. per tested way of deploying a model. Deploy one with `hf endpoints catalog deploy --recipe `. r/)rbrrr;rrr,zCatalog fetch failed: r1Nc |g|]9}|jD]/}|j|j|j|j|j|j|jd0:S))rr;rrbrrr)recipesrr;rrbrrry)rir4rs rTrjz list_catalog..Nso    m   }J}!-m)      r`rrAz6Deploy one with: hf endpoints catalog deploy --recipe r) rlist_inference_catalogr rrDrErFrGrHrIrOr) rbrrr;rrr,rPmodelsrDrs rT list_catalogrs!@ 5 ! ! !C P++#,   PPP 2522333##)C#DD%OP     EIeK(((( c a%PQ(S^J_aabbbbbccs/ BAA;;Bzhf endpoints catalog lsz%hf endpoints catalog ls --engine vllmz list-catalogT)r7hiddencatalogr7z!hf endpoints describe my-endpointc4t|} ||||}nT#t$rG}tjd|t j|jj |d}~wwxYwtj |j dS)z+Get information about an existing endpoint.r/r7r+r,zFetch failed: r1N) rget_inference_endpointr rrDrErFrGrHrIrNrJr7r+r,rPrSrDs rTdescriberis 5 ! ! !CP--49TY-ZZ PPP *5**+++##)C#DD%OPHX\+ A<AA77A<)z/hf endpoints update my-endpoint --min-replica 2z8hf endpoints update my-endpoint --tensor-parallel-size 8zdhf endpoints update my-endpoint --container-args "--enable-auto-tool-choice --tool-call-parser lfm2"zBThe machine learning framework used for the model (e.g. 'custom').zrThe specific model revision to deploy on the Inference Endpoint (e.g. '6c0e6080953db56375760c0471a8c5f2929baf11').aDocker image URL for the container to run (e.g. 'nexagi/sglang:v0.5.12'). Replaces the image currently configured on the endpoint rather than patching it, so pass the engine and container settings you want to keep along with it, run 'hf endpoints describe NAME' first to see them.zOverride the container entrypoint, as a quoted string split into tokens (e.g. "python -m sglang.launch_server"). Replaces the current value; pass an empty string to clear it.a_Arguments appended to the container entrypoint, as a quoted string split into tokens (e.g. "--enable-auto-tool-choice --tool-call-parser lfm2"). Replaces the arguments currently set on the endpoint rather than adding to them, so include the ones you want to keep, run 'hf endpoints describe NAME' first to see them. Pass an empty string to clear them.ct| | | | | | nd| |nd}t|} |||||||||tj|nd|tj|nd| | nd| |nd|||||||||}nT#t $rG}t jd|tj |j j |d}~wwxYwt j |jdS)zUpdate an existing endpoint.N)rrrrrr/)r7r+r9r<rr;rrrrrrbrurcrrrrrr,zUpdate failed: r1)rrupdate_inference_endpointrrr rrDrErFrGrHrIrNrJ)r7r+rrbrurcr<rr;rrrrrrrrrrrrrr,rrPrSrDs rTupdaterzsn|,! 5A5M11SW1=1I--t  5 ! ! !CP00*@Q@]ek*;<<AA99A>z&hf endpoints scale-to-zero my-endpointc4t|} ||||}nT#t$rG}tjd|t j|jj |d}~wwxYwtj |j dS)z$Scale an Inference Endpoint to zero.r/rzScale To Zero failed: r1N) r scale_to_zero_inference_endpointr rrDrErFrGrHrIrNrJrs rT scale_to_zerorcs 5 ! ! !CP77TY^c7dd PPP 2522333##)C#DD%OPHX\rrc|R|||||||d}d|Dx} r&tdd| ddS||td||td|||td d |i} ||| d <||| d <|| Stt||| i|| S)aBuild the `custom_image` argument of `HfApi` from the flat image flags. A bare image URL describes a custom container and is passed flat, `HfApi` keys it as `custom`. `--engine` selects a managed engine image instead, whose config is keyed by the API's variant name. An engine this version doesn't know about is forwarded as typed, so the API names the bad variant. N)r&z--health-routez--portz--container-registry-usernamez--container-registry-passwordr(r)cg|] \}}|| Srgrl)rirrs rTrjz'_build_custom_image..s!UUU[T55CTDCTCTCTr`z&--custom-image is required when using rrzE--container-registry-password requires --container-registry-username.zWContainer registry credentials can only be set for a custom container without --engine.zV--tensor-parallel-size and --data-parallel-size require --engine (e.g. --engine vllm).url healthRouter)rr)rrrrENGINE_IMAGE_KEYSrK) rrrrrrrr image_flagsusedconfigs rTrrtsC"*-H-H&:$6  VUK,=,=,?,?UUU U4 XVDIIdOOVVVWW Wt".3N3V^___ 9Epqqq ~/;?Q?]oppp<(F ,} v ~  %   vv . .71-   r`)NN)NNNNNFN)NNNNNNN)NNNNNNNNNNNNNNNNNNNNNN)NFN)@__doc__rtypingrrE$huggingface_hub._inference_endpointsrrrrhuggingface_hub.errorsrr _cli_utilsr r r rrrrrrrr _frameworkrr_outputrrie_cli catalog_appstrNameArgNameOpt NamespaceOptrlist EngineOptHealthRouteOptintPortOptTensorParallelSizeOptDataParallelSizeOptcommandrUrZboolr_rafloatrrr add_grouprrrrrrrNrrlr`rTrsH88   <;;;;;;;                          )(((((((,,,,,,,, F G G Gm!QRRR   H"###%  $J F !!!#  $J F n       $J F:dd#45566 q  $J F e $J F S "$J F i $J F U '8:^&_``"&&& & &&&a`&F'80T>#     5c c c  d TUUU W c  d [\\\ ^  c  d ]^^^ `c  d RSSS Uc $   u    %c 4 5c 6 7c c c c L M Oj#    #' 26.2  #' WITITIT IT  x     IT S    IT R    IT( a    )IT4 l    5IT@  g     AITL  i     MITZ[IT\  d V     ]ITh iITj i    kITv e    wITB% d Y    CITN&-4    OITZ!  x    [ITfgITh d  {    iITv wITx!yITz {IT|"+ d Z    "}ITH"+ d c    "IITT0UITV,WITX! d  !;     YITl d  <     mIT@ AITBCITDEITF!GITH d  !z"788g    IITX YITITITITX  MH   "=:]:]  d M     :]  d /     :] :]  d !z":":":;;W    !:]. d h    /:]:;:]< =:]> ?:]:]:]:]H    Y]3>c>c d !z":":":;;A    >c  d !z"G"G"GHHC     >c d NOOO Q>c&  d NOOO Q '>c. />c0 S4Z-T!U!U!UU V1>c2 3>c4 5>c>c>c>cBu 0IKr/stt1N400>>>9---=>??#          @?  #  #'26.2  e[[ [[  d x     [ d R    [ d a    [* d l    +[6 d U    7[B d F    C[N  d V     O[Z d  o     [[p q[r!s[t u[v0w[x,y[z! d  !4     {[P d  f     Q[h d i    i[t d e    u[@% d Y    A[L&-4    M[X!  x    Y[d e[f g[[[[|;<==# 00 00  w9::: < 0 0 000>=0,:;<<#          =<  ;<==#  '  'j      >=6BCDD#          ED &#.2.2'+%)888*8 $J8* 8 * 8 "%t 8"%t8*8d 8 D[888888r`