Axj%>dZddlmZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZeZeejd d Zeejd d ZddZddZddZddZddZddZdS)a1 GPU fitness check system for worker startup validation. Provides comprehensive GPU health checking using: 1. Native CUDA binary (gpu_test) for memory allocation testing 2. Python fallback using nvidia-smi if binary unavailable Auto-registers when GPUs are detected, skips silently on CPU-only workers. ) annotationsN)Path)Anyget_binary_path)register_fitness_check) RunPodLoggerRUNPOD_GPU_TEST_TIMEOUT30RUNPOD_GPU_MAX_ERROR_MESSAGES10return Path | Nonec tdS)zj Locate gpu_test binary in package. Returns: Path to binary if found, None otherwise gpu_testrYC:\Projects\Historical\venv\Lib\site-packages\runpod/serverless/modules/rp_gpu_fitness.py_get_gpu_test_binary_pathrs : & &&routputstrdict[str, Any]c|d}dddgid}d}d}|D]csdr8ddd|dd <nd r8ddd|dd <n^d rId vrE td}||d<n#tt f$rYnwxYwdvr|dz }tfddDr|de||d<|dko||kot|ddk|d<|S)ag Parse gpu_test binary output and detect success/failure. Looks for: - "GPU X memory allocation test passed." for success - Error patterns: "Failed", "error", "cannot" for failures - GPU count from "Found X GPUs:" line Args: output: Stdout from gpu_test binary Returns: Dict with keys: - success: bool - True if all GPUs passed tests - gpu_count: int - Number of GPUs that passed tests - found_gpus: int - Total GPUs found - errors: List[str] - Error messages from output - details: Dict - CUDA version, kernel version, etc  Fr)success gpu_count found_gpuserrorsdetailszCUDA Driver Version::rr cuda_versionzLinux Kernel Version:kernelFoundGPUsrzmemory allocation test passedc3DK|]}|vVdS)N)lower).0errlines r z)_parse_gpu_test_output..as0VVssdjjll"VVVVVVr)failederrorcannotunablerrr) stripsplit startswithint IndexError ValueErrorr'anyappendlen)rlinesresult passed_countrr*s @r_parse_gpu_test_outputr<(s( LLNN  & &E FLJ**zz||   ??1 2 2 04 30B0B10E0K0K0M0MF9 n - - __4 5 5 *.**S!*<*r?r error_outputr: error_msgexcs r_run_gpu_test_binaryrbls,--K H FGGG 9["' * *PNNNOOOII7+778880P6   %*%*          '/    ! !?         wy99}}WY}??  /v//000  ; II9<99 : : :(//i  *;Ih UT$))F8,<=P>P=P,Q"R"RTTT y)) )  Ivk': I II&**>9EE I I I      llnn A A A A      PPPEEEFFCOPs1FG::A3J-JJNonecNtd tjddgdddd}|jdkrt d |jd |jd D}t|}|dkrt d t d |ddS#t$rt ddtj $rt ddt $rt$r}t d||d}~wwxYw)a Python fallback for GPU testing using nvidia-smi. Less comprehensive than binary (doesn't test memory allocation) but validates basic GPU availability by checking GPU count. Raises: RuntimeError: If GPUs not available or unhealthy z!Running Python GPU fallback check nvidia-smiz --list-gpusT Fcapture_outputtextr@checkrznvidia-smi --list-gpus failed: c:g|]}||Sr)r0)r(r*s r z*_run_gpu_test_fallback..s%PPPd4::<<PTPPPrrzNo GPUs detected by nvidia-smizGPU fallback check passed: z5 GPU(s) detected (Note: Memory allocation NOT tested)z7nvidia-smi not found. Cannot validate GPU availability.Nznvidia-smi timed outz"nvidia-smi fallback check failed: )rJrKrNrun returncoderVr?r>r1r8rWrETimeoutExpiredr\)r: gpu_linesres r_run_gpu_test_fallbackrrsII1222!L = )       ! !PPPQQ QPPfm&9&9$&?&?PPP  NN >>?@@ @  3) 3 3 3      E     $===122<  LLLCCCDD!KLsB#CA D$ DD$c@Kd}d} td{VdS#t$r)}td||}Yd}~nld}~wt$r)}td||}Yd}~n;d}~wt $r+}td|d}|}Yd}~nd}~wwxYwtd tdS#t $r0}|rtd|d ||td ||d}~wwxYw) a$ Comprehensive GPU health check (internal implementation). Execution strategy: 1. Try binary test if available 2. Fall back to Python check if binary fails/missing 3. Raise RuntimeError if all methods fail Raises: RuntimeError: If GPU health check fails FNzGPU binary not found: zGPU binary not executable: zGPU binary check failed: Tz$Attempting Python GPU fallback checkz&GPU health check failed. Binary test: z. Fallback test: zDGPU health check failed (binary disabled/missing, fallback failed): ) rbrErJrKrIr\warnrrrV)binary_attempted binary_errorra fallback_excs r_check_gpu_healthrxsL "$$$$$$$$$  0300111  555666  2S22333  II4555          1 ,11".11  4%144    sD B5A  B5A== B5 !B00B5C## D-+DDctjdddkrtddSd} t jdgddd d }|jd kod |j v}n+#tt j f$rd}Ynt$rd}YnwxYw|r,td td}dStddS)ak Auto-register GPU fitness check if GPUs are detected. This function is called during rp_fitness module initialization. It detects GPU presence via nvidia-smi and registers the check if found. On CPU-only workers, the check is skipped silently. Environment variables: - RUNPOD_SKIP_GPU_CHECK: Set to "true" to skip auto-registration RUNPOD_SKIP_GPU_CHECKtruez._gpu_health_check9s,$%% % % % % % % % % %rz8No GPU detected, skipping GPU fitness check registration)rFenvironrXr'rJrKrNrmrnr>rEror\r )has_gpur:rs rauto_register_gpu_checkrs8 z~~-r2288::fDD PQQQG  N    #q(J\V]-J z8 9 N IJJJ  & &  & & & LMMMMMs.BB- B-,B-)rr)rrrr)rr)rrc)__doc__ __future__rrLrFrNpathlibrtypingrrunpod._binary_helpersr rp_fitnessr rp_loggerr rJr3rrXrRrUrr<rbrrrxrrrrrs#""""" 222222......######lnn#bjnn%>EEFFS(GNNOO''''AAAAHDPDPDPDPN-L-L-L-L`/ / / / d)N)N)N)N)N)Nr