Axj>dZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Ze Zeejd d Zeejd d Zejd dZeejddZeejddZd$dZd%dZd&dZd&dZd&dZd'dZd&d Zd&d!Z d&d"Z!d&d#Z"dS)(a" System resource fitness checks for worker startup validation. Provides comprehensive checks for: - Memory availability - Disk space - Network connectivity - CUDA library versions - GPU compute benchmark Auto-registers when worker starts, ensuring system readiness before accepting jobs. ) annotationsN)register_fitness_check) RunPodLogger) is_availableRUNPOD_MIN_MEMORY_GBz4.0RUNPOD_MIN_DISK_PERCENTz10.0RUNPOD_MIN_CUDA_VERSIONz11.8RUNPOD_NETWORK_CHECK_TIMEOUT5RUNPOD_GPU_BENCHMARK_TIMEOUT2version_stringstrreturntuple[int, int]ctjd|}|rDt|dt|dfSdS)z Parse version string to tuple for comparison. Args: version_string: Version string like "12.2" or "CUDA Version 12.2" Returns: Tuple of ints like (12, 2) for comparison z (\d+)\.(\d+)rr)rr)researchintgroup)rmatchs \C:\Projects\Historical\venv\Lib\site-packages\runpod/serverless/modules/rp_system_fitness.py_parse_versionr$sQ Io~ 6 6E :EKKNN##SQ%8%899 6dict[str, float]c ddl}|}|jdz }|jdz }|j}|||dS#t $r t d5}i}|D]W}|dd\}} t| d|| <X| ddd z }| d dd z }|dkr d d||z z znd}|||dcdddcYS#1swxYwYYdS#t$r} td | | d} ~ wwxYwwxYw) z Get system memory information. Returns: Dict with total_gb, available_gb, used_percent Raises: RuntimeError: If memory check fails rN@)total_gb available_gb used_percentz /proc/meminfo:rMemTotali MemAvailabledzFailed to read memory info: ) psutilvirtual_memorytotal availablepercent ImportErroropensplitrstripget Exception RuntimeError) r'memr r!r"f meminfo_kblinekeyvaluees r_get_memory_infor:5s#J ##%%9(}0 { !((   JJJ Jo&& !-/ DDD!%C!3!3JC.1%++--2B.C.CJsyy{{++&>>*a88GD)~~na@@GL ;Ca<    """""""""  T T T T T T T        *$ * * * *% * * *    " O$ O O O O    MMMDDDEE1LMsB3B==AD+D&&D+ str | NonecKd} tjddtjjtjjd{V}tj|dd{V\}}|jdkrq|dd }|d D]D}d | vsd | vr| cSEnk#t$r^}|r5|j.| | d{Vtd|Yd}~nd}~wwxYwd} tjdtjjtjjd{V}tj|dd{V\}}|jdkr|dd }|d D]g}d|vra|d}t!|dkr9|d d}d|cShtdnk#t$r^}|r5|j.| | d{Vtd|Yd}~nd}~wwxYwdS)z Get CUDA version from system. Returns: Version string like "12.2" or None if not available Raises: RuntimeError: If CUDA check fails critically Nnvccz --version)stdoutstderrrPrzutf-8replace)errors releaseversionznvcc not available: z nvidia-smiz CUDA Version:rzCUDA Version: z7nvidia-smi output found but couldn't parse CUDA versionznvidia-smi not available: )rWcreate_subprocess_exec subprocessPIPErX communicate returncodedecoder.lowerr/r1killwaitr?debuglen)processrirboutputr6r9parts cuda_versions r_get_cuda_versionrsnG.6  %*%*           "*7+>+>+@+@!LLLLLLLLL    " "]]79]==F T** ( ( ,, TZZ\\0I0I::<<'''1J ...  !w)1 LLNNN,,..  ,,,-------- .G46 %*%*          "*7+>+>+@+@!LLLLLLLLL    " "]]79]==F T** ? ?"d** JJ77E5zzA~~',Qx~~'7'7'='='?'?'B > >>>>> IIO P P P 444  !w)1 LLNNN,,..  2q2233333333 4 4s?C'C0.C00 E:AEED J(J K,AK''K,c Ktd{V}|stddSt|}tt}||kr4t d|dd|dd|dd|dd td |dd|dd |dd|dd dS) z Check CUDA library versions meet minimum requirements. Raises: RuntimeError: If CUDA version is below minimum Nz0Could not determine CUDA version, skipping checkzCUDA version too old: r.rz found, z requiredzCUDA version check passed: z (minimum: ))rr?warnrMIN_CUDA_VERSIONr2r@)cuda_version_strr min_versions r_check_cuda_versionsrs;/00000000  CDDD""233L !122Kk!! :\!_ : :|A : :1~ : : +A : : :   HH 8l1o 8 8 Q 8 8 ^ 8 8&1!n 8 8 8rcKtstddS ddl}|jstddS|j|j|j}|dkrtdt|D]} |j |}|j dkrtd|d| dd | }|jo#t$r}td |d ||d}~wwxYwtd |ddS#t $rtdYn$t$r}td||d}~wwxYw ddl}|j|jj}|dkrtdt|D]} |j|| d}|jp#t$r}td |d||d}~wwxYwtd |ddS#t $rtdYdSt$r}td||d}~wwxYw)aW Verify CUDA can be initialized and devices are accessible. Tests actual device initialization, memory access, and device properties. This catches issues where CUDA appears available but fails at runtime. Skips silently on CPU-only workers. Raises: RuntimeError: If CUDA initialization or device access fails z3No GPU detected, skipping CUDA initialization checkNrz>AEFL $$&&&w5577 1  CDD D|$$  A q!!%%'''HHTNN  ,,....   "BBBqBB   \< \ \ \     LLL JKKKKKK LLLCCCDD!KLs7F 'A1F A+EF  E*E%%E**!F $G3 G<GGA3L A,J98L9 KKK!L$M ( M 1MM c$KtstddS ddl}|jstddSd}t j}|||d}|||d}| |||j t j|z dz}tdz}||krtd |d d |d d t d |d ddS#t$rtdYn=t$rt$r'}td|Yd}~nd}~wwxYw ddl}d}t j}|j||}|j||}| |||j t j|z dz}tdz}||krtd |d d |d d t d |d ddS#t$rtdYn=t$rt$r'}td|Yd}~nd}~wwxYwtddS)a Quick GPU compute benchmark using matrix multiplication. Tests basic tensor operations to ensure GPU is functional and responsive. Skips silently on CPU-only workers. Raises: RuntimeError: If GPU compute fails or is too slow z/No GPU detected, skipping GPU compute benchmarkNrz1CUDA not available in PyTorch, skipping benchmarkrrrrRz+GPU compute too slow: Matrix multiply took rSz ms (max: rTz;GPU compute benchmark passed: Matrix multiply completed in msrz$PyTorch GPU benchmark setup failed: z*CuPy not available, skipping GPU benchmarkz!CuPy GPU benchmark setup failed: zHPyTorch/CuPy not available for GPU benchmark, relying on gpu_test binary)rr?rzrrrrUrVrandnmatmulrGPU_BENCHMARK_TIMEOUTr2r@r,r1rrrandomr) rsizeraABrdmax_msr9rs r_check_gpu_compute_benchmarkrvs ?? CDDD$= z&&((  III J J J F&((  KKd6K 2 2 KKd6K 2 2 Q    '))J6$> &-   )jQ))()))   \* \ \ \ \     ;;; 9:::::  === ;;;<<<<<<<<=:&(( IOOD$ ' ' IOOD$ ' ' !Q $$&&&'))J6$> &-   )jQ))()))   \* \ \ \ \     @@@ >?????  ::: 8Q8899999999:IIRsC7D='CD==$F#F6FF!C0J$K39K3 K..K3ctdtd d}td d}td d}trOtdtd d}td d }td d }d Std d S)z Auto-register system resource fitness checks. Registers memory, disk, and network checks for all workers. Registers CUDA version, initialization, and GPU benchmark checks only if GPU is detected. z*Registering system resource fitness checksrr;c"tdS)z!System memory availability check.N)rBrr _memory_checkz2auto_register_system_checks.._memory_checks #$$$$$rc"tdS)zSystem disk space check.N)rMrrr _disk_checkz0auto_register_system_checks.._disk_checks rc2Ktd{VdS)zNetwork connectivity check.N)rerrr_network_checkz3auto_register_system_checks.._network_checks,*+++++++++++rz5GPU detected, registering GPU-specific fitness checksc2Ktd{VdS)zCUDA version check.N)rrrr_cuda_version_checkz8auto_register_system_checks.._cuda_version_checks,'(( ( ( ( ( ( ( ( ( (rc2Ktd{VdS)z!CUDA device initialization check.N)rrrr_cuda_init_checkz5auto_register_system_checks.._cuda_init_checks,-.. . . . . . . . . .rc2Ktd{VdS)zGPU compute benchmark check.N)rrrr_benchmark_checkz5auto_register_system_checks.._benchmark_checks,/00 0 0 0 0 0 0 0 0 0rz5No GPU detected, skipping GPU-specific fitness checksNrr;)r?rzrr)rrrrrrs rauto_register_system_checksrs"II:;;;%%%%,,,, K IJJJ  ) ) )  )  / / /  /  1 1 1  1 1 1 IJJJJJr)rrrr)rrr)rrf)#__doc__ __future__rrWosrrErU rp_fitnessr rp_loggerr utils.rp_cudarrr?floatenvironr0r>rHrrrZrrr:rBrMrerrrrrrrrrs  #""""" ......######999999lnnbjnn%;UCCDD 5(A6JJKK:>>";VDDBJNN+I3OOPPBJNN+I3OOPP"-J-J-J-J`.B M M M MF7777t:\L\L\L\L~ZZZZz,K,K,K,K,K,Kr