Axj\8 <dZddlZddlZddlZddlmZddlmZmZm Z ddl m Z m Z ddl mZddlmZddlmZmZdd lmZd Zed Zd Zd ZdZejdgZdZGdde Z!de"fdZ#de$de"fdZ%de$de&dee$fdZ'de&fdZ(de$ddfdZ)de$de$fdZ*GddeZ+d e$defd!Z,d"eddfd#Z-d"eddfd$Z.d%e e&ddfd&Z/d'e&d%e e&dee0fd(Z1de$fd)Z2d*d+dded,de$d-e"d.e&d/e e3d%e e&d0e&f d1Z4dS)2a PodWorker | serverless | utils | rp_ssrf.py SSRF-safe fetching of user-supplied (job-input) URLs. Job input carries arbitrary URLs that the worker downloads. Without restriction a job can point the worker at cloud instance-metadata (169.254.169.254) or other non-public addresses (CWE-918). This module blocks any destination that is not a globally-routable IP, pins connections to a pre-validated address to defeat DNS rebinding, re-validates redirect hops, and caps download size. Pinning requires this process to open the socket, so a URL that the environment routes through a proxy is refused rather than fetched unprotected; see _refuse_if_proxied. N)suppress)IteratorListOptional)urljoinurlparse)ConnectionError) HTTPAdapter)get_environ_proxies select_proxy)SyncClientSession)httphttps)i-i.i/i3i4RUNPOD_MAX_DOWNLOAD_BYTESlz 100.64.0.0/10"RUNPOD_ALLOW_PRIVATE_DOWNLOAD_URLSceZdZdZdS) SSRFErrora Raised when a URL is rejected as unsafe (non-public destination, blocked scheme, or oversized body). Subclasses ValueError and deliberately NOT requests.RequestException so it bypasses the download path's backoff retry and `except RequestException` handler: a blocked internal URL must fail loudly, not be retried or silently reported as an ordinary failed download. N)__name__ __module__ __qualname____doc__PC:\Projects\Historical\venv\Lib\site-packages\runpod/serverless/utils/rp_ssrf.pyrr0srrreturnctjtddvS)N)1trueyes)osenvironget_ALLOW_PRIVATE_ENVstriplowerrrr_ssrf_protection_enabledr(<s> :>>,b 1 1 7 7 9 9 ? ? A AJ ripc& tj|n#t$rYdSwxYwtdd}||t fdt DrdSjs#jsjsj sj sj rdSj S)a. Return True only for a globally-routable unicast IP address. Blocks loopback, link-local (incl. 169.254.169.254 metadata), RFC1918, CGNAT, reserved, multicast, unspecified, IPv6 unique-local/link-local, and IPv4-mapped IPv6 forms of any of the above. Unparseable input is unsafe. F ipv4_mappedNc3 K|]}|vV dSNr).0networkaddresss r z"is_safe_address..Vs( E E'7g  E E E E E Er) ipaddress ip_address ValueErrorgetattrany_EXTRA_BLOCKED_NETWORKS is_private is_loopback is_link_local is_reserved is_multicastis_unspecified is_global)r)mappedr0s @ris_safe_addressr@Ds&r** uuWmT 2 2F  E E E E-D E E EEEu             ! u  s  &&hostportc tj||tj}n-#tj$r}t d|d||d}~wwxYwg}|D])}|dd}||vr||*|st d|t r)|D]&}t|st d|d |'|S) a Resolve `host` and return its IPs, failing closed on any unsafe address. Raises SSRFError if the host cannot be resolved, resolves to nothing, or resolves to any non-global address (a host resolving to both a public and a private IP is treated as hostile). Validation is skipped when the escape hatch env var is set, but resolution still occurs so callers can pin. )typezcould not resolve host : Nrzno addresses resolved for host zhost z resolves to non-public address )socket getaddrinfo SOCK_STREAMgaierrorrappendr(r@)rArBinfoserripsinfor)s rresolve_and_validaterPfs$L"4F4FGGG ?LLLA$AACAABBKLC !WQZ S== JJrNNN DB$BBCCC!!  B"2&& HDHHBHH  Js!$AA  Actjt}|tS t |}n#t $r tcYSwxYw|dkr|ntS)z4Configured per-download byte cap; defaults to 5 GiB.Nr)r"r#r$_MAX_BYTES_ENV_DEFAULT_MAX_BYTESintr4)rawvalues rmax_download_bytesrWsl *.. ( (C {!!"C """!!!!"AII55#55s?AAurlc tsdSt|t|}|rtd|d|dtddS)ab Reject `url` when the environment routes it through a proxy. Pinning only holds if this process opens the socket: through a proxy the request carries the hostname and the proxy resolves it, so the validated IP is not what gets dialed and the rebind protection silently stops applying. Failing here turns that into a loud, actionable error instead of a false sense of protection. NO_PROXY exclusions are honored (such a host is fetched directly, so pinning still holds), and the escape hatch env var allows the proxied fetch for deployments that accept the trade-off. Nzrefusing to fetch z through proxy z1: IP pinning cannot be enforced via a proxy; set z=true to allow it)r(r r rr%)rXproxys r_refuse_if_proxiedr[s $ % % 1#66 7 7E   O O OU O O); O O O     rcpt|}|jpd}d|vrd|d}|j |d|jS|S)a Build the Host header authority for `url`. The pinned adapter dials an IP but must present the original authority, so this reconstructs it from the URL rather than the socket. urlparse strips the brackets from an IPv6 literal, and RFC 7230 requires them in Host, so they are restored: without this an IPv6-literal URL yields a malformed header such as "2606::1:443". r:[])rhostnamerB)rXparsedrAs r_host_header_forrbsTc]]F ? bD d{{4{{{ {&&&&& KrcVeZdZdZdeffd ZfdZdefdZd fd Zd fd Z xZ S) PinnedIPAdaptera Requests adapter that dials a pre-validated IP instead of re-resolving the URL host, defeating DNS rebinding. The superclass builds the connection pool with its normal TLS/verification configuration; this adapter only redirects the socket to the pinned IP and keeps the TLS SNI, certificate hostname check, and Host header bound to the original URL host. It never alters certificate verification. pinned_ipc H||_tjdi|dS)Nr) _pinned_ipsuper__init__)selfrekwargs __class__s rrizPinnedIPAdapter.__init__s-#""6"""""rc pt|j|jd<tj|fi|S)NHost)rbrXheadersrhsend)rjrequestrkrls rrpzPinnedIPAdapter.sends7"27;"?"?uww|G..v...rrXct|}|jdkr6|j|_t |jpi|_|j|jd<|j|_|S)Nrserver_hostname)rschemer`assert_hostnamedictconn_kwrgrA)rjpoolrXras r_pinzPinnedIPAdapter._pinsW# =G # ##)?D  233DL.4oDL* +O  rNct||||}|||jS)N)proxiescert)rhget_connection_with_tls_contextryrX)rjrqverifyr{r|rxrls rr}z/PinnedIPAdapter.get_connection_with_tls_contextsBww66 VW47  yyw{+++rcvt||}|||S)N)r{)rhget_connectionry)rjrXr{rxrls rrzPinnedIPAdapter.get_connections3ww%%c7%;;yys###r)NNr-) rrrrstrrirpryr}r __classcell__)rls@rrdrds########/////c,,,,,, $$$$$$$$$$rrdrect}t|}|d||d||S)z:A session whose http(s) connections dial only `pinned_ip`.zhttp://zhttps://)r rdmount)resessionadapters r_build_pinned_sessionrsG!!Gi((G MM)W%%% MM*g&&& Nrrctt5|ddddS#1swxYwYdS)a Close a pinned session best-effort, tolerating teardown errors. This runs only on cleanup paths (a failed request, a closing response) where the caller's original exception is the meaningful one. A failure while closing the session's connection pools must not mask that exception, so it is suppressed rather than raised. Session.close() is idempotent and does no actionable I/O, so nothing worth surfacing is lost. N)r Exceptionclose)rs r_close_session_quietlyrs )   s 7;;c2|jfd}||_dS)a Tie `session` cleanup to `response.close()`. Each safe_get() call owns a single-use session; the caller only holds the response. Closing a requests.Response releases its connection but not the session (its adapters and connection pools), so without this the session leaks one set of pooled sockets/FDs per download in a long-lived worker. Wrapping close() makes `with safe_get(...) as r:` (and any direct response.close()) tear down both. c` |i|tdS#twxYwr-)r)argsrkoriginal_closers r close_bothz-_bind_session_to_response..close_both sF , ND +F + + + "7 + + + + + "7 + + + +s-N)r)responserrrs ` @r_bind_session_to_responsers8^N,,,,,,  HNNNr max_bytesc|dS|jd}|dS t|}n#t$rYdSwxYw||kr*|t d|d|ddS)NzContent-Lengthzdownload exceeds size cap: z >  bytes)ror$rTr4rr)rrdeclaredsizes r_enforce_content_lengthrs##$455H8}}  iPdPPyPPPQQQs4 AA chunk_sizec#Kd}||D]J}|s|t|z }|-||kr'|td|d|VKdS)z Yield the response body in chunks, aborting with SSRFError if the running total exceeds `max_bytes`. Guards against a lying/absent Content-Length. r)rNzdownload exceeds size cap: >r) iter_contentlenrr)rrrtotalchunks riter_content_cappedr"s E&&*&==   U  UY%6%6 NN   L9LLLMM M rcd}|D]s}t|} |||d||}nA#t$r} t|| }Yd} ~ Md} ~ wt$rt|wxYw||fcS|t d||)u GET `url` pinned to each validated IP in turn, returning the first that connects. A hostname commonly resolves to several addresses (dual-stack AAAA + A), and the first one may be unroutable from the worker — e.g. an IPv6 address on a pod with no IPv6 route. Plain requests walks the address list; pinning to a single IP would turn that into a hard download failure, so this preserves the fallback while keeping every candidate pre-validated. Only connection-level errors advance to the next address; an HTTP response (any status) or any other exception is returned/raised immediately. Returns the (session, response) pair so the caller owns teardown. Raises the last connection error if no address is reachable. NF)roallow_redirectsstreamtimeoutz&no candidate addresses to connect to: )rr$RequestsConnectionErrorr BaseExceptionr) rNrXrorr last_errorr)rrrMs r_get_via_first_reachable_ipr2sJ!!'++ {{ % #HH'    "7 + + +J HHHH    "7 + + +      FFFGGG s2 A0AA0T)rrror max_redirectsrrrorc|t}|}t|dzD]M}t|}|jtvrt d|jd||jst d|t||jp|jdkrdnd} t|j| } t| |||| \} } | j tvry| j d } | t!| n#t!| wxYw| st d |t#|| }+t%| | t'| || cSt d |d |)a Fetch a user-supplied URL with SSRF protections. Enforces an http(s) scheme allowlist, resolves and validates the host to a public IP, pins the connection to that IP, and re-validates every redirect hop. Rejects a body whose declared size exceeds the cap. Returns the (streamed) requests.Response for the caller to consume; use iter_content_capped to enforce the cap while reading. Raises SSRFError on any block. Nzblocked URL scheme rEzURL has no host: riP)rorrLocationz"redirect without Location header: zexceeded maximum redirects (z): )rWrangerrt_ALLOWED_SCHEMESrr`r[rBrPr status_code_REDIRECT_STATUSror$rrrrr)rXrrrorrcurrent_rarBrNrrlocations rsafe_getrZs&&(( G =1$ % %'"" = 0 0 0N&-NNWNNOO O ;999:: :7###{Gfmw&>&>ssB"6?D997 '&'     #3 3 3'++J77H 0   &w////&w//// P NW N NOOOgx00G  "(G444)444 J=JJSJJ K KKs 2DD')5rr2r"rG contextlibrtypingrrr urllib.parserrrequestsr rrequests.adaptersr requests.utilsr r runpod.http_clientr r frozensetr_DEFAULT_MAX_REDIRECTSrRrS ip_networkr7r%r4rboolr(rr@rTrPrWr[rbrdrrrrbytesrrrvrrrrrs  ++++++++++********??????))))))<<<<<<<<000000$9677-"09/@@A:        $Ds#$s)B 6C 6 6 6 6 C D    2##&$$$$$$$$$$k$$$$$$NS-> $5 $     1B t    , R# R4 R R R R c hsm PXY^P_     %#%%%%V"#/7L7L7L 7L 7L 7L d^ 7L } 7L7L7L7L7L7L7Lr