Fetching user-supplied URLs safely on shared hosting
By Huzaifa Awais ·
Any feature that says “enter a URL and we will fetch it” turns your server into a proxy. If it is careless, an attacker can point it at addresses only your server can reach: the hosting panel on localhost, other customers’ machines on a private network, or a cloud metadata service that hands out credentials. That class of bug is server-side request forgery, or SSRF.
The security-headers checker on this site has an optional fetch mode, backed by a single PHP file. These are the rules it follows.
1. Allow, don’t deny
Only http and https are accepted, only on ports 80 and 443, with no username or password in the URL. Everything else, file:, gopher:, odd ports, is refused before any network activity happens.
2. Resolve the name yourself, then check every address
A hostname can resolve to anything, including 127.0.0.1. The endpoint resolves both IPv4 and IPv6 records and rejects the request if any of them is loopback, private, link-local, carrier-grade NAT, multicast, reserved or a metadata address such as 169.254.169.254.
3. Pin the connection to the checked address
Checking the DNS answer and then letting the HTTP library resolve the name again leaves a gap: a hostile DNS server can answer differently the second time (DNS rebinding). The endpoint passes the already-validated IP to cURL with CURLOPT_RESOLVE, so the connection goes exactly where the check said it would.
4. Follow redirects by hand
Automatic redirect following would skip all of the above for the second hop. Redirects are followed manually, at most three times, and every new location goes through the same validation.
5. Limit everything else
- Only a HEAD-style request for headers; the response body is never read.
- Short connect and total timeouts.
- A per-visitor rate limit stored in a file outside the web root, keyed by a hash of the IP address.
- Fixed, generic error messages, so the endpoint cannot be used to map which internal ports are open.
None of this makes the endpoint perfectly safe, no code is. It does mean the obvious attacks fail, and the tests in the repository try each of them on every build.