How cloudfetch gets past Cloudflare

Go's HTTP client is turned away before it sees a single page. The reason is in the TLS handshake, so that is where the fix has to go.

Vĩ Lê2 November 2026About 6 minutes

Blocked on first contact

Point curl or Go's net/http at a site behind Cloudflare's bot protection and the first response is often not the page at all:

$ curl -sD - https://<protected-site>/
HTTP/2 403
content-type: text/html; charset=UTF-8
cf-mitigated: challenge
server: cloudflare

Setting a browser's User-Agent does not change this. Cloudflare has already decided by the time it reads the headers.

The check happens in the handshake

Every HTTPS connection starts with a TLS ClientHello: the cipher suites the client supports, the extensions it sends, and the order it sends them in. Each TLS library builds this message its own way, so it identifies the client the way a signature would. Fingerprints such as JA3 and JA4 reduce it to a short string that an edge network can compare against known browsers.

Go's crypto/tls sends a ClientHello that no browser sends. A request can claim to be Chrome in every header and still introduce itself as Go in the first message on the wire.

Reproducing a browser with uTLS

uTLS is a fork of Go's TLS stack that can build a specific browser's ClientHello byte for byte. cloudfetch dials the TCP connection itself, then hands it to uTLS with a browser preset:

tlsConn := utls.UClient(conn, &utls.Config{
	ServerName: host,
}, utls.HelloChrome_133)

The handshake is only half of the disguise. The headers have to describe the same browser at the same version: a Chrome 133 handshake with a Chrome 120 User-Agent is its own kind of tell. So each preset in cloudfetch pins both together. cloudfetch.Chrome is the Chrome 133 ClientHello plus Chrome 133's User-Agent, client hints and Sec-Fetch-* headers, and the same goes for Firefox 120, Safari 16.0 and Edge 85.

HTTP/2 on a connection net/http won't touch

Browsers negotiate HTTP/2 through ALPN during the handshake, so cloudfetch has to speak it too. The obvious route fails: net/http only switches to HTTP/2 when the connection is a *crypto/tls.Conn, and a uTLS connection is not one. Hand it over and net/http writes HTTP/1.1 onto a connection that just agreed to HTTP/2.

Instead, when ALPN picks h2, cloudfetch runs the HTTP/2 client from golang.org/x/net/http2 directly on the uTLS connection and keeps those connections in its own small cache, one per host. Connections nobody is using can be closed with CloseIdleConnections, the same method name net/http uses.

When a challenge appears anyway

Some sites still serve Cloudflare's classic "Checking your browser" page, which asks the client to solve a small piece of obfuscated arithmetic before it can continue. cloudfetch solves it without a browser:

• It parses the page with an HTML parser rather than regular expressions. Real challenge pages hide decoy form inputs inside HTML comments, and a parser ignores them.
• It runs the challenge script in goja, a JavaScript interpreter written in Go, with just enough of document stubbed out for the script to finish.
• It waits the delay the page asks for, then submits the answer with the form's other fields.

The solver is tested against real challenge pages recorded by the Python project cloudscraper, and it produces the known answer for all three that have one. Newer challenges are another matter. Turnstile and CAPTCHAs need a real browser, so cloudfetch reports them as ErrManagedChallenge and ErrCaptcha instead of pretending.

Result

On the same site that answered curl with a 403, cloudfetch's default Chrome profile got the real homepage on 10 out of 10 requests, with one reused client and with a fresh client each time. No challenge was solved along the way: the handshake alone was enough.

Using it looks like using net/http:

client := cloudfetch.New(cloudfetch.WithBrowser(cloudfetch.Firefox))

req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
	return err
}
resp, err := client.Do(req)

Where it stops

The HTTP/2 settings and header order still come from Go rather than from the browser, so a site that fingerprints those can still tell the difference. Making those match too is the next layer down.

The source, tests and examples are on GitHub.