Anyfetch · comparison
curl alternative for AI agents: read the pages that answer curl with 403
A curl alternative for AI agents must do two things that curl does not do: get an answer from sites that refuse curl, and return text that an agent can use. This page shows why curl gets 403, what Anyfetch returns for the same URL, and the command that replaces curl in your agent.
Direct answer
Anyfetch is a free, open-source alternative to curl for AI agents. An agent that runs curl gets 403 from many sites, because the TLS handshake and the HTTP/2 settings of curl are not those of a browser. Anyfetch sends the request a browser sends and returns the page as Markdown. In our test of 97 popular sites that block AI agents, curl read 46 and Anyfetch read 82, from the same server and the same address. The command is: anyfetch fetch <url> --format markdown.
The same URL with curl and with Anyfetch
$ curl -sL -o /dev/null -w "%{http_code}\n" \
https://www.reuters.com/technology/
401$ anyfetch fetch https://www.reuters.com/technology/
{
"status": 200,
"title": "Tech News | Today's Latest Technology News | Reuters",
"total_chars": 26126,
"content": "Strong investment in AI infrastructure ..."
}Both commands ran on the same server, one after the other, on 2026-10-12. Nothing else changed: no proxy, no cookies, no API key.
Why sites answer curl with 403
A site with bot protection decides before it reads your headers. It looks at the TLS handshake: the list of ciphers and extensions, and their order. Each browser has one known pattern. curl has a different one, which comes from its TLS library. Then the site looks at the HTTP/2 settings and at the order of the headers. curl differs there too.
That is why curl -A with the name of Chrome changes so little. The flag changes one header, and the handshake still says curl. We measured it: with the user agent of Chrome, curl read 52 of the 97 sites, 6 more than plain curl and 30 fewer than Anyfetch.
In our test, curl missed 51 of the 97 sites. 33 answered 403, 4 answered 402 (“payment required”, a request that AI crawlers pay), 2 answered 429, 2 answered 401 or 999, and 2 closed the connection. 8 more sent a page with almost no text, because the content is built by scripts.
- 82 of 97
- 46 of 97
We read each of 97 popular sites once with each tool, on 2026-10-12, from one cloud server with no proxy. A page counts as read when the tool returns at least 500 characters of text and the text is not a bot check. Most of the sites refuse AI agents in some way. A few open sites, such as GitHub and Wikipedia, are in the list as a control. We make Anyfetch and we ran this test, so the list of sites, the script, and the result for every site are public. Run it yourself.
Sites that refuse curl and answer Anyfetch
Anyfetch read 36 sites that curl could not read: nytimes.com, bloomberg.com, usatoday.com, washingtonpost.com, theatlantic.com, apnews.com, cnbc.com, ft.com, people.com, thehill.com, reuters.com, economist.com, arstechnica.com, linkedin.com, tumblr.com, medium.com, quora.com, reddit.com, vimeo.com, pexels.com, behance.net, starwars.fandom.com, yahoo.com, investopedia.com, ebay.com, unesco.org, stackoverflow.com, glassdoor.com, indeed.com, zillow.com, imdb.com, sciencedirect.com, youtube.com, scholar.google.com, openai.com, bestbuy.com.
curl read no site that Anyfetch could not read.
| Site | Anyfetch | curl |
|---|---|---|
| nytimes.com | ✓read | ✕not read |
| reuters.com | ✓read | ✕not read |
| bloomberg.com | ✓read | ✕not read |
| wsj.com | ✕not read | ✕not read |
| washingtonpost.com | ✓read | ✕not read |
| ft.com | ✓read | ✕not read |
| economist.com | ✓read | ✕not read |
| forbes.com | ✓read | ✓read |
| reddit.com | ✓read | ✕not read |
| x.com | ✓read | ✓read |
| linkedin.com | ✓read | ✕not read |
| instagram.com | ✓read | ✓read |
| tiktok.com | ✕not read | ✕not read |
| quora.com | ✓read | ✕not read |
| medium.com | ✓read | ✕not read |
| youtube.com | ✓read | ✕not read |
| amazon.com | ✓read | ✓read |
| ebay.com | ✓read | ✕not read |
| walmart.com | ✓read | ✓read |
| etsy.com | ✕not read | ✕not read |
| booking.com | ✕not read | ✕not read |
| tripadvisor.com | ✕not read | ✕not read |
| yelp.com | ✕not read | ✕not read |
| zillow.com | ✓read | ✕not read |
| indeed.com | ✓read | ✕not read |
| glassdoor.com | ✓read | ✕not read |
| stackoverflow.com | ✓read | ✕not read |
| github.com | ✓read | ✓read |
| en.wikipedia.org | ✓read | ✓read |
| imdb.com | ✓read | ✕not read |
| sciencedirect.com | ✓read | ✕not read |
| scholar.google.com | ✓read | ✕not read |
| Read, of all 97 sites | 82 | 46 |
What an agent needs that curl does not give
| Need | curl | Anyfetch |
|---|---|---|
| Text the model can read | Raw HTML. One Wikipedia article is 236,000 bytes. | Markdown. The same article is 46,000 characters. |
| Only the part that answers a question | No | Yes. Add --query, and the result has only the matching parts, 5,000 characters by default. |
| Search the web | No | anyfetch search, with no API key |
| PDFs | A binary file | Text, page by page |
| Pages built by scripts | An empty page | The data inside the page, or the public API of the site, for Reddit, Stack Overflow, IMDb, YouTube, and others |
| A clear failure | An HTTP code, or a block page that looks like content | An error code such as challenge_required, http_error, or rate_limited |
Replace curl in your agent
curl -fsSL https://aliasmode.github.io/anyfetch/install.sh | shirm https://aliasmode.github.io/anyfetch/install.ps1 | iex| Task | With curl | With Anyfetch |
|---|---|---|
| Read a page | curl -sL <url> | anyfetch fetch <url> --format markdown |
| Read several pages | One curl for each URL | anyfetch fetch <url> <url> <url> |
| Find one fact on a long page | curl -sL <url> | grep ... | anyfetch fetch <url> --query "the question" |
| Read a PDF | curl -sLO <url>, then a PDF tool | anyfetch fetch <url> |
| Search | Not possible | anyfetch search "the query" |
An agent uses the tool that its instructions name. Put these lines in AGENTS.md, CLAUDE.md, or the instruction file of your harness:
## Web access
- Do not use curl to read web pages.
- Search the web: `anyfetch search "<query>"`
- Read pages: `anyfetch fetch <url> [<url>...] --format markdown`Does your harness support MCP? Then the agent gets the same functions as tools, with no shell command. The installer connects Anyfetch to Claude Code, Codex, Gemini CLI, Cursor, and the other harnesses it finds.
When curl is still the right tool
- An API call with a token, a POST request, or your own headers. Anyfetch only reads public pages with GET.
- A file download. Anyfetch returns text, not files.
- A server of your own, where no bot protection is in the way.
- A full copy of the HTML for a parser. Anyfetch returns the content, not the markup.
Another option is curl-impersonate, a build of curl that sends the handshake of a browser. It keeps the curl command line and returns raw HTML. It has no search and no Markdown, and we did not measure it.
On a server, some sites refuse every cloud address, whatever the request looks like. A residential IP opens those. With one NobleProxy address, Anyfetch read 87 of the 97 sites.
Frequently asked questions
Why does curl get 403 when the browser shows the page?
The site compares the request with the pattern of a real browser: the TLS handshake, the HTTP/2 settings, and the order of the headers. curl has its own pattern, so a site with bot protection answers 403.
Does changing the curl user agent fix 403?
On a few sites. In our test of 97 sites, curl with the user agent of Chrome read 52, against 46 for plain curl and 82 for Anyfetch. The -A flag changes one header. The TLS handshake and the HTTP/2 settings stay those of curl.
What is the best curl alternative for AI agents?
It depends on the task. To read public web pages as text, use a tool that sends a browser request and returns Markdown. In our test of 97 sites that block AI agents, Anyfetch read 82 and curl read 46. For API calls and downloads, curl is still the right tool.
Does Anyfetch run a browser?
No. Anyfetch sends plain requests with the pattern of a browser. One page takes about 0.19 seconds and 17 MB of memory. A headless Chromium took 1.15 seconds and 544 MB in the same test.
Can Claude Code or Codex use Anyfetch in place of curl?
Yes. Both can use it as an MCP server, with the tools web_fetch and web_search. They can also run the anyfetch command in the shell.