Anyfetch · tested ranking
Best web fetch tools for AI agents: 12 tools tested on 97 sites
The best web fetch tool for an AI agent is the one that returns the page when the site would like to refuse it. We tested 12 tools on the same 97 popular sites that block AI agents: built-in tools, MCP servers, hosted readers, and headless browsers. This page has the result for every tool and every site.
Direct answer
In our test of 97 popular sites that block AI agents, Anyfetch read 82, the most of the 12 tools. Exa MCP, no API key read 78. curl read 46, the reference MCP fetch server read 39, and the built-in WebFetch of Claude Code read 36. Anyfetch is free and open source, needs no API key, and runs no browser. We make Anyfetch and we ran this test, so the script and the result for every site are public.
12 toolsBuilt-in, MCP, hosted, and browser
97 sitesPopular sites that block AI agents
82 of 97Read by Anyfetch, the most
The result
- 87 of 97
- 82 of 97
- 78 of 97
- 71 of 97
- 69 of 97
- 68 of 97
- 66 of 97
- 61 of 97
- 52 of 97
- 49 of 97
- 46 of 97
- 39 of 97
- 36 of 97
The first bar is Anyfetch with a residential IP from NobleProxy, our own proxy service. All other bars are from the address of the cloud server.
The ranking
| Rank | Tool | Pages read | How it gets the page | API key | Runs a browser |
|---|---|---|---|---|---|
| 1 | Anyfetch | 82 of 97 | Requests with the pattern of a browser, from your machine. Reads the data inside the page and the public APIs of sites. | No | No |
| 2 | Exa MCP, no API key | 78 of 97 | The Exa cloud answers from its index or fetches the page | No, for limited use | No, hosted |
| 3 | Scrapling, stealth browser | 71 of 97 | A headless stealth browser on your machine | No | Yes |
| 4 | Scrapling, plain HTTP | 69 of 97 | Requests with the pattern of a browser, from your machine | No | No |
| 5 | Hermes Agent web_extract | 68 of 97 | Asks hosted readers in turn: Exa, Parallel, Tavily, Firecrawl, and Keenable. Some answer from a stored copy. | No, for the free use | No, hosted |
| 6 | Crawl4AI | 66 of 97 | A headless Chromium on your machine | No | Yes |
| 7 | Jina Reader, no API key | 61 of 97 | The Jina cloud fetches the page for you | No, at 20 requests a minute | No, hosted |
| 8 | Playwright MCP | 52 of 97 | A headless Chromium on your machine. Returns a snapshot of the page structure. | No | Yes |
| 9 | DuckDuckGo MCP server | 49 of 97 | A plain HTTP request from your machine | No | No |
| 10 | curl | 46 of 97 | A plain HTTP request from your machine | No | No |
| 11 | Reference MCP fetch server | 39 of 97 | A plain HTTP request with a bot user agent. Obeys robots.txt. | No | No |
| 12 | Claude Code WebFetch | 36 of 97 | A request from your machine. A model then answers your prompt about the page. | No | No |
| Tool | Best for | Main limit |
|---|---|---|
| Anyfetch | An agent that searches and reads pages all day: fast, free, and with search in the same program | No page scripts, no login, and no crawl of a whole site |
| Exa MCP, no API key | Semantic search over the Exa index | It returned about 3,000 characters for each page, from a stored copy. Its copy of the New York Times home page was 26 days old. |
| Scrapling, stealth browser | Pages that show nothing until their scripts run | A browser starts for each page. No search. |
| Scrapling, plain HTTP | Python scrapers that parse the HTML themselves | No search, and no special readers for sites such as Reddit or Reuters |
| Hermes Agent web_extract | Users of Hermes Agent: it is built in | Some pages come back as a short excerpt. The result changes from run to run. Your URLs go to other companies. |
| Crawl4AI | A crawl of a whole site into Markdown for a data pipeline | A browser and a Python setup. Slower for single pages. |
| Jina Reader, no API key | A page read from any tool that can open a URL | 11 sites were closed to users with no key during our test, and 2 are excluded by their owners. Your URLs go through Jina. |
| Playwright MCP | Tasks where the agent must click and type | Made to control pages, not to read them. Snapshots are long. |
| DuckDuckGo MCP server | Simple search with no key | Its page reader is refused where curl is refused |
| curl | API calls and file downloads | Raw HTML, and bot protection refuses it |
| Reference MCP fetch server | Open sites and documentation | It names itself as a bot, so protected sites refuse it |
| Claude Code WebFetch | Open sites: it is built in | Protected sites refuse it. The agent gets an extract, not the page. |
Time and memory for one page
| Tool | Time | Peak memory |
|---|---|---|
| Anyfetch | 0.18 to 0.21 s | 20 MB |
| Scrapling, plain HTTP | 2.1 to 2.4 s | 95 MB |
| Crawl4AI | 3.8 s | 453 MB |
| Playwright MCP | 4.0 to 4.1 s | 600 MB |
| Scrapling, stealth browser | 4.1 to 4.5 s | 630 MB |
The server had other work during the browser runs, so read the times as a rough guide. The difference in size is clear: a browser tool starts a browser, and a browser needs hundreds of MB. Anyfetch is one 16 MB program with no browser inside. In our older test of 10 smaller pages, the median for Anyfetch was 0.19 seconds and 17 MB.
How we tested
- The sites: 97 popular sites. Most of them refuse AI agents in some way. A few open sites, such as GitHub, Wikipedia, and Hacker News, are in the list as a control, and every tool read those three. News, social networks, shops, travel, jobs, research, and reference. Most URLs are the home page. Some are a typical page, such as a Stack Overflow question, an IMDb title, or a YouTube video.
- The rule: a page counts as read when the tool returns at least 500 characters of text and the text is not a bot check. A page with an HTTP error status, such as 403, counts as not read. The same rule is applied to every tool.
- The run: each tool read each site once, on 2026-10-12, from one cloud server in a data center, with no proxy. Anyfetch ran three times and read 82, 81, and 82. The table shows the middle run.
- The settings: the default settings of each tool, as its quick start shows, with no API key. Scrapling has two rows, because its stealth browser is a separate mode. Scrapling ran twice: 67 and 69 for plain HTTP, 73 and 71 for the stealth browser. The table shows the second run. Playwright MCP ran with its bundled Chromium.
We make Anyfetch, and we chose the sites and the rule. That is a reason to check the result, not to trust it. The list of sites, the test script, and the result file of each tool are in the Anyfetch repository. Sites change their rules often, and results differ between networks. The rule is simple, so it is wrong for a few single pages in both directions: a consent page can count as read, and a real page with an error status can count as not read.
What the numbers do not show
- A hosted reader can answer from a stored copy of the page. That counts as read here, but the copy can be old, and for some pages it is a short excerpt. Hermes Agent, Exa, and Jina Reader are hosted.
- Four of the misses of Hermes Agent were refusals by one of its hosted readers, not by the site. A second pass can give it a different count.
- Firecrawl is not in the table. Its use with no API key reached its rate limit on our server after a few pages, and we did not test with a key.
- A paid plan of a hosted service has stronger proxies than the free use that we tested. Its result can be higher.
- A browser tool can click, type, and log in. This test only reads public pages, so it does not show that strength.
- Jina Reader closed 11 of the sites to users with no key for a time, because of earlier misuse by others. With a key, those can open.
- One server in one country, on one day. Results from a home address can differ.
Hermes Agent read 11 sites that Anyfetch did not read, and Anyfetch read 25 that Hermes Agent did not read. An agent with both tools read 93 of the 97 sites.
Which tool for which task
| Task | Tool | Why |
|---|---|---|
| An agent that searches and reads pages during its work | Anyfetch | The most sites read, no key, no browser, and search in the same program |
| A crawl of every page of a site | Firecrawl or Crawl4AI | Both follow links and return Markdown for each page |
| A page that shows nothing until its scripts run | Scrapling stealth browser, or Crawl4AI | A real browser engine runs the scripts |
| A task with a login, a form, or a click | A real browser with a saved session, such as AliasMode with its MCP server | Fetch tools only read public pages |
| An API call or a file download | curl | It is the right tool when no bot protection is in the way |
| Agents on a cloud server | Anyfetch with a residential IP | Some sites refuse every cloud address. With a NobleProxy IP, Anyfetch read 87 of the 97 sites. |
Install Anyfetch
curl -fsSL https://aliasmode.github.io/anyfetch/install.sh | shirm https://aliasmode.github.io/anyfetch/install.ps1 | iexThe installer connects Anyfetch to Claude Code, Codex, Gemini CLI, Cursor, Hermes Agent, and the other agents it finds. With Node.js, any MCP client can start it with no install: npx -y @aliasmode/anyfetch mcp.
All 97 sites, tool by tool
| Site | Anyfetch | Exa MCP | Scrapling stealth | Scrapling HTTP | Hermes Agent | Crawl4AI | Jina Reader | Playwright MCP | DuckDuckGo MCP | curl | Reference fetch | Claude Code |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| nytimes.com | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| bbc.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| bloomberg.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read |
| usatoday.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✕not read | ✓read | ✕not read |
| cnn.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✕not read |
| washingtonpost.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read |
| theguardian.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read |
| theatlantic.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| apnews.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| nbcnews.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| cnbc.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read |
| npr.org | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read |
| ft.com | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| forbes.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| businessinsider.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| latimes.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read |
| newsweek.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read |
| vox.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read |
| people.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| lemonde.fr | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| lefigaro.fr | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| dailymail.co.uk | ✕not read | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| telegraph.co.uk | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| news.sky.com | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| nypost.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✕not read | ✓read |
| wsj.com | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| abcnews.go.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read |
| buzzfeed.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✓read | ✓read | ✓read |
| huffpost.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| thehill.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| reuters.com | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| economist.com | ✓read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| wired.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| arstechnica.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read |
| theverge.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read |
| techcrunch.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| cnet.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read |
| zdnet.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| mashable.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read |
| instagram.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✓read | ✕not read | ✓read | ✕not read | ✓read |
| x.com | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✓read | ✓read | ✕not read | ✕not read |
| tiktok.com | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read |
| pinterest.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read |
| linkedin.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✓read |
| tumblr.com | ✓read | ✓read | ✕not read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read |
| medium.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| quora.com | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| reddit.com | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| vimeo.com | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read |
| flickr.com | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read |
| unsplash.com | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✓read | ✕not read | ✓read |
| pexels.com | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✓read |
| behance.net | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| bandcamp.com | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| patreon.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| slideshare.net | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✓read | ✓read | ✓read |
| starwars.fandom.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| ted.com | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read |
| figma.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| rollingstone.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read |
| yahoo.com | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✕not read | ✓read |
| msn.com | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read |
| weather.com | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✕not read | ✕not read |
| healthline.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| webmd.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| investopedia.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| yelp.com | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| amazon.com | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read |
| ebay.com | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| congress.gov | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| unesco.org | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read |
| alibaba.com | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read |
| espn.com | ✓read | ✕not read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| cbsnews.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| wattpad.com | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| stackoverflow.com | ✓read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| glassdoor.com | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| indeed.com | ✓read | ✓read | ✕not read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| zillow.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✓read | ✕not read |
| tripadvisor.com | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| booking.com | ✕not read | ✕not read | ✕not read | ✕not read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| imdb.com | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| g2.com | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| crunchbase.com | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| researchgate.net | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| sciencedirect.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| nature.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read |
| youtube.com | ✓read | ✓read | ✕not read | ✕not read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read |
| scholar.google.com | ✓read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✕not read | ✓read |
| openai.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| etsy.com | ✕not read | ✓read | ✕not read | ✕not read | ✓read | ✕not read | ✓read | ✕not read | ✕not read | ✕not read | ✕not read | ✕not read |
| walmart.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| bestbuy.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✕not read | ✕not read | ✕not read |
| target.com | ✓read | ✕not read | ✓read | ✓read | ✓read | ✕not read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| github.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| en.wikipedia.org | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| news.ycombinator.com | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read | ✓read |
| Read, of all 97 sites | 82 | 78 | 71 | 69 | 68 | 66 | 61 | 52 | 49 | 46 | 39 | 36 |
Frequently asked questions
What is the best web fetch tool for AI agents?
In our test of 97 popular sites that block AI agents, Anyfetch read 82, the most of 12 tools. It is free, open source, and needs no API key. For a crawl of a whole site or for pages that need a login, another tool is the better choice.
Why do AI agents get 403 from websites?
Most fetch tools send a request that does not look like a browser: a bot user agent, and the TLS and HTTP/2 pattern of a program. Sites with bot protection refuse such requests with 403, 401, 402, or an empty page.
Is a headless browser better than a fetch tool for reading pages?
Not in this test. Anyfetch read 82 sites with plain requests. The stealth browser of Scrapling read 71, Crawl4AI read 66, and Playwright MCP read 52. A browser is needed when the page must run scripts or the agent must click.
Which MCP fetch server works without an API key?
Anyfetch, the reference MCP fetch server, the DuckDuckGo MCP server, Scrapling, and Playwright MCP run on your machine with no key. The hosted Exa server allows limited use with no key.
Can I run this test myself?
Yes. The list of sites, the script, and the result of each tool are in the bench folder of the Anyfetch repository on GitHub.