Machines, Money and Web Content
TL;DR: AI agents now consume more web content than ever, yet send almost no traffic back to publishers. Blocking them no longer works; charging them is fairer, cleaner, and finally practical thanks to x402 and stablecoin micropayments. We built a live Ghost paywall that machines can unlock with one HTTP request.
There’s a shift happening in plain sight: the dominant visitors to websites are no longer people, they’re bots. Intelligent agents. AI crawlers. Retrieval-Augmented Generation (RAG) bots like PerplexityBot and ChatGPT-User are scouring the web, consuming and synthesizing content for end users.
According to TollBit’s Q1 2025 report, RAG bot scraping grew 49% in just one quarter, surpassing traditional training crawlers. Some AI bots now account for up to 30% of all Googlebot-equivalent scraping volume.
These bots are voracious readers, but stingy with referrals. Despite massive content consumption, AI agents drive just 0.04% of external traffic back to publishers. Google, by comparison, drives over 85%.
The web was built on a contract: I link to you, you link to me. But machines don’t click links. They consume without engaging.
Free riders?
Historically, websites have tried to block bots using robots.txt. That worked... until it didn’t.
Bots now regularly bypass rules via masked agents, residential IPs, or third-party crawlers. In March 2025, over 26 million scrapes ignored robots.txt in a single month, and that's only counting Tollbit's users.
The default reaction has been defensive: detect and block. But blocking is brittle, adversarial, and increasingly ineffective.
There’s a better approach: charge them. If they’re deriving value from your data, they should contribute value in return.
It’s the digital equivalent of a toll road: open access, but paid.
Enter x402: HTTP-Native Paywalls
Released by Coinbase, x402 leverages stablecoins for processing programatic payments.
It works like this:
1) Your server returns 402 Payment Required.
2) Along with a header that includes payment details (amount, network,...).
3) The client (browser, script, or agent) programmatically pays.
4) Client retries including the header in the request and the server responds with200 OK.
No accounts. No cookies. No UI. Just HTTP.
Using stablecoins on bridge also means
- Global settlement in 2 seconds
- Payments of fractions of a cent (true micropayments)
- No credit card fees
- No banks
- No user signup
You can charge 0.1¢ to read an article, 1¢ to call an API, or $0.05 for high-value data. All directly from HTTP headers.
We finally have web-native micropayments.
We built a working prototype for Ghost
This blog actually implements machine-payable paywalls.
This post is free. The next one isn’t.
https://blog.proxy402.com/post/congratulations-youve-just-stepped-into-the-future/
If you pay (via browser or bot), you’ll see:
“Congratulations. You’ve just made your first machine-native payment on the internet.”
And then you’ll see the rest of the post.
A Web That Pays for Itself
The internet has always rewarded openness. But we’re entering an era where machines are the majority users and the economics must evolve.
Blocking them is unsustainable. Letting them in for free is unfair. Charging them is the path forward.
Fewsats lets us do that with the simplicity of HTTP and the power of stablecoins.
We’re not just imagining a machine economy. We’re building it.
