If your site is hostile to bots, your site is hostile to humans too.

The old model

Previously, all bots were considered bad actors, scrapers, competitors, spammers; but there was a distinction for the good bots, the search engine bots.
So, you had to live with it and you had to adapt to filter the good from the bad.
But still, you were already ready to welcome bots – only the specific ones, only the search engines. You made a move to adapt to them specifically, to make them welcome, to make them really understand your site, to follow the best practices from the search engines, and so on.

Well, but for all others, not so much:
Everyone started building walls, CAPTCHAs, rate limits, IP blocks, and other weird stuff like geoblocking or, eventually, paywalls.
It became absurd. Everyone using a VPN could constantly face your stupid CAPTCHAs, even if they were really just humans who didn’t want to be tracked.

But the bots were mostly run by major corporations, and while you might have a personal grudge against them, the share of humans running personal scripts had actually been much smaller before. You had to have technical knowledge to write a parsing script in the first place, so the fraction of the population who were doing it independently was small.

That made sense when you wanted to have only valuable visitors on your site, meaning the humans who render it in a real browser and so on.
Not anymore.

The bots are the new interface.

Bots are now acting on behalf of humans, so if you block them, you block humans too.
Soon that’s going to be how most people access a lot of content.

Now everyone has their own AI secretary sitting between them and your content.
Nobody’s going to visit your site anymore – their agent will.
So, if you are hostile to such visitors, you are hostile to the humans behind them.

You are hostile to the average reader who wants to read the evening news and sends their personal openclaw to go and fetch the headlines to pick what is interesting.
If they can’t get a quick summary, they might not be interested in reading the whole thing.
If they cannot run the agent to get a quick understanding of what is being discussed, they won’t even bother looking at it at all.

So if you start blocking such bots, you basically block your audience.

A personal story. I wanted a quick way to search for flight tickets and not have to bother browsing a site, but to let my agent search programmatically and keep a memory of what it found.
I didn’t care which service. I just wanted to get in, search, and leave with the data.

Out of three services, only one actually worked. I signed up in about 10 seconds, got an API key pasted back into my chat, and was done. The moral? They got me as a customer without even trying to sell me anything.

But here’s the real trick – that company only provides an API, not a way to buy. But what if they did? I already have an API key in my storage. My agent already has memory of using their service. If I want to buy a ticket, I’m one keystroke away from being their customer.

That’s the new customer funnel: give them the API key, and your service lives in their agent’s memory forever. Don’t sell. Just let them use it. A new way to be remembered. A new way to bookmark yourself to your customers.

And here’s the kicker – compared to humans, agents have perfect memory. They’ll remember you a year from now. A human might forget your site in a week, but their agent? It still has the API key, still knows how to use your service, and is one call away from making you a returning customer.

The wrong response

Any attempt to strike license deals to prevent scraping, or to find a way to retain the income of an old era, is just luddites crying.
All of these solutions are irrelevant. It’s a game of whack-a-mole. You just can’t block them all. If you start making too many crazy CAPTCHAs, you just discourage the humans from visiting. If you try to block the bots, you’re making yourself invisible and all that.

Accessibility is binary – it’s either present or not; all the in-betweens are just going to annoy one side or the other.

You are already welcoming the Google bots because they are beneficial, but other bots you see as not beneficial. Well, that was before – as I said, they were stupid programmatic machines, and now they are most likely to actually have a human behind them.

You welcome the search bots because they send you traffic. Now you would want to welcome human operated bots because they send you the real customers. It may not look like it now, but it potentially could be the future.

If you cannot win, then lead.

You should take the initiative and prepare your site and its resources for programmatic access before they start doing it sideways.

Why are you against bots anyway? Typically, only because they are eating your resources, the ones humans would otherwise be using.
And why are they doing that? Because they’re using the same interface as humans. They have to render your whole site – all the layout, all the JavaScript, all the styling – when all you need to do is send the text out.

Give them RSS, give them an API, give them a way to utilize it more efficiently.

Compared to the old stupid scripts that companies run en masse, AI-powered bots are actually smart, and you can redirect them.
You can redirect them to more efficient resources, and you can command and help this process. You can give the link up front and provide guidance on how to navigate it all. And they would actually listen to your instructions, unlike humans.

For humans, you have to create a UI, polish it, think twice, iterate for days – and even then they might not understand how to use it.
With humans you have to think about a lot of edge cases: Will they understand it? Will they know what the hell that button does? Will they get lost?
With agents, you have no such problem. You just give them text, create a skill that explains how to use your website, and that’s it.

Your hardware would be thankful, and your users behind bots would be thankful.

Even the users who are not using bots directly to interact with your site would benefit from having a programmatic channel to access it or use it for help. If I’m faced with a complex website and I’m lost in the interface, I can just use the agent in my browser to look at your API and help me navigate, understand how to get things done, and complete my task faster.

You must make your site API-first, above all.

Allow them to search easily, allow them to understand and navigate easily. And this way, even if you might not see the same bot traffic, this structure could be used alongside a human-based interface to help humans navigate.

You already have a practice of adapting to bots. You already make your sites SEO-ready.
Why not go one step further and create an easy API and a marker for any agent to stumble upon real quick?
You already have files like robots.txt. There’s even llms.txt now.
Now you have to go one step further and provide them with the same functionality a user would have!

Give them the capacity to register real accounts, have access to all the content – queryable, searchable, easy, accessible, sortable. They would understand what they want straight away, without humans wasting time hunting for it. Much easier on your server’s load, and that makes everyone happy.

With the bots being the new interface, the sites that are reliable, that are accessible by them, will be remembered eventually.

You now have a short window of time to obtain a higher-quality audience, so to speak – the people who are on the frontier of technology, who are already starting to run their personal bots before it becomes mainstream. Typically those people are more tech-savvy, more educated, and more likely to buy your products if you make a direct offer.
But only if you make yourself friendly to their agents.

The resource problem we just can’t avoid.

Yes, you cannot avoid the resource problem – if you allow bots to run rampant, they’re going to squeeze all of your server’s capacity and you’ll basically become permanently DDOSed.
I’m not crazy enough to suggest just opening the gates and doing nothing, seeing how your CPU is constantly at 100% and everyone is screaming.
After all, half the internet is running on $5 VPSes and it’s crazy to expect everyone to be able to handle the increased load.

However, if you just block all the bots, you’re an asshole and you alienate the people who are living on the edge.

You just do it intelligently. Do the rate limits. Do the queues.
You may apply a soft CAPTCHA that just detects whether a browser is a real one running on a real operating system, to prioritize the real users who are likely running the agent in the browser along with everything else.

Yes, you cannot serve them all, but at the same time you must serve them all, so just make it accessible with limits.
You could tell them what the current rate limits are instead of just denying them straight away.
An agent can understand this – unlike a stupid script. You can tell it: "This site is accessible, you’re in a queue, come back in 5 minutes."
Or you could redirect them to another slow priority endpoint that you host.
Or, if you have an isolated copy or a secondary API just for them, you could suggest they access the database directly.

You can potentially create a priority queue with paid access. Before, it would be absolutely crazy to ask for payment data just to view your website. But now, everything can happen programmatically in an instant.

One idea: you have a private wallet and you can negotiate with a visitor to get into the priority queue or access extended functionality. Like a small smart contract for a temporary transaction – you expose your wallet, they send you a small amount, you return an API key to access. The key here is that those transactions should be small and atomic and not become the next level of subscription, because there’s a very low level of trust in the first place. They won’t just take your money and scam you, or change the conditions, tariffs, revoke it the next second and all that.

Of course I’m not suggesting you build paywalls instead, but that you use the tools to help manage the demand if that happens.

Being able to send instructions on how to act opens up many possibilities to manage the flood.

Prepare or be forgotten.

If you make web content inaccessible, then you become invisible. If you don’t make yourself accessible to humans running bots, then you are forgotten.
Same as not being indexed.

You may resist it, and you may have delusions that you’ll always be able to resist this way – humans-only, no AIs, whatever.
But prepare also to live in a future when humans are going to atrophy their capacity to visit the web and navigate anything like that.
I don’t believe in this future. However, I’m thinking more about the future where old static websites become obsolete because an agent can generate a website for your preference, query, find, and present information in the way you like.
So in the end, your static site not being flexible is just going to be like Web 1.0 compared to the modern web.

An agent running research across many sites will find that you’re one of a hundred, but inaccessible, and it will just skip you and move on.
How else are you going to be relevant when across a hundred search results you are not accessible and therefore you don’t get into the final output for the users?

All you need to do is fight back. Or rather: stop fighting back. If you start making CAPTCHAs and more aggressively blocking bots, you just hurt yourself and eventually you start working against your audience.

To work with your audience you have to prepare for the new interface, or else they’re going to leave for another place where there is less resistance.