We launched an empty website. Here’s who showed up.

Three AI crawlers. Three completely different jobs.

Here’s the part a hit counter can’t show you. Three big AI crawlers visited, all three verified against their operators’ own published IP ranges — and Frenemy recorded not just that they came, but what kind of thing each one fetched: articles, page scripts, or the rules files (robots.txt and the sitemap).

Same word — “bot” — in every analytics dashboard. Three completely different relationships with your site:

A search scout that checks the rules is not a training crawler that reads everything, and neither is a rule-watcher that reads almost nothing. You can’t decide how to treat visitors you can’t tell apart — and by name alone, you can’t.

One in eight visitors wore no name at all

About one in eight requests — from 31 different IPs, spread across the whole two weeks — arrived with no user-agent string whatsoever, probing for /wp-includes/wlwmanifest.xml, /actuator/env, /graphql/api, and other classic break-in paths. Stripping your identity is how you dodge tools that identify traffic by name. So the “unknown” slice of traffic isn’t a blind spot — it’s a category: the visitors actively trying not to be seen.

Alongside them, plain old curl hammered PHPUnit and Laravel remote-code-execution URLs within hours of launch. Nobody was told this site existed. The scanners found it anyway — that’s the background radiation of the internet, and it hits every address, including yours.

Everyone’s writing llms.txt. On our site, almost nobody read it

We published an llms.txt — the proposed “rules for AI” file the AI-search world is busy telling everyone to add. In 14 days it was fetched twice, both times by an agent we couldn’t verify. Over the same window, robots.txt and the sitemap were fetched hundreds of times, led by the biggest AI crawlers in the world.

One small site is one data point, not a verdict on the standard. But if you’re prioritizing where your crawler-facing effort goes, our data says the files the big crawlers actually poll — obsessively — are the boring old ones: robots.txt and your sitemap.

The one caught wearing someone else’s name

On day four, a request arrived wearing CCBot’s name — Common Crawl’s crawler — from an IP address that failed verification against Common Crawl’s published ranges. We didn’t block it; this site observes. We named it: a claimed identity that didn’t check out, on the record.

One catch in two weeks on a tiny site is about what you’d expect — the point isn’t volume. The point is that the name in a user-agent is just text anyone can type, and the only way to trust it is to check the connecting IP against the operator’s own published ranges. (The honest caveat, which we’ll keep making: a single unmatched hit can’t be perfectly distinguished from a legitimate crawler using an address its operator hasn’t published yet — which is exactly why the safe move is to name and record it, not reflexively block it.)

Five weeks on, the pattern is holding

The site is still up, still unannounced, and we kept watching after the 14-day window closed. Three more weeks brought roughly 390 more crawler requests — and the three crawlers stayed perfectly in character. ClaudeBot made 90 requests: every single one was robots.txt or the sitemap, zero content. GPTBot kept reading — about half its requests were articles, and it still pulled the page scripts. OAI-SearchBot stayed at 100% rules files.

No new impostors. The nameless scanners never stopped. And five weeks in, Google and Bing have still never visited — we still haven’t told them we exist, and unlike the AI crawlers, they apparently wait for an invitation.

What this means for your site

If a 12-page site nobody linked to drew 700+ crawler requests, full content coverage, a verified impostor, and a steady stream of nameless scanners in two weeks — your real site is being read, trained on, scouted, and probed right now. The question was never whether bots visit. It’s whether you can tell the friend (the search scout that can send you customers) from the frenemy (the training crawler taking your work) from the foe (the scanner, the impostor) — when every one of them files under “bot” in the tools you already have.

That’s what Frenemy does: every visitor named, every claimed identity verified against the operator’s own published ranges, each crawler’s real behavior on your site made visible — and, when you’re ready, a safe way to act on it.