Bot Internet Indexing

What Bots Index Websites?

This page shows the bots and automated agents visiting our secret root domain endpoint since 18 June 2026. To make it harder to find, the domain name was an obscure random string of letters and numbers on a rarely used TLD. The website loaded JavaScript, Images and Media and premium web technologies to determine how deep the site was being indexed.

Where are the competitors? 😳

Since our secret website was created on 18th June 2026, not a single company that considers themselves a technology indexing competitor found the website in the 64 days the experiment has run for.

 

BuiltWith found the website within 2 days of the site existing without any insider knowledge from ourselves. This means if you were using a competitor for technology tracking it will miss sites by a very, very large margin if it finds them at all.

183 identified user agents made 84,857 requests across 53,653 sessions from 8,926 IP addresses. Data last generated 20 August 2026 00:20 UTC.

developers.facebook.com was identified through 6 user agents, making 25 requests from 22 IP addresses.

Its first visit was 19 hours, 35 minutes, 3 seconds after the site was made live.

August 2026
developers.facebook.com
developers.facebook.com Social Media

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

55 days since site launch
developers.facebook.com
developers.facebook.com Social Media

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

30 days since site launch
developers.facebook.com
developers.facebook.com Social Media

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

30 days since site launch
developers.facebook.com
developers.facebook.com Social Media

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

30 days since site launch
developers.facebook.com
developers.facebook.com Social Media

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

27 days since site launch
July 2026
developers.facebook.com
developers.facebook.com Social Media

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

0 days since site launch
Site Launched on 18 June 2026