Bot Internet Indexing

What Bots Index Websites?

This page shows the bots and automated agents visiting our secret root domain endpoint since 18 June 2026. To make it harder to find, the domain name was an obscure random string of letters and numbers on a rarely used TLD. The website loaded JavaScript, Images and Media and premium web technologies to determine how deep the site was being indexed.

Where are the competitors? ๐Ÿ˜ณ

Since our secret website was created on 18th June 2026, not a single company that considers themselves a technology indexing competitor found the website in the 109 days the experiment has run for.

 

BuiltWith found the website within 2 days of the site existing without any insider knowledge from ourselves. This means if you were using a competitor for technology tracking it will miss sites by a very, very large margin if it finds them at all.

424 identified user agents made 121,950 requests across 74,056 sessions from 10,316 IP addresses. Data last generated 05 October 2026 00:20 UTC.

anthropic.com was identified through 45 user agents, making 1,889 requests from 80 IP addresses.

Its first visit was 1 hour, 31 minutes, 43 seconds after the site was made live.

October 2026
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/147.0.2579.34 Safari/537.36

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.4992.118 Safari/537.36 Edg/131.0.4992.118; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.1981.4 Mobile Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.211.19 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Chrome/85.0.7775.97 Mobile Safari/537.36

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Chrome/131.0.4065.144 Mobile Safari/537.36

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.2852.228 Mobile Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:149.13) Gecko/20100101 Firefox/149.13; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.6707.118 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5483.216 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot) Chrome/151.0.8341.51 Safari/537.36

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Chrome/124.0.7846.145 Safari/537.36 Edg/124.0.7846.145

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64; rv:121.6) Gecko/20100101 Firefox/121.6; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6176.102 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.4332.186 Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-User/1.0; +Claude-User@anthropic.com)

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.4312.22 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.13) Gecko/20100101 Firefox/121.13; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.1098.181 Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.7183.226 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.8328.75 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.1848.119 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.1126.222 Mobile Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/91.0.2234.205 Safari/537.36

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.8512.166 Safari/537.36 Edg/134.0.8512.166; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.8171.162 Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot) Chrome/85.0.8300.176 Safari/537.36

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/118.0.2246.25 Safari/537.36 Edg/118.0.2246.25

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Chrome/148.0.3976.71 Safari/537.36

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15; compatible; Claude-User/1.0; +Claude-User@anthropic.com

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.2937.79 Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.137.158 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot) Chrome/130.0.7044.228 Safari/537.36

84 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

2 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-Web/1.0; +https://www.anthropic.com)

0 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +mailto:support@anthropic.com

0 days since site launch
anthropic.com
anthropic.com Artificial Intelligence

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

0 days since site launch
June 2026Site Launched on 18 June 2026