Bot Internet Indexing

What Bots Index Websites?

This page shows the bots visiting our secret root domain endpoint between 30th October 2022 and 30th October 2023. To make it harder to find, the domain name was an obscure random string of letters and numbers on a rarely used TLD. The website loaded JavaScript, Images and Media and premium web technologies to determine how deep the site was being indexed.

Where are the competitors? ๐Ÿ˜ณ

Since our secret website was created on 30th October 2022 not a single company that considers themselves a technology indexing competitor found the website in the 365 days the experiment ran for.

 

BuiltWith found the website within 5 days of the site existing without any insider knowledge from ourselves. This means if you were using a competitor for technology tracking it will miss sites by a very, very large margin if it finds them at all.

AHREFs and Bing are excluded from the results as they indexed the site multiple times per day.

SEMRUSH has visited our secret endpoint 86 times since the website was created on 30th October 2022, 364 days ago. That is on average once every 4 days.

SEMRUSH's first visit was 25 days after the site was made live.

October 2023
SEMRUSH

https://semrush.com
SEO Toolset

47 days, 11 hours, 47 minutes, 44 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

48 days, 18 hours, 43 minutes, 29 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

84 days, 22 hours, 34 minutes, 25 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

89 days, 3 hours, 47 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

121 days, 4 hours, 59 minutes, 45 seconds ago
June 2023
SEMRUSH

https://semrush.com
SEO Toolset

128 days, 16 hours, 42 minutes, 44 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

173 days, 8 hours, 27 minutes, 28 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

173 days, 8 hours, 28 minutes, 16 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

173 days, 9 hours, 59 minutes, 46 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

173 days, 9 hours, 59 minutes, 47 seconds ago
May 2023
SEMRUSH

https://semrush.com
SEO Toolset

173 days, 14 hours, 9 minutes, 18 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

173 days, 14 hours, 9 minutes, 34 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

173 days, 17 hours, 2 minutes, 22 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

173 days, 17 hours, 2 minutes, 23 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

174 days, 3 hours, 54 minutes, 45 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

174 days, 3 hours, 54 minutes, 45 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

174 days, 10 hours, 7 minutes, 3 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

174 days, 10 hours, 7 minutes, 8 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

174 days, 14 hours, 18 minutes, 9 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

174 days, 14 hours, 18 minutes, 10 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

174 days, 16 hours, 8 minutes, 22 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

174 days, 16 hours, 8 minutes, 23 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

175 days, 3 hours, 31 minutes, 58 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

175 days, 6 hours, 7 minutes, 4 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

175 days, 6 hours, 7 minutes, 4 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

175 days, 10 hours, 52 minutes, 47 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

175 days, 10 hours, 52 minutes, 48 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

188 days, 14 hours, 20 minutes, 34 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

188 days, 14 hours, 20 minutes, 37 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

224 days, 11 hours, 56 minutes, 51 seconds ago
March 2023
SEMRUSH

https://semrush.com
SEO Toolset

224 days, 11 hours, 56 minutes, 51 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

232 days, 20 hours, 55 minutes, 9 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

232 days, 20 hours, 55 minutes, 10 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 2 hours, 16 minutes, 3 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 2 hours, 16 minutes, 4 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 8 hours, 53 minutes, 46 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 8 hours, 53 minutes, 47 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 11 hours, 14 minutes, 52 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 11 hours, 14 minutes, 52 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 12 hours, 35 minutes, 43 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 12 hours, 35 minutes, 46 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 22 hours, 1 minute, 59 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 22 hours, 48 minutes, 45 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

233 days, 22 hours, 48 minutes, 47 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

234 days, 5 hours, 59 minutes, 31 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

234 days, 6 hours, 6 minutes, 9 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

234 days, 6 hours, 6 minutes, 10 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

234 days, 12 hours, 17 minutes, 2 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

234 days, 12 hours, 27 minutes, 29 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

234 days, 12 hours, 27 minutes, 36 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

257 days, 7 hours, 37 minutes, 2 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

257 days, 7 hours, 37 minutes, 2 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

294 days, 14 hours, 14 minutes, 10 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

294 days, 14 hours, 14 minutes, 10 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

294 days, 18 hours, 28 minutes, 7 seconds ago
January 2023
SEMRUSH

https://semrush.com
SEO Toolset

294 days, 18 hours, 28 minutes, 7 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

294 days, 20 hours, 21 minutes, 38 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

294 days, 20 hours, 21 minutes, 38 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

294 days, 22 hours, 59 minutes ago
SEMRUSH

https://semrush.com
SEO Toolset

294 days, 22 hours, 59 minutes, 1 second ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 1 hour, 21 minutes, 42 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 1 hour, 21 minutes, 49 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 4 hours, 25 minutes, 19 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 4 hours, 25 minutes, 19 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 10 hours, 54 minutes, 26 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 10 hours, 54 minutes, 26 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 12 hours, 23 minutes, 17 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 12 hours, 47 minutes, 30 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 12 hours, 47 minutes, 31 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 17 hours, 40 minutes, 6 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 17 hours, 40 minutes, 7 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 18 hours, 43 minutes, 7 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

295 days, 18 hours, 43 minutes, 7 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

297 days, 17 hours, 35 minutes, 56 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

297 days, 17 hours, 35 minutes, 56 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

300 days, 6 hours, 4 minutes, 6 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

302 days, 9 hours, 8 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

337 days, 15 hours, 51 minutes, 31 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

337 days, 22 hours, 38 minutes, 15 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

338 days, 3 hours, 55 minutes, 5 seconds ago
November 2022
SEMRUSH

https://semrush.com
SEO Toolset

338 days, 7 hours, 28 minutes, 54 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

338 days, 8 hours, 57 minutes, 34 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

338 days, 12 hours, 4 minutes, 1 second ago
SEMRUSH

https://semrush.com
SEO Toolset

338 days, 14 hours, 45 minutes, 42 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

339 days, 8 hours, 57 minutes, 21 seconds ago
SEMRUSH

https://semrush.com
SEO Toolset

339 days, 8 hours, 57 minutes, 22 seconds ago
Site Launched on 30 October 2022