Bot Internet Indexing

What Bots Index Websites?

This page shows the bots visiting our secret root domain endpoint between 30th October 2022 and 30th October 2023. To make it hard to find, the domain name was an obscure random string of letters and numbers on a rarely used TLD. The website loaded JavaScript, Images and Media to determine how deep the site was being indexed.

Why just known bots?

Only legitimate companies use known bots. Ethical companies require other ethical companies like ourselves to only index the web in an identifiable manner, using an identifiable bot user-agent and complying with robots.txt directives.

Why are no companies that consider themselves your competitors listed?

Since our secret website was created on 30th October 2022 not a single company that considers themselves a technology indexing competitor found the website in the 365 days the experiment ran for.

BuiltWith found the website within the first 5 days of it being launched without any insider knowledge from ourselves. This means if you were using a competitor for technology tracking it will miss sites by a very large margin.

AHREFs and Bing are excluded from the results as they indexed the site multiple times per day.

INTERNETMEASUREMENT has visited our secret endpoint 11 times since the website was created on 30th October 2022, 364 days ago. That is on average once every 33 days.

INTERNETMEASUREMENT's first visit was 32 days after the site was made live.

October 2023
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

41 days, 9 hours, 4 minutes, 20 seconds ago
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

73 days, 17 hours, 44 minutes, 6 seconds ago
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

101 days, 2 hours, 40 minutes, 50 seconds ago
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

131 days, 10 hours, 37 minutes, 38 seconds ago
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

161 days, 16 hours, 3 minutes, 21 seconds ago
May 2023
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

190 days, 11 hours, 14 minutes, 35 seconds ago
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

218 days, 4 hours, 57 minutes, 16 seconds ago
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

245 days, 9 hours, 19 minutes, 4 seconds ago
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

276 days, 5 hours, 49 minutes, 6 seconds ago
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

305 days, 17 hours, 35 minutes, 43 seconds ago
December 2022
INTERNETMEASUREMENT

https://driftnet.io
Web Security Services

331 days, 21 hours, 1 minute, 47 seconds ago
Site Launched on 30 October 2022