HollowSearchBot

HollowSearchBot is the crawler that collects pages for Hollow Search. Its requests carry this user agent:

HollowSearchBot/2.0 (+https://search.projecthollow.com/bot)

What it crawls

• The home pages of widely linked sites, chosen from the Common Crawl host rankings.
• Sites on our curated list, which it follows more deeply and revisits on a schedule.
• The public RSS and Atom feeds of news publishers, for the News tab. It reads each feed once an hour with a conditional request, so an unchanged feed costs the publisher a single small reply, and keeps only the headline, a short summary and the link, for a week.

It fetches public pages only. It does not sign in, submit forms or run scripts.

The rules it follows

• robots.txt, read under the HollowSearchBot user agent or, when there is no group for it, *. If robots.txt returns a server error, it crawls nothing on that site until it can read the file.
• Crawl-delay in robots.txt. Without one, it makes at most one request per second to a site.
• noindex, in a robots meta tag or an X-Robots-Tag header: the page is not indexed.

How to limit or block it

To keep HollowSearchBot out of your whole site, add this to your robots.txt:

User-agent: HollowSearchBot
Disallow: /

Use Disallow with a path to block part of a site, or Crawl-delay: 10 to slow it down. Changes take effect on its next visit; pages already indexed drop out at the next weekly rebuild.

Publishers

A publisher whose robots.txt disallows its feed is not read at all. To be removed from News, or to have a site, a page or a picture delisted, email search@projecthollow.com with the address; a news publisher is removed together with what is already held at the next hourly update. Pictures come from Wikimedia Commons, not from your site.

Contact

If HollowSearchBot is causing a problem, email search@projecthollow.com with your site and, if you can, the time and the address it requested.