AmigadexBot

AmigadexBot is the crawler behind the Amigadex community web index. It makes the pages of Amiga websites searchable in one place and sends every visitor back to the original site.

How it behaves

• It identifies itself as AmigadexBot with a link to this page.
• It follows robots.txt, including Crawl-delay, and noindex / nofollow meta tags.
• It waits at least two seconds between two requests to the same site and reads a limited number of pages per site each week.
• It only downloads HTML pages: never archives, disk images or media.
• It keeps the page title and the first 20,000 characters of text, to build search results that link to you.

Where is the rest?

Amigadex does not have every page of every Amiga site yet, on purpose: it reads the web slowly, so that small sites on small servers never notice it. Today it knows 1,682 sites and has 25,247 of their pages; 2,168 sites found by following links are waiting for a first look, and 598 kept sites have no page read yet ("indexing to come").

• Each site gets at most 40 pages per visit (more for the big forums), one every 2 seconds or slower if its robots.txt asks, and a visit again after 7 days, newest pages first.
• Each run visits 40 sites and stops after 45 minutes, every 3 hours: the index fills up week after week, not in a day.
• Sites found by following links are checked first, and only kept when their pages talk about the Amiga.
• News, forum threads and releases do not wait for this: they come from the sites' feeds at every run. Some forums, the English Amiga Board for one, only open their feeds to robots; Amigadex reads those and nothing more.
• Sites that ask robots to stay away are never read.

Missing a site, or want one read sooner? Submit it: a submitted site gets its first look at the next run. The state of every source is on Sources.

Opting out

To keep your site out of the index, add this to your robots.txt:

User-agent: AmigadexBot
Disallow: /

Pages already indexed are removed at the next crawl. To keep a single page out, use <meta name="robots" content="noindex">.