Reference

EarlyWave bots

View as Markdown

EarlyWave sends a few automated requests on its own behalf. Each one names itself in its User-Agent and links to this page.

EarlyWaveCatalogBot

EarlyWaveCatalogBot/1.0 (+https://www.earlywave.ai/developers/docs/bots; newsletter catalogue sign-up)

What it does. EarlyWave keeps a catalogue of free newsletters that readers can follow. When a newsletter has no RSS feed, EarlyWave subscribes one address of its own (catalog-…@earlywave.email) through the newsletter's public sign-up form, then clicks the confirmation link of the double opt-in once. The issues it receives are shown, with a link back to the publisher, to EarlyWave readers who follow that newsletter. Paid newsletters are never subscribed.

How it behaves.

  • It reads robots.txt first and obeys it for every request: the sign-up page, the form submission and the confirmation link alike. The product token to name is EarlyWaveCatalogBot. A robots.txt that cannot be read (a challenge, 401, 403, 429 or a server error) counts as "disallow everything"; only a missing file (404, 410) counts as "no rules". Where a platform's robots.txt disallows its sign-up or confirmation path, as Mailchimp and Buttondown do today, the newsletter is subscribed by hand or not at all.
  • It sends at most one request per second to a host.
  • It never solves or works around a CAPTCHA or any bot protection. When it sees one, it stops, and a person subscribes by hand or not at all.
  • It submits a form at most twice in the life of an address (once, plus one manual retry), with nothing filled in but the address. A form that needs a name, a consent box or a choice is left to a person.
  • From a confirmation e-mail, it follows only the confirmation link, and only when that e-mail passes DMARC from the newsletter's own domain or from its platform's known sending domain (Mailchimp, Kit, Buttondown, Ghost, Substack, beehiiv). It never follows an unsubscribe link or any other link, and never guesses between two. A confirmation page that asks for one more click is left to a person.

Stopping it. Disallow EarlyWaveCatalogBot in your robots.txt:

User-agent: EarlyWaveCatalogBot
Disallow: /

To have a catalogue address unsubscribed, or for anything else, write to contact@earlywave.ai.