Email Newsletter Science

Managing email newsletters is not as simple as one might think ...

The most simple approach is doing it manually: maintain the list of recipients manually, manually add or remove subscribers upon request and send out the email with a normal email client using the BCC field. There are several problems with this approach:

• Limited to small number of recipients: the number of emails accepted in the TO or BCC field is limited of course.
• Newsletter might get rated as spam and might help virus spread: if you use the BCC field, your email might be killed by spam filters. You shouldn't put all recipients into the TO field as well, because then every recipient sees the email addresses of all the other recipients and could send a newsletter himself via "reply all". And today's viruses are actually looking for emails with many recipients in the email archive of infected PCs and spreading viruses between the list of recipients.
• Implementing a double-opt-in policy is difficult: The motivation for double-opt-in is typically legal reasons. You want to make sure that the person who adds an email address to the list is really the owner of this email. This requires sending the subscriber a "secret link" which is unique for each subscriber. Only after the subscriber has sent this secret and unique information back, he is activated as newsletter subscriber.
• handling bounced email: the larger the mailing list becomes, the more important is automated bounced email handling. Recently I sent out a 300-recipient newsletter. I got about 30 bounce messages back: 12x "unknown user", 5x "mailbox quota exceeded", 3x "unknown domain", 3x "invalid email address", 3x "email delivery delayed, 1x "invalid data in message", etc ... Now imagine this with a 3000 recipient newsletter or even larger ones.

Most problems mentioned above can be solved quite easily by using database-driven mailmerge and mailing list management functionality, either on the webserver or as a special email marketing software installed on your PC. However, not all of those products provide double-opt-in, and even less of them provide a good solution to the bounced mail problem.

I have been using Yahoo Groups for managing the 300-recipient mailing list for a long time. Yahoo Groups handle automated list management (subscribe/unsubscribe), however did not provide double-opt-in. Yahoo Groups also completely automated bounced email handling. The mailmerge engine of Yahoo Groups puts a special "return-path" information into each mail, so all bounced email is not sent to the newsletter sender, but to the Yahoo Groups system where it is processed automatically: "Return-Path: [unique-mailing-code]@returns.groups.yahoo.com". The Yahoo Groups system automatically detects the bounce reason and reacts accordingly, i.e. either retries several times to deliver the email, remove the recipient from the list, etc. When the newsletter recipient is using the "Reply" button in his email client, this reply email will still go directly to the sender of the newsletter. Very nice!

However Yahoo Groups had a huge security problem: the list owner can post a message to the list simply by sending an email to the list's email address ([mailing-list-name]@yahoogroups.com). The Yahoo Groups system would only check if the "From" address equals the list owners' email address. Someone of my 300 recipients got infected with one of those viruses that replicates itself by sending emails with forged "From" addresses, and so his virus was distributed to the 300 recipients of my mailing list. Not very funny. This was the last day I ever used Yahoo Groups.

So, I switched over to the mailmerge function of the CMS I am using for the website (Peachtree WebsiteCreator). Unfortunately this one neither has double-opt-in, nor does it have automated bounce handling... (at least not yet).

For another website I am currently working on I use the open source CMS "Mambo", and there is also a newsletter and mailing list management component available ("YANC") which does support double-opt-in. YANC has no automated bounce-processing (it does not use the Return-Path header field), however, it does something I would call "bounce prevention". Its SMTP engine (PHPMailer) first checks if the recipient's email server exists. If not, the email is not sent, a non-delivery-report is not created. If the recipients email server is available but refuses to accept the recipient email address, YANC will immediately abort this email transfer. Again no bounce / non-delivery email is created. Finally, if the email gateway of the recipient does accept the email, and then forwards it to the recipients email server and then this email server detects that the recipient address is unknown, generation of a bounce email is still prevented because YANC has added the "Precedence: bulk" field to the envelope of the email. This does also suppress "Out-Of-Office" replies and "mailbox quota exceeded" bounce messages.

So in summary, there are two approaches:

1. Adding "Precedence: bulk" to the email envelope to suppress any bounce email
2. Adding "Return-Path: ..." to the email envelope to direct any bounce email to a special mailbox where all bounced email is processed by an intelligent algorithm that will automatically purge dead email addresses from the mailing list

I think the first option is the perfect solution for mailing lists up to several 1000 recipients - several 100 dead email addresses just don't matter in this case. If you get beyond 10000 recipients it starts making sense to purge dead email addresses from the list because of processing time / bandwidth cost.