Protection from junk e-mail

Protection from junk e-mail - for everyone - and for free!

It's not a perfect plan. I estimate it will block only about ... 99.5% of the junk e-mail "spam" that most people get. And for the moment it will require folks to create a new address to get this much protection. But it can be done *today* and it can be done for *free* so I think it will become clear to most ISPs (Internet Service Providers) that if it's what people want, and the techniques required are already understood, then they should be providing it to everyone.

But, I get ahead of myself. Later in this article I will give a detailed example of how anyone can get this level of spam protection, but first, some background information to help everyone understand how and why it actually works.

Have you looked closely at the spam you get? I think most people haven't bothered, because that would take up even more of their time, and I really don't blame anyone for that. Wasted time is probably the biggest cost that spam puts upon us all. But if you did peek inside, you would probably see that a great deal of spam fails to have any workable return address. And quite a bit of the spam which does have a working return address ... well, it's often forged, in the name of someone innocent. Some other victim of the spammer would receive your reply, if you went to the trouble to try to tell the spammer what you think of them.

Now, let's compare that to the legitimate e-mail you receive. Does any of that, any at all, fail to have a working return address? For almost everyone, the answer is NO. Never. Not only that, but almost all of your legitimate e-mail is from people with whom you've previously exchanged e-mail. Of course we do add new people to our address book now and then (if we actually use an address book) but that operation is usually infrequent.

Essentially all of our legitimate e-mail is from senders we already knew. Essentially all of our junk e-mail is from senders we have never heard of before. This is the beauty of whitelisting. We can build a list (or an address book) containing the addresses of everyone from whom we do want to receive e-mail. It's called whitelisting because it's the opposite of blacklisting, where you make a list of everyone from whom you don't want e-mail. Blacklisting often fails because there are literally millions and millions of addresses where e-mail might come from, and more are created every hour of every day. To try to keep a complete blacklist would require more computing power than almost anyone has.

"Pretty Good" Protection

Whitelisting alone is hardly perfect protection. In order to get their junk to us, spammers could forge the addresses of our friends. This already does happen in small numbers today, but if that was the only kind of spam which got through, the amount of our time it could waste would probably be much more manageable than it presently is. There are measures already designed which could defeat that problem, but those must be saved for another article.

We could just stop with that - use a whitelist, and anyone else simply can't get in. That's pretty good spam protection, and many e-mail service providers make it possible already - but what about those occasional new correspondents, the ones you really would want to hear from, if only they could get through? Well, there is still a way. Whitelisting alone is like locking your door, and giving keys only to your friends. That works pretty well for front doors, but what about when someone rings the doorbell? If just anyone can ring, and you answer, it takes up your time, and you're right back to square one, with spammers wasting your time whenever they feel like it. But the Internet isn't like a physical neighborhood - in a sense, there aren't any distances - everyone is right outside your front door. They "ring" the bell on your door by sending e-mail to your address. This article will show you how to create a much smarter doorbell. Anyone can push the button, all they need is your button address, but this doorbell is smart - it uses the address they gave, and it rings them right back! If they can answer, then you know they've given a real address. Then, and only then, your smart doorbell will let them ring where you can actually hear it. And until then, it just doesn't let them bother you.

And with that, you have a mechanism which will stop the vast majority of the kinds of spam seen today. If enough people employ measures like it, the spammers will have to either work harder to defeat it, or fade away. But if a spammer does go to the trouble to join your whitelist, there's still a "ban" button, just a click away, and you'll never hear from that one again. That kind of blacklisting probably won't ever grow out of control, because the spammers would have to invest even more effort than you do. Not only that, but any victims of such spam who choose to could share the spammers' sender addresses, saving others the trouble of ever letting the spammers join. A whole network of people exchanging evidence on spammers might emerge, one humorous vision of which has details being exchanged like collectable "whack-a-spammer" game cards.

Interestingly, the software necessary to implement such a system is not only already in existence, it's already been deployed and is in use by thousands of people every day. They are just using it to meet a different goal. The goal it was designed for is mailing lists. Lists where generally e-mail is sent to the members, not just received from them. There are quite a number of established e-mail service providers who issue list management accounts. Several even do so at no charge, one of the most obvious among them being the Groups service at Yahoo! And that free service is the first one I chose for my "proof of concept" testing.

Free Demonstration

I will demonstrate how anyone, using Yahoo!Groups, could set up a free account with a free address, and have that address forward e-mail wherever they wish, but only the e-mail which passes the closed-loop confirmation test. As we've seen, most typical spam would never get through. You could feel fairly safe to share such an address with anyone, possibly even publish it online, or anywhere, with a degree of safety from junk e-mail which has previously mostly been known only to programmers, e-mail administrators, and other "techie" type folks. Since we are using online tools to do something which they were not quite specifically designed for, there will be compromises to be made, but bear in mind that this is only to prove that the software necessary to do an even better job is only a few changes away from what's already in use.

Similar procedures at any of the many other providers of Mailing List Manager (MLM) services could probably produce similar results. Much of the software MLM services run on is open-source, meaning that any service provider using it has the freedom to make modifications of just the sort that I will later describe.

Mailing List Manager (MLM) systems are generally used for different purposes than intercepting spam, of course. One typical use is to set up a list of members, each of whom is authorized at any time to send any message to the list address, from which copies then get forwarded out to every member. As one would expect from their name, MLMs are about mailing - sending out messages. And about managing lists of the recipients of those sent-out message. But, in order to prevent certain types of misuse, such as spam getting sent to every member, most lists are set up to restrict submissions to members only. These settings are often under the control of a list's owner, who usually also is the person who created the list, and who makes any other administrative decisions. Another of the typical configurable functions of a list is moderation of submissions. When moderation is enabled in a list, members of that list may freely submit messages, but those messages won't be forwarded out to the general membership until after a person designated as "moderator" approves the message. Often, this moderator is the same person as the owner, but in general that's not a requirement of the MLM. Some lists are set up to allow anyone at all to submit message, which would then be copied out only to the members, but usually such "open submission" lists are also moderated - again, to prevent spamming, among other possible problems. Another function of the MLM software in the management of the lists themselves is the confirmation functions. The initial purpose of these confirmations was to insure that someone who is about to start receiving (potentially lots of) e-mail from the list is actually someone who wants that e-mail. This is to prevent errors, such as when someone with an address similar to yours makes a typographical error, or when some well-meaning friend of yours signs you up for something they think you'll like. But at the same time, these confirmations prevent many other forms of abuse, such as when a spammer intentionally signs up huge numbers of addresses and spams them all, then attempting to blame the MLM.

But abuses, unwanted "favors", and typographical errors are just about the same things which cause us all to receive spam in the first place. Since there are already tools designed and deployed to prevent exactly those kinds of problems, I took on the challenge of finding a way in which practically anyone could use them to reduce incoming spam. And it turned out to be a pretty simple re-configuration of a typical list, as managed by a typical MLM. One restricts posting to members only, configures the list to require all posts be approved by a moderator, and then simply never approves any of those posts. The key is the moderator address. The only messages which the MLM will forward to the moderator address of such a list are submissions from established members. What we are doing is simply not using many of the major features of an MLM, and making judicious use of a certain few of the others.

Tutorial

So let's go to Yahoo!Groups and see an example of just how much "tweaking" is necessary. Very little, really.
You can follow along and create your own if you like, but it might be a better use of your time to read through the procedure once, to pick up the various drawbacks I'll mention, all of which could be removed if the service providers chose to make a few modifications and create a service distinct from their MLMs.

One thing we're going to eventually need, to make this work, is an e-mail account for the "owner" of the list. We won't be able to completely protect this account from spam, but we will be able to insure that nothing else we care very much about goes there.
For convenience, I simply created a Yahoo!mail account to do this.
Then, logged-in to that Yahoo!mail account, I clicked a link which said "Groups" and then a link "Start a Group"
At this point Yahoo asked me to choose a category for my new Group.
I chose (Computers and Internet - Internet - Spam) but you may wish to choose something else entirely. However, if you do choose the same category, it might help the folks at Yahoo to measure the demand for this sort of service to be added to their regular lineup.
Then I clicked the link which said "Place my group in Spam" Not a particularly appetizing prospect, but there you are.
Next, I chose a name for my group, and the address. For both of these, I picked "filterbob"
The same page asks for a paragraph to describe the Group, and I just put in a few temporary words, knowing I could polish it up later (I really still haven't, though).
Then I clicked the "Continue" button and came to the first place where Yahoo!Groups tested me to see if I was really a human being.
So I read the word from the image, typed it into the little box, and pressed the button.
And now, it's a group! But that's not quite what I'm here for.
Time to reconfigure it as a spam protection service.
I clicked the "Customize Group" button. Yahoo!Groups uses a "wizard" interface, taking the user through several screens of options.
The first page simply told me this. I pressed "Get Started" and found myself on the "Step 1 of 3" page.
On page one, you can choose whether or not you want your group listed in the Yahoo! directory, but leave the other two options on their default settings. ('anyone can join immediately', and 'only group members can post')
The critical setting to change for turning a group into a spam protection service is on page two, where you should choose "Yes, messages require my approval" - this step makes the list (group) moderated, a required step for our purposes. It is the combination of requiring approval, but then never intending to actually approve messages for re-distribution, which turns a list into a spam-blocker.
The other two options on page two are up to you for your own group, but the one about message replies really won't matter either way, because we don't intend to 'approve' any messages so no one's going to be replying to them. Personally, I turned off the "web" features because I don't know what I'd do with them. I can always change that later.
On page 3 of 3, all the choices there are up to you. If you do turn on the web features, I recommend leaving them all at their "members only" default settings for now. Of course you may already be a veteran owner of Yahoo!groups and way ahead of me at this stage of the game.
The two important configuration details are that only members can post to the list, and that all messages require moderator approval.
And now, the Yahoo!Group is created and configured the way we need.
But we're not done yet. As it stands the owner account (for me, that's the Yahoo!mail account I created first) is also the moderator account. That's not going to work in the long run, because anyone who researches Yahoo!Groups can learn that sending e-mail to the name of the group, followed by a dash and the word owner (in my case <filterbob-owner@yahoogroups.com>) will reach the owner account directly. Well, we don't really want that e-mail.
What we have to do is add a new member to the list, using an e-mail address which we intend to "protect" and then we need to give that list member the "moderator" duties.
Since I have my own domain for e-mail, I simply created a new account there. You may want to use your regular e-mail account there, if it doesn't already get much spam, or you may want to create a new account somewhere else, or even ask your present ISP for another. One thing I have recommended in the past is to choose a random string of letters and digits for your username, something which doesn't even resemble a name, and make it more than ten characters long. Spammers often just guess at anything which looks even remotely like a name, and randomly insert numbers in the middle or at the end - it's shocking how often they can guess or decypher simple things like "fred006a" or "john03doe50"
From my new and soon-to-be-protected account, I sent a message to my new group <filterbob@yahoogroups.com> and in a short time, the Yahoo!Groups system sent back an automatic reply, with instructions for how to sign up. And this automatic reply is exactly what would be sent back in response to a spam. By the way, I'm sorry if a spammer some day forges your address in the From field of a spam sent to <filterbob> or another Yahoo!Groups address, but in that case it's the spammer's fault, please don't blame me or Yahoo for the automatic reply - I hope you can just ignore it and go on with your life. You could blacklist the automatic sender address at Yahoo in that case, but remember to turn off that rule in your blacklist for a few minutes if you ever do want to join a Yahoo!Group.
Anyway, since I'm planning to be the moderator of my group with this address, I do want to follow the sign-up instructions.
I did my sign up entirely by e-mail, rather than clicking any of the links in the e-mail, but that's only because I didn't want my browser window to change - it's still showing the page where I control my new group.
Once my e-mail address received the automatic message from Yahoo!Groups saying "Welcome to filterbob" I went back to the browser and clicked the "Members" link (on the left, in the box) on my group management page. Once there, I now see two list members - the "owner" account, which in my case is at Yahoo.com, and the soon-to-be moderator account, the real address I want to protect.
To the left on each member line there is an "Edit" button, and I clicked the one for the account which I want to make the moderator.
Now, under "Membership Privileges" I click the link "Change to Moderator"
On the "Make Moderator" page, I checked all the boxes. It's my account, I don't intend to give it to anyone else, so I might as well give myself all the privileges of Group ownership. Except the one where the account ends up with unwanted <filterbob-owner> e-mail. I don't mind if Yahoo!Groups sends me a notice any time someone joins; for the time being I find that information interesting. Maybe tomorrow after this article is published there will be more activity there and I'll just come back here and turn that part off.<g>
So now it's time to click the "Make Moderator" button, and then the "Save Changes" button.
On that member's line now, we see a symbol (a yellow star) indicating moderator privileges have been granted. We're almost done!
I've already mentioned that the "owner" account may be receiving unwanted email sent to the "-owner" form of the list's address. But I haven't mentioned that, by default, the owner of a group is automatically also a moderator. We now want to edit that away. In my case, the owner account is a Yahoo!mail account which I may not be checking very often, and, just to be polite to Yahoo!, I want to limit the email which goes there as much as I practically can. It will make less of a task for me anyway, when I log in once a week or so to delete it all.
So I click the "Edit" button next to the owner account.
I then UN-check the checkboxes under "Email Notification" (only one was checked) and click the "Save Changes" button.
And now, we really are done!

Caution: if you start sending yourself messages right away to test your new group, do not be surprised if nothing seems to work at first. I have found that it can take a couple of days for the first messages to work their ways through the Yahoo!Groups system. Yes, I've actually seen a "test" message take a little more than fourty-eight hours to get back to me. Have a little faith. I think what's probably happening is that the many, many servers at Yahoo! have to spread the word about your new Group. Once a couple of days have gone by, all the relevant Yahoo! servers know about the Group, and I find that new messages are forwarded to my moderator account in mere seconds. Which is really pretty darned good service for the price!

If Yahoo!Groups might not be your choice, but you'd like to achieve the same kind of function with another MLM, the key features to look for are:

closed-loop confirmation built in
submissions can be restricted to only confirmed members
all submissions can be emailed to a moderator address for approval
non-members who write to the list address can automatically receive a reply containing instructions for how to subscribe

Any or all of these features might be selectable in an MLM, but you'd need for them all to be present, at least as options.

Drawbacks

Earlier I mentioned drawbacks, mostly due to Yahoo!Groups (and other MLMs) having been designed for slightly different purposes. In addition to the delays just mentioned, there are some drawbacks which won't go away.

First and, I think, foremost, you won't be able to use a list address for everything. Yahoo!Groups addresses, for example, will never be allowed to 'join' other Yahoo!Groups. I'm sure this is a wise security measure on Yahoo's part, as two lists subscribed to each other might go into a loop, using up server resources until something breaks. Many other MLMs are probably also smart enough to recognize list addresses, and subscribe messages, and be unwilling to forward those on to the list owner. It's probably best not to even try.
Another drawback to using an MLM in our "new" way is the requirement to send "invite" messages. It may be slightly annoying to your friends to have to go through the confirmation process, evem though you may already have exchanged lots of email previously. These invitations make sense to prevent list abuse for the reasons already outlined, but if the MLM software were modified for our specific use, it should be practical to relax the confirmation requirement for whoever the list owner may wish to add. Since the modified list won't be sending out email, the need to confirm is mostly absent when the owner approves directly.
You won't want to use a modified-MLM address to sign up on an ordinary mailing list. Whenever new members joined that list, your modified-MLM might send them its autoresponse, and that would get annoying to other list members quickly
Using a modified-MLM, you can only whitelist members by their sender address - no other characteristics can be used, even though you might know exactly what to look for. For example, some people use what are called 'vanity' domains, where any address at that domain is probably the same person, so you might prefer to simply allow any address in that domain to mail you.

Also, some not-quite-as-serious drawbacks:

You might accidentally "approve" a message, sending it to everyone
You may still have to filter your moderator address to restrict it to receiving moderator message from the list and discard all else
You have to discard mail sent to the owner address and possibly first sift through it for important messages from the host (in my case, Yahoo) about managing your list.

Conclusion

If you, someone you know, or even just someone you've heard of, has already done this, using Yahoo!Groups or any other MLM, then my heartfelt congratulations to them! I do not claim to be the first or only person to have thought of this technique. I just feel it's vitally important that service providers have pointed out how easily they all could provide similar and even more powerful services for all their users: install an MLM system, remove darned near half of it, then customize what's left a little. To modify an MLM into a spam blocking tool as I'd recommend, the first step would be removal of all the mail-out facilities. We won't need a list tool to send email to our friends and other contacts. Once those features are gone, it's safe to remove some of the security aspects added to prevent using an MLM for sending spam, like the restrictions on owners simply adding addresses. We won't need a list tool to tell our friends that we like them as friends. A modified tool used for spam blocking won't have any need for many of the messages MLMs normally send to owners, so we can dispense with those. And then there's far less reason to separate the owner from the moderator, futher simplifying our hypothetical new tool. Make these changes, and perhaps a few more, run some tests, debug a little, and it's ready to go, with very low development costs! After a while, some even more advanced features could be added, such as the ability to "handshake" with other whitelisting tools, eliminating the need for the user to manually confirm when they really did request to be added to one another's lists. But again, let's leave detailed discussion of more advanced features for another time.

It should be stated that I don't actually know that the engineers at Yahoo! who developed the Groups services never thought of using it this way. It could be that they did, and left it for someone to find, like an "Easter Egg" - a term commonly used in the software industry for a hidden feature with interesting or fun capabilities.

What I do know is that I'd like for Yahoo, MSN, AOL, and every other service provider to see from this example just how easily existing solutions can be adapted to provide this valuable service to everyone. The motivation behind the preparation and dissemination of this article is the hope that providers may actually enter into a bit of a feature race, providing more and better e-mail protection for everyone, leading to eventually putting spammers out of business forever.

Feel free to send me your comments, feedback, even questions, at the e-mail address below ... note that it's at yahoogroups! And especially, please write if you represent MSN, AOL, Yahoo, Hotmail, Earthlink, or any provider and you'd like my advice on how you can provide these expanded services to your members - I am available for consulting.

Bob O`Brien
<filterbob@yahoogroups.com>