Design Tips
Hiding email address details
Having email links
<a href="mailto:me@domain.com">email</a> embedded into the page code is an extremely useful way of letting customers contact you simply by clicking on a link. This should open the default email client on the users machine and let them type in a message directly. However this ease of use also has a downside. By having the link plainly visible in the source code makes it easy for the harvesters and spambots to collect the addresses.
Spambots and harvesters
While both of these serve the same purpose, That of collecting open email addresses for later use or sale as addresses to be used for Unsolicited Commercial Email (UCE) recipients. More commonly known as "spam". Spambots tend to behave more like a normal crawler and follow links from website to website, while harvesters will be more commonly used on guestbooks, newsgroups and now blogs where many addresses can be "scraped" in short period.
While both of these serve the same purpose, That of collecting open email addresses for later use or sale as addresses to be used for Unsolicited Commercial Email (UCE) recipients. More commonly known as "spam". Spambots tend to behave more like a normal crawler and follow links from website to website, while harvesters will be more commonly used on guestbooks, newsgroups and now blogs where many addresses can be "scraped" in short period.
Several methods exist for hiding email addresses from crawlers and bots some more effective than others. One excellent method is not to have email addresses anywhere on a page and use a contact form for visitors to send a message or request information. This method uses the fact robots and crawlers do not submit forms so the mail address is never shown. The drawback is that some coding may be needed to configure a server-side script to send the email, this could be beyond the capabilities of some webmasters or may not be allowed by the hosting company.
Another method is to write the username and domain name onto the page using html entities (character codes), unicode entities or ASCII codes, this approach has been successful for quite some time but is approaching the point of being ineffective as more bots become capable of reading and converting character codes. Mainstream crawlers have no problem reading character codes so the spambots that scrape the search engine results for emails will be getting these adresses anyway.
Another method is to write the username and domain name onto the page using html entities (character codes), unicode entities or ASCII codes, this approach has been successful for quite some time but is approaching the point of being ineffective as more bots become capable of reading and converting character codes. Mainstream crawlers have no problem reading character codes so the spambots that scrape the search engine results for emails will be getting these adresses anyway.
Using Client Side Javascript Code
Using coded methods for hiding email addresses are very effective, the simplest and most common is to use javascript to write the text of the email address to the page. This relies on the principle that currently no robots or crawlers used for harvesting are javascript enabled and to add the capabilities to them is not a trivial programming task. However many other user agents used to browse the internet are also incapable of triggering javascripts. These are such things as accessibility devices, screen readers and braille devices, so you may well be stopping a large number of potential customers from contacting you. The way around this is to make use of the <noscript> </noscript> tags and a contact form, then put a link to the contact form in the <noscript> block.
Using Server Side Code
To use server side ASP code as a means of not showing email addresses to crawlers could be done in several ways.
One method would be have a list of the known user agent signatures used by the various bots in a database and use a lookup table for the script to base a decision on. This can be a time consuming business as new user agents are put online on a very regular basis and the database would need to be kept up to date maybe for several sites. Add to this that it is a simple matter for the spambots to change the UA signature to show that of a browser and bypass this.
So what can be used is a feature of browsers but is not normally present in crawler type user agents, and that is cookies or sessions.
They are also used to store your preferences for content or personalized pages. Forums and shopping carts use cookies. These allow you to automatically logon on when returning or choose items and leave the virtual store, then return later and find that all the items are still in your shopping cart.
Cookies are also used to build a profile of which sites you visit and which banner ads you click on. Advertisers use this information to deliver targeted ads directly to your computer. Some sites save your preferences on the cookie itself.
Other sites assign users ID numbers or encoded passwords and keep records of your preferences at their end.
Persistent cookies can stay on your hard drive for long periods they can also have expiry dates coded into the file to allow the browser to overwrite them when the space is needed.
One method would be have a list of the known user agent signatures used by the various bots in a database and use a lookup table for the script to base a decision on. This can be a time consuming business as new user agents are put online on a very regular basis and the database would need to be kept up to date maybe for several sites. Add to this that it is a simple matter for the spambots to change the UA signature to show that of a browser and bypass this.
So what can be used is a feature of browsers but is not normally present in crawler type user agents, and that is cookies or sessions.
Persistent Cookies
Persistent Cookies are small text files that websites can place in a cache on your computer to let browsers remember specific information. For example, they might store your passwords and user IDs.They are also used to store your preferences for content or personalized pages. Forums and shopping carts use cookies. These allow you to automatically logon on when returning or choose items and leave the virtual store, then return later and find that all the items are still in your shopping cart.
Cookies are also used to build a profile of which sites you visit and which banner ads you click on. Advertisers use this information to deliver targeted ads directly to your computer. Some sites save your preferences on the cookie itself.
Other sites assign users ID numbers or encoded passwords and keep records of your preferences at their end.
Persistent cookies can stay on your hard drive for long periods they can also have expiry dates coded into the file to allow the browser to overwrite them when the space is needed.

