UPDATE: Since writing this article, BT have moved off the Yahoo platform (see: www.bt.com/newemail) – I’m unsure if the same issue is apparent for BT users, but the same issue exists for Yahoo as far as I know.

If you are a BT or Yahoo email user, then you may notice occasionally that your email takes an age to arrive. Sometimes you may even discover that an email someone sent you you has never arrived. If you experience this, here’s a bit of an explanation as to why this may be happening…

Who’s your mail provider?

If you are a BT broadband (or even dialup) customer, and you have an address which ends @btopenworld.com, or @btinternet.com (or one of the other domains that BT use), then you may think that you mail will be handled by BT themselves. Wrong. In June 2003, BT announced that they would be dropping the BT Openworld brand and would instead become “BT Yahoo! Broadband” from September 2003 – if you now try to deliver mail to a BT mail account, it goes via the Yahoo! MX servers (eg mx1.bt.mail.yahoo.com).

Symptoms

The major symptom of slow mail delivery is (rather obviously) that mail appears to take a long time to arrive in a users Inbox (or even spam folder!) after the moment it is sent – of course, email is a non-guaranteed medium of communication (that is, mail can fail to arrive if there is an ongoing problem – it will simply timeout, or it can easily take a few hours to arrive due to some transient issue). However, in normal operation, it would not be expected that mail should be as severely delayed as is sometimes experienced by Yahoo!’s users – sometimes from a few hours to a few days delays are seen. Often, users sending mail to a BT/Yahoo! address will receive messages from the delivering mail server stating that the mail delivery has been delayed – there is nothing a user can do and all they can do is wait.

In the worst cases, the mail can eventually just timeout – the mail server fails to complete the delivery successfully and it is returned to sender with a delivery failure. Of course, there’s no real reason for a failure to occur – the BT/Yahoo! mailservers are still online, the mail account is still active, and the address is correct.

“Deprioritization”

The reason behind these delays is what Yahoo! call “deprioritization”. By performing a manual telnet session to one of the BT/Yahoo! mailservers (mx1/mx2.bt.mail.yahoo.com on port 25) . The result you may see (and the reason why the delay occurs) is:

421 Message from (80.68.88.14) temporarily deferred - 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html

Connection closed by foreign host.

This occurs before a successful connection can even be made to the remote SMTP server – even before a “HELO” command can be issued.

Now, “greylisting” is a common and effectve way of recucing spam email – the idea being that a receiving SMTP server rejects the first connections from a remote client, but will accept a subsequent connection. Most zombie PCs that send spam will not retry, and the mail is stopped at source. A normal mail server will queue and retry delivery at a later time (according to the retry interval on the smtp server), and the mail is delivered after a short delay.

If the Yahoo! mailservers accepted subsequent connections, then there would be no problem. However, in most cases, the Yahoo! servers simply carry on rejecting connections from the same server, often for hours or days at a time – and so if you are unlucky, your mail will timeout.

Fixing the problem

Yahoo!’s own advice to prevent “deprioritization” includes such advice as:

• Remove email addresses that bounce
• Examine your retry policies
• Pay attention to the responses from our SMTP servers
• Don’t send unsolicited email
• Provide a method of unsubscribing
• Ensure your mail servers are not open relays

As a fairly responsible sysadmin, there’s no spam originating from my servers, any mailing lists have auto-unsubscribe functionality and are “closed” (there’s no spam on them), I don’t continuously try to resend emails with permanent failure codes, etc, etc. But still, there’s no sign of being removed from the “deprioritization” list.

I have even (at Yahoo!’s suggestion), setup DomainKeys and also SPF in my DNS records, but still I get the problems.

I even tried jumping through the hoops of Yahoo!’s support routines: they have a questionnaire that you can fill out and answer – here’s some guidance on how you may want to consider answering the questions. I even tried filling this out, and I apparently failed in getting my server whitelisted.

Here’s a response from Yahoo! after I tried to ask them the reason why I am being “deprioritized”:

Hello Graeme,

 

Thank you for writing to Yahoo! Mail.

 

After looking into the issue, it appears that your emails may have become deprioritized due to a number of potential issues with your mailings. To continue to receive prioritized delivery, we recommend ensuring that your email lists are well maintained. If you conform to industry standard practices, then you will not be deprioritized and you should see improvements in delivery times as well as available connections.

 

Please visit the following URL for information on improving list maintenance:

 

http://help.yahoo.com/help/us/mail/spam/spam-18.html

 

http://help.yahoo.com/help/us/mail/defer/defer-01.html

 

http://help.yahoo.com/help/us/mail/bulk/bulk-01.html

 

The above help page lists several techniques and technologies that will help you to effectively maintain your email lists. If you are not conforming to these standards, then you will likely experience significant delays in connections, as our system automatically prioritizes connection availability based on the type of information contained in this FAQ. By trying the basic suggestions listed on the help page, you can potentially:

 

* decrease your cost of mailings — in bandwidth, hardware, AND management.

 

* increase the effectiveness of your mailings — nonexistent users can’t open your mail or respond to your offers!

 

* increase the speed of your delivery — mails sent to large numbers of nonexistent users can definitely impact the time it takes for you to deliver your important messages.

 

 

Plus, you’ll ensure that your messages are prioritized by our delivery system which helps us differentiate you from spammers.

Please let us know if you still need assistance so we may assist you further.

 

Your patience during this process is greatly appreciated.

 

Thank you again for contacting Yahoo! Mail.

 

Regards,

 

Thomas Immer

 

Yahoo! Customer Care

Given that the server I am trying to sort out runs only two mailing lists for two sports teams, forwards mail to final recipients from some domains we host and sends mail from online forms (so no bulk commercial mailing); and given that there has not been a single item of spam originating from this server (sure it may receive spam on addresses it is the MX for and pass it over to the final destination address – but there’s little I can do about that – I already greylist inbound mail!), I’m at a loss as to what more I can do to try to adhere to their guidance.

In what is a brilliant response, they apparently can’t tell me what the specific “potential issue” is (leaving me to guess the solution or the answers they want to hear):

Hello Graeme,

Thank you for writing to Yahoo! Mail.

Unfortunately, we cannot provide you with specific information other than to suggest a review of the questionnaire we supplied and try to determine where your mailing practices may be improved upon.

In addition, you may refer to the Help page below, which offers information on some of our recommended best practices for bulk mailers:

http://help.yahoo.com/help/us/mail/spam/spam-17.html

Please let us know if you still need assistance so we may assist you further.

<snip>

In other words, we’ll do what we want – and we don’t care if you have problems with this. (An odd way of doing business – it’s their customers who are inconvenienced by this – if their mail never arrives they are the ones to complain).

A solution?

So, after installing DomainKeys and SPF records, I was close to giving up, and then discovered that the Yahoo! servers appear to be slightly random in how they reject connections – if you try and try and try, eventually, one will get through… so, by making the following changes to Sendmail, you can get it to try at a must higher rate than normal:

define(`confTO_HOSTSTATUS',`1m')dnl

define(`confQUEUE_INTERVAL',`2m')dnl

I don’t like doing this, but in practice, it doesn’t seem to be too bad at the minute – the increasednumber of attempts to deliver seems to eventually “get lucky” and the delivery will be accepted without issue.

Maybe at some stage I’ll try and jump through their hoops again – but at the minute, my best advice if you have these issues is to simply move mail provider. At Distilled we use Tuffmail – their support is excellent, their spam filtering is very good and user-configurable, even with the option of Bayesian filtering, and you’ll never have a problem with massively delayed mail again.

BT/Yahoo? Slow Email? by Graeme Coates is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.