Generally, this job involves a fair bit of evangalism – sometimes it’s quite successful (eg Promoting the use of Bugzilla as a defect tracking tool). One of my favourite diagrams is that shown in this post – I like this graph a lot. It is a graph showing the rough relationship between the cost of fixing a bug or defect, and what stage of the development process that bug or defect was found.

It’s fairly clear from the graph that, the later you realise there is a problem, the more it costs you to go back and unravel what is wrong and sort it. The reasons are fairly clear – if you find a problem at a later stage, you often have to go right back to the beginning of the process of development, testing and so on.

Some notes I like to make relating to this:

1. Even if you are already employed by a company, you are not “free”. Having someone fix a problem, and work repeated costs money – “we already pay their wages” is not an argument! Sheffield Teaching Hospitals Trust – take heed (Quote from the Reg article: The trust argued that the consequences of its decision making had not cost public money, “just time and effort by the IT teams”.).
2. Accurate. timely requirements are essential. Finding out that you have mis-specified something as it is nearing the release date is a Bad Thing™.
3. Not having requirements before coding is asking for even more trouble.
4. Changing requirements part way through the process (or, worse, finding out during testing that your requirements were duff!) is much along the lines of 2 and 3 with similar outcomes (moving goalposts anyone?).
5. Doing unit testing is much better than sending code straight to the testers – it saves a lot of heartache on both sides…
6. Actually having enough time to perform a sufficient level of testing can save you an enormous amount of hassle and cost.
7. Squishing bugs as you go at the earliest possible opportunity is much advised – multiple bugs can quickly make a system unusable and costly to fix up. (There is another similarly shaped graph – see it as The Law Of Bugterial Infection)
8. No one is perfect… not even me

Feel free to use the graph above if you want and evangelise away…

However, recently (June ish?) they have “upgraded” their website, and all seems to have gone to pot for me since. It started off last Friday (25th July) when I was trying to get some pictures from my albums like this:

- Login to Photobox
- Use the “Albums” dropdown and select the album I want
- Click the thumbnail to go to the individual image page
- Click the image to view the full size version so I can save it locally.

Easy? Not so – unfortunately, several things went wrong, though on a sporadic basis – sometimes they worked and sometimes they didn’t.

• Problem 1: The album sometimes didn’t bother loading the thumbnails – so you’re left on page 1 (of 2!) with apparently no pictures in the album. A reload often fixed this.
• Problem 2: Clicking on the thumbnail (when visible…) often results in an error page like that at the top of this post!
• Problem 3: If you manage to get the individual image page to load (after having a few goes) then it sometimes takes you back to your main screen (http://www.photobox.co.uk/my) instead of loading the image as expected.

Slightly pissed off by the fact it took me an hour to get copies of 10 images, I braved customer support. Apparently (after consulting technical), some of my images won’t be visible because they are moving servers. And, very true – in some of my galleries, there are images and thumbnails that are missing – which is fine if they subsequently reappear.

However, I’m not sure how (unless they have some oddly behaved, and not very resilient load-balanced image storage servers) you can have an image load during one attempt and then subsequently fail in another. It also fails to explain the rather spurious redirects I was getting, and the complete failures in populating the galleries on occasion.

This looks suspiciously like a fob off, or at best, support/technical who don’t understand what the issues are.

The troubles continue

Just when you thought it couldn’t get worse, I now (as of Thurs 31st July) cannot even login to Photobox. In Firefox, I get the rather odd: “Content Encoding Error“, while in IE6 I get a blank page with just the following code in the source:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

Attempting to do anything else during this session after trying a login will throw the Photobox Error Page again, and so you are stuffed. Strangely, other people can apparently login, but they are suffering from problems with whole albums of images being unavailable, and are waiting on responses from photobox support…

Hmmm. Anyone know of any other good online printing services?

Of course, increasing MaxClients doesn’t help if you start to be RAM/CPU limited, and it helps even less when the kernel starts sending out syncookies in light of the barrage of requests.

Anyhow, back to normal for the minute, but we need to move to a better front end… (at least sometimes) we are:

Who’s your mail provider?

If you are a BT broadband (or even dialup) customer, and you have an address which ends @btopenworld.com, or @btinternet.com (or one of the other domains that BT use), then you may think that you mail will be handled by BT themselves. Wrong. In June 2003, BT announced that they would be dropping the BT Openworld brand and would instead become “BT Yahoo! Broadband” from September 2003 – if you now try to deliver mail to a BT mail account, it goes via the Yahoo! MX servers (eg mx1.bt.mail.yahoo.com).

Symptoms

The major symptom of slow mail delivery is (rather obviously) that mail appears to take a long time to arrive in a users Inbox (or even spam folder!) after the moment it is sent – of course, email is a non-guaranteed medium of communication (that is, mail can fail to arrive if there is an ongoing problem – it will simply timeout, or it can easily take a few hours to arrive due to some transient issue). However, in normal operation, it would not be expected that mail should be as severely delayed as is sometimes experienced by Yahoo!’s users – sometimes from a few hours to a few days delays are seen. Often, users sending mail to a BT/Yahoo! address will receive messages from the delivering mail server stating that the mail delivery has been delayed – there is nothing a user can do and all they can do is wait.

In the worst cases, the mail can eventually just timeout – the mail server fails to complete the delivery successfully and it is returned to sender with a delivery failure. Of course, there’s no real reason for a failure to occur – the BT/Yahoo! mailservers are still online, the mail account is still active, and the address is correct.

“Deprioritization”

The reason behind these delays is what Yahoo! call “deprioritization”. By performing a manual telnet session to one of the BT/Yahoo! mailservers (mx1/mx2.bt.mail.yahoo.com on port 25) . The result you may see (and the reason why the delay occurs) is:

421 Message from (80.68.88.14) temporarily deferred - 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html

Connection closed by foreign host.

This occurs before a successful connection can even be made to the remote SMTP server – even before a “HELO” command can be issued.

Now, “greylisting” is a common and effectve way of recucing spam email – the idea being that a receiving SMTP server rejects the first connections from a remote client, but will accept a subsequent connection. Most zombie PCs that send spam will not retry, and the mail is stopped at source. A normal mail server will queue and retry delivery at a later time (according to the retry interval on the smtp server), and the mail is delivered after a short delay.

If the Yahoo! mailservers accepted subsequent connections, then there would be no problem. However, in most cases, the Yahoo! servers simply carry on rejecting connections from the same server, often for hours or days at a time – and so if you are unlucky, your mail will timeout.

Fixing the problem

Yahoo!’s own advice to prevent “deprioritization” includes such advice as:

• Remove email addresses that bounce
• Examine your retry policies
• Pay attention to the responses from our SMTP servers
• Don’t send unsolicited email
• Provide a method of unsubscribing
• Ensure your mail servers are not open relays

As a fairly responsible sysadmin, there’s no spam originating from my servers, any mailing lists have auto-unsubscribe functionality and are “closed” (there’s no spam on them), I don’t continuously try to resend emails with permanent failure codes, etc, etc. But still, there’s no sign of being removed from the “deprioritization” list.

I have even (at Yahoo!’s suggestion), setup DomainKeys and also SPF in my DNS records, but still I get the problems.

I even tried jumping through the hoops of Yahoo!’s support routines: they have a questionnaire that you can fill out and answer – here’s some guidance on how you may want to consider answering the questions. I even tried filling this out, and I apparently failed in getting my server whitelisted.

Here’s a response from Yahoo! after I tried to ask them the reason why I am being “deprioritized”:

Hello Graeme,

Thank you for writing to Yahoo! Mail.

After looking into the issue, it appears that your emails may have become deprioritized due to a number of potential issues with your mailings. To continue to receive prioritized delivery, we recommend ensuring that your email lists are well maintained. If you conform to industry standard practices, then you will not be deprioritized and you should see improvements in delivery times as well as available connections.

Please visit the following URL for information on improving list maintenance:

http://help.yahoo.com/help/us/mail/spam/spam-18.html

http://help.yahoo.com/help/us/mail/defer/defer-01.html

http://help.yahoo.com/help/us/mail/bulk/bulk-01.html

The above help page lists several techniques and technologies that will help you to effectively maintain your email lists. If you are not conforming to these standards, then you will likely experience significant delays in connections, as our system automatically prioritizes connection availability based on the type of information contained in this FAQ. By trying the basic suggestions listed on the help page, you can potentially:

* decrease your cost of mailings — in bandwidth, hardware, AND management.

* increase the effectiveness of your mailings — nonexistent users can’t open your mail or respond to your offers!

* increase the speed of your delivery — mails sent to large numbers of nonexistent users can definitely impact the time it takes for you to deliver your important messages.

Plus, you’ll ensure that your messages are prioritized by our delivery system which helps us differentiate you from spammers.

Please let us know if you still need assistance so we may assist you further.

Your patience during this process is greatly appreciated.

Thank you again for contacting Yahoo! Mail.

Regards,

Thomas Immer

Yahoo! Customer Care

Given that the server I am trying to sort out runs only two mailing lists for two sports teams, forwards mail to final recipients from some domains we host and sends mail from online forms (so no bulk commercial mailing); and given that there has not been a single item of spam originating from this server (sure it may receive spam on addresses it is the MX for and pass it over to the final destination address – but there’s little I can do about that – I already greylist inbound mail!), I’m at a loss as to what more I can do to try to adhere to their guidance.

In what is a brilliant response, they apparently can’t tell me what the specific “potential issue” is (leaving me to guess the solution or the answers they want to hear):

Hello Graeme,

Thank you for writing to Yahoo! Mail.

Unfortunately, we cannot provide you with specific information other than to suggest a review of the questionnaire we supplied and try to determine where your mailing practices may be improved upon.

In addition, you may refer to the Help page below, which offers information on some of our recommended best practices for bulk mailers:

http://help.yahoo.com/help/us/mail/spam/spam-17.html

Please let us know if you still need assistance so we may assist you further.

<snip>

In other words, we’ll do what we want – and we don’t care if you have problems with this. (An odd way of doing business – it’s their customers who are inconvenienced by this – if their mail never arrives they are the ones to complain).

A solution?

So, after installing DomainKeys and SPF records, I was close to giving up, and then discovered that the Yahoo! servers appear to be slightly random in how they reject connections – if you try and try and try, eventually, one will get through… so, by making the following changes to Sendmail, you can get it to try at a must higher rate than normal:

define(`confTO_HOSTSTATUS',`1m')dnl

define(`confQUEUE_INTERVAL',`2m')dnl

I don’t like doing this, but in practice, it doesn’t seem to be too bad at the minute – the increasednumber of attempts to deliver seems to eventually “get lucky” and the delivery will be accepted without issue.

Maybe at some stage I’ll try and jump through their hoops again – but at the minute, my best advice if you have these issues is to simply move mail provider. At Distilled we use Tuffmail – their support is excellent, their spam filtering is very good and user-configurable, even with the option of Bayesian filtering, and you’ll never have a problem with massively delayed mail again.