O2: How to turn a cockup into a clusterfuck

Every IT company makes cock-ups. It’s pretty much unavoidable: you’re dealing with a lot of data, and there’s hundreds of ways things can go wrong. The difference between a good IT company and a bad one is that when something goes wrong with a good IT company, the damage is minimal, and the fallout is properly controlled; when something goes wrong with a bad IT company, the damage is big, and the fallout immersive.

By this definition, the revelation that O2 have been sending out their user’s phone number to every website their customers visited, and the aftermath and PR handling firmly puts O2 in the “not good” category.

Here’s The Register on what happened:

The info leak was highlighted yesterday by O2 customer Lewis Peckover, who set up a littleweb tool that displays all the HTTP header information sent to sites by connecting web browsers. These strings of data include details such as the URL of the page requested, and the web browser and operating system versions used by the person visiting the site.

For customers browsing on an O2 3G connection, these headers also include their telephone number in an x-up-calling-line-id line – added in by proxy server software most likely running on the telco’s network, rather than disclosed by a gadget’s browser or software.

The crucial part is the bit I’ve highlighted, and if you’re not in IT or familiar with the technology that makes the web work, this may not mean much to you, but what it essentially means is that O2 take every page request your browser makes, and neatly tags your phone number to the end. At the risk of making your eyes glaze over, this is roughly what a normal web request looks like:

1) User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7
2) Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
3) Accept-Language: en-us
4) Accept-Encoding: gzip, deflate
5) X-Forwarded-For: 87.114.39.218

(the numbers have been added for clarity). Whilst this may look meaningless, it’s actually really easy to understand. Lines 1-4 simply identify, in turn, the type of browser I’m using, what types of files are supported, what languages I can accept, and whether I can accept compressed (zipped) data. All of this is vital information for web-developers: if the browser identifies itself as Internet Explorer 5, for example, the developers know to weep in frustration and existential despair. The fifth line is a kind of de-facto standard that’s used to identify and log IP addresses. There’s a whole host of valid information that can be added to those 5 lines, but the general idea is that these headers are necessary for your browser to play nice with the site you’re visiting (These headers are known as HTTP headers, where HTTP is the Hypertext Transfer Protocol – one of the main protocols that the web is based on – and where that HTTP is the same http you see in almost every website address as “http://%5Bsomewhere%5D”. The more you know, and all that.)

What O2 have done is add a 6th line that looks something like this:

x-up-calling-line-id: 447590XXXYYY

Where XXXYYY is your phone number. What this boils down to is that for quite some time, if you were an O2 customer with a 3G phone, every website you visited over O2’s data network got your phone number absolutely free. This doesn’t just mean “every website you deliberately visit”, it means “every site you visit, every advertising service they use, and everything that uses the in-built browsing facilities of your phone.”

That’s a massive cockup. It’s quite difficult to overstate how big a cockup this is. But O2 managed to make it worse.

Bear in mind that the line “x-up-calling-line-id” that O2 added to the browser is not part of the standard. There’s no reason for that line to exist. It has no purpose. It’s a bit like ordering a lamp, and finding your medical history has been sent to the lamp company. So the question has to be asked: how did that get there? It’s clearly something more than a simple “oops”. There is no possible reason for O2 to add that phone number field at all. So why is it there? How did this otherwise inexplicable cockup occur?

Well O2, helpful as ever, have admitted the mistake and issued a PR disaster release explaining:

AEvery time you browse a website (via mobile or desktop), certain technical information about the machine you are using, is passed to website owners. This happens across the internet, and enables website owners to optimise the site you see.

So far, so good. That “technical information” is the HTTP headers I’ve shown above, it’s absolutely essential to your browsing experience. But the phone number isn’t. O2 continue:

When you browse from an O2 mobile, we add the user’s mobile number to this technical information, but only with certain trusted partners. This is standard industry practice.

Taking the two statements as a whole, it tends to suggest that O2 are claiming that it is standard industry practice to include certain information about the machine you are using and your phone number to website owners. But the phone number is most certainly not standard industry practice. It may be standard O2 practice, and it may even be standard practice amongst mobile phone 3G operators, but that’s hardly standard practice! There’s a standard industry response to this: its to call “BULLSHIT”.

But this still doesn’t answer why they’d want to do something so obviously stupid. O2 have this covered:

We share mobile numbers with selected trusted partners for 3 reasons: 1) to manage age verification, which manages access to adult content, 2) to enable third party content partners to bill for premium content such as downloads or ring tones that the customer has purchased 3) to identify customers using O2 services, such as My O2 and Priority Moments. This only happens over 3G and WAP data services, not WiFi.

If you’re an O2 customer, read that again and let it sink in for a moment. If you’re not an O2 customer, but have a 3G enabled phone, you might still want to let that sink in for a moment; there’s every chance that, whilst O2 got caught with their pants down, your provider is doing this too.

“We share mobile numbers with selected trusted partners”.

After asking “WTF?!”, the next most obvious question is “what trusted partners?” We don’t know, and at the time of writing O2 aren’t saying. You might also want to ask “I didn’t agree to this, where did I agree to this?”. Well, for O2, their broadband terms and conditions are online. Good luck finding such authorisation, because I sure as hell can’t find it.

So what can we surmise so far?

O2 have been caught with their pants down. They started sending their customer’s phone numbers all around the world because they have always been sending their customer’s phone numbers to unnamed “trusted” third parties — parties trusted by O2, of course, not necessarily trusted by their customers who’s confidential information they are so cavalier about. We can further guess that they cocked up configuring their proxies (systems vaguely similar to the wireless router you’re probably connecting to the internet to), and it is that that lead to private information being leaked to all-and-sundry.

But it’s only the phone number, right? No.

This is, I think, where the shit really hits the fan. In their FAQ, O2 state:

Q: Which of my information can website owners access?

A: The only information websites had access to is your mobile number, which could not have been linked to any other identifying information we have about customers.  

O2 highlighted the question, I think the answer is more interesting.

Look back up to why O2 are sharing your information: to manage age verification to manage access to adult content, to enable third part content partners to bill you, and identify customers using O2 services.

Can you guess which of those services can be carried out solely through the use of a phone number without revealing any further information about the person? Because I sure as hell can’t. Indeed the very act of answering of the question provides more information about a person. Want to use your mobile number to verify age in order to access adult services (which does not necessarily mean porn, BTW)? You have just disclosed that the person with this mobile number is over 18 years of age. You’ve potentially associated a useful demographic with a mobile number (and everything else that you may be providing in order to access adult content)

The simple fact is that it is simply not possible for the answer to that question to be true. I don’t think O2 are lying, I think their PR people haven’t the foggiest clue of the utter clusterfuck they’ve got on their hand. I don’t think they have the faintest clue of the enormity of both their cockup and their subsequent PR clusterfuck. More importantly, I don’t think they have the faintest clue just how dodgy their system was before “business as usual” – treating customer’s data with utter contempt – was shown up to the world by a simple mis-configured proxy server.

And, as I said before, this isn’t just O2 doing this. They’re just the ones that got caught out this time.