A System Upgrade Caused The BlackBerry Outage

Thought
#1. February 14th, 2008, at 12:29 PM.

An excuse is still no excuse. And in this day of so many sites, like Google, that manage to keep their servers up and running without these sort of outages, why cannot RIM do the same?

I realize that RIM has a lot of traffic, but it is mostly text based email. It shouldn’t be that difficult to set up a distributed network of servers that is virtually failsafe.

hellno
#2. February 14th, 2008, at 12:47 PM.

Exactly Thought! But it was “unambiguously solved.” said Balsillie back on 5/13/07.

Only fanboy’s make excuses for RIM and believe:

http://www.fiercemobileenterprise.com/story/rim-failure-won-t-repeat/2007-05-14
“RIM failure won’t repeat
May 13, 2007
RIM CEO Jim Balsillie explained the recent Blackberry outage as a process problem and said that steps have been taken to make sure it doesn’t happen again. In an interview with eWeek’s Wayne Rash, Balsillie talked about the sequence of problems that caused the system-wide outage, and the steps RIM has taken to make sure this was a one-time catastrophic event. How sure is he that they’ve got the problem solved? He’s confident enough to say that it’s “unambiguously solved.”

——

“”From the CEO all the way to IT managers and the average person walking into a wireless store at the mall, they are all going to ask the question,” Levy said.
“Isn’t that the device that’s always going down? At some point, it is potentially damaging to the brand said Levy, senior-vice president of strategic consulting at Toronto’s AR Communications.

“Clearly an architecture where all of your traffic is routed into a relatively small choke point is not sufficient when you are responsible for servicing tens of millions of customers,” Levy said.
“Imagine if Google were suddenly unavailable - the hue and cry that would result from such an outage.”

The BlackBerry service has become a lifeline across North America for many business executives, and workers in public-sector services, ranging from political aides to fire, police and ambulance services and others. (And Fanboys, quick to make excuses)

“But because BlackBerry is really using a proprietary network approach, all it takes is their servers to go off line and the entire network crawls to a halt.”

The whole RIM/Blackberry system is looking less and less mission critical, IT’s are finding other solutions, a proprietary single NOC is not acceptable.

Fletchi18
#3. February 14th, 2008, at 1:50 PM.

I thought for a second that maybe it is because there has been such an explosion of interest in BB devices and RIM has been able to keep up with demand, but not able to keep up with its failovers/DR. But then I thought, when was the last time I saw Google or Wikipedia go down (following your thought…errr….Thought heh heh)? Those sites have seen even larger growth spikes over the last few years and they’ve got the infrastructure in place so they’re up darn near 100% of the time.

Not that I was terribly upset at the outage. In fact, if it wasn’t for a mailing list I’m on, and I didn’t have users whining and moaning about the outage, I would never have known. But, to have an outage on that scale is pretty inexcusable.

Thought
#5. February 14th, 2008, at 4:48 PM.

Fletch: that’s exactly my point…take a site like Google…which serves up far more bytes of data on a daily basis than RIM does. I would bet the majority of the traffic through RIM’s servers consists of simple, relatively short text based emails.

If Google, Yahoo, et al. can keep their sites up virtually 100% of the time, then why not RIM? Take Hostway for another example…they are a great hosting company, and are almost never down.

I guarantee you that if RIM were to hire Google to manage their server operation, their service would never go down.

bluehorseshoe
#7. February 15th, 2008, at 9:21 PM.

Still, Google has a market cap of over 100 billion (and it was twice that amount not too long ago) over RIM, so yeah, I’d think they can afford to put such an infrastructure in place. Not to mention they are in a higher margin business. So, before we hit RIM below the belt for some networking they need to address, let’s at least not compare them to Google.

As for the reason they were down, sure, it happens. Nothing is ever perfect, and we all settle for the service that has the least amount of dropped calls, etc.

Thought
#8. February 15th, 2008, at 10:01 PM.

bluehorseshoe: granted, Google does have huge resources. But RIM has a market cap of about 53 billion as of today, and so they are not exactly starved for cash. And their margins on their hardware, along with their continuing service revenues, gives them a good profit margin as well.

So are they as rich as Google? No, but they do not need to be to have a reliable network infrastructure, and they certainly are a company with a very healthy financial picture that can afford to fix their network.

Plus, it’s not just Google that doesn’t go down. There’s Yahoo (market cap 41 billion), and even service providers like Hostway, which is just a privately owned company. So you don’t have to be Google to set up a reliable network.

Plus, I would honestly surmise that serving up web pages, which often have rich multimedia components, along with operating an email service, a messenger service, etc, like Google, Yahoo, involves more data served than the type of email service that RIM offers, as the vast majority of the emails that pass through RIM’s servers are relatively small and entirely text based.

Johnny
#9. February 16th, 2008, at 1:32 AM.

Well! What can I say? Although I’m a proud owner myself of a 8830 BB, my view on this BlackBerry “madness” differs from most people: It’s true that communications has done a great leap forward thanks to mobile devices but to rely only on this for getting critical informations (like police and fire services, the manager who is absolutly dependant on it, etc.) is nonsense!

I’m an IT manager and relying on my BB only would quite stupid! What would my users say if I were to say “I’m sorry I didn’t realize the [company] servers were down because my BB didn’t work”? My boss would probably want to meet me the next morning in his office!

Since when did we start underestimating the “plan B approach”?

Think people! Think! Phones still exist and stop crying for Pete’s sake! It was a few hours outage, not a full week!

Johnny

bluehorseshoe
#10. February 16th, 2008, at 12:49 PM.

@ Thought

I’m sure RIM didn’t anticipate such rapid growth and expansion when they first started and developed the BB. Furtermore, their network I’m sure was built based on anticipated business models that they had when introduced to outside investors. Companies don’t just toss away an infrastructure and equipment right away. There’s a replacement model for the migration I’m sure…

KP (Kliner Perkins) for example doesn’t just give money away to firms they invest in. They protect themselves by sending in their own to guide their investments (I’m very familiar with them and the process having met with them from the outside…they interviewed me for one of the companies I had worked for), etc. I’m sure RIM is working on the hardware side to ensure less of these situations occur, andit’s pretty eveident based on the reasons of upgrading software, etc. Google is so SW centric that the money they do spend is talent and HW. Since they are a Linux shop, that keeps costs down on the HW, and they just create their own Linux support by hiring it. Perhaps it’s a plan that RIM should look into, but then again, they are not on the growth projection like Google is, so smaller steps for them. Heck, even hosting companies offer 99.999% uptime. Now why is that? Because sometime sh*t happens. I’ve seen corporate email go down more than the BB network, but it happens. Internet access, down as well. Even in my house with Time Warner, it goes down a few times a year.

So, with all that being said…yes, it would be nice to have 100% uptime and no issues, an ideal world, but if it’s for a few hours I’m sure we can live with it. Slight downtimes isn’t going to make people or companies just flat out dump a service or product because of issues, etc. If that were the case, Microsoft and Windows would be bye-bye.

hellno
#11. February 16th, 2008, at 1:12 PM.

Not sure whats worse RIM’s repetitive system outages, RIM fanboy’s posting constant FUD, or RIM fanboy’s posting FUD while desperately trying to make excuses for RIM’s repetitive system outages.

I’m sure public-sector services, ranging from political aides to fire, police and ambulance services and others find that a “few hours” every few months of RIM system outages is peachy too.

RIM knows whats going on just like others in the know, know the TRUTH and know RIM’s self made single choke point, a proprietary single NOC is not acceptable for mission critical data.

Sorry fanboys there is NO EXCUSE. Oh and Time Warner doesn’t provide mission critical data, sorry, nice try, but again shows the FOOL. Also INCORRECT (again) RIM’s repetitive system outages WILL do nothing but help Palm and Windows as IT’s dump RIM’s flawed BB service in favor of devices which don’t live or die when a single choke point, RIM’s single NOC goes down, turning BB devices into paper weights.