RIM… What The %$#%

hellno
#1. February 20th, 2008, at 1:02 PM.

Robb,

RIM’s service failures and outages are sadly past the point of surprise and are to be expected knowing RIM’s proprietary, single choke point, single NOC network.

Where is Balsillie’s next statement? “It shouldn’t have happened, and it won’t happen again,” Balsillie said (April 2007)

RIM’s competitors have technologies available which are much more suited for mission critical data communication. Palm was so right with their ad’s from last week, sure Palm’s current hardware and software is WAY behind current standards however everyone with a Palm and EVERY other device not dependent on RIM’s proprietary, single choke point, single NOC network were able to function as normal while all those Blackberry’s, paying for blackberry service had nice looking cellphones without access to any data communication. Fanboys more worried about defending RIM against RIM’s own failures instead of discussing the truth can try to FUD and say “once every few months isn’t going to hurt. I just resort back to using my laptop and booting it up. Big deal. It’s also a great excuse to just stop looking at the damn thing anyway.” However that doesn’t help those which thought RIM and it’s Blackberry was mission critical and have again lost a vital communications link. Bet fanboy’s don’t get that the U.S. government is (maybe was after all these outages) RIM’s biggest customer. One can only guess the mess RIM’s now rather constant network failures cause when public safety, secret service, government and security forces data communication fails. To say nothing about the $$$ business and corporate lose with RIM’s now rather constant network failures.

Robb, IMO RIM’s problems go WAY beyond not having someone on the payroll who knows how to run a data center. To start “a data center” in the single sense is a just a piece of the problem.

Competitors have technologies available which are much more suited for mission critical data communication, now that RIM has completely dropped the ball multiple times with their poorly designed proprietary, single choke point, single NOC network. It’s currently christmas in february for RIM’s competitors

Mickey
#2. February 20th, 2008, at 1:31 PM.

Hellno, we get it…we know that RIM has a proprietary, single choke point, single NOC network. Don’t you get tired of typing that in every paragraph?

bluehorseshoe
#3. February 20th, 2008, at 1:31 PM.

@ hellno

Uh…the Government runs on BES, thus are not affected by the outage, hence why they run on BES (hint, hint…it’s called security). Corporations…same deal for the most part unless they are too small to run BES.

I think the person that doesn’t get it is you. It’s funny to watch you type all that crap and then get the ‘big dumb-ass award’.

Preach on hellno…I’m sure you will.

jshel
#4. February 20th, 2008, at 1:35 PM.

I just got a push with about five emails.

Perhaps the outage is over.

Carnell W
#5. February 20th, 2008, at 1:58 PM.

I haven’t had any issues today. ATT BIS in California

hellno
#6. February 20th, 2008, at 2:06 PM.

bluehorseshoe seems your “crap get(s) the ‘big dumb-ass award’.” Yet again your diarrhea posts come back to prove you the ignoramus you truly are. Did you ever notice or read that BOTH BIS and BES customers are effected since all the data goes thru RIM’s proprietary, single choke point, single NOC network. You do know how to read right? I know you’ve grandstanded before that you like to ignore the TRUTH and REALITY that others post. Sit back, READ and LEARN for a change. Your pie hole only reaffirms your ignorance. Not that we expect anything different from a vealot.

“Typically, enterprise customers install the BlackBerry Enterprise Server software behind their firewall, and link it to the corporate e-mail system, usually Microsoft Exchange. The BES constantly checks with Exchange, asking if new e-mails have been delivered to the users’ inboxes. If there is new e-mail, it’s handed over to the BES, which then goes through the corporate firewall to the RIM NOC, handing it over to BlackBerry servers there. Those systems connect with the appropriate cellular carrier network to deliver the e-mail to the recipient’s BlackBerry device.”

For the failure which happened Feb 11th 2008
Reportedly, an e-mail sent by RIM support account manager Bryan Simpson affirmed that a “critical severity outage” had taken place. The message went on to say that it was acting as an “emergency notification regarding the current BlackBerry Infrastructure outage,” that was (is) affecting enterprise subscribers and “users of the Americas network.”

Is Most Recent BlackBerry Outage Strike Three For Failing to Test System Changes?
http://blogs.ittoolbox.com/security/connection/archives/is-most-recent-blackberry-outage-strike-three-for-failing-to-test-system-changes-22474

TRUTH and REALITY before constant bluehorseshoe ignorance.

bluehorseshoe
#7. February 20th, 2008, at 2:14 PM.

Where do you see ‘BES’ in Robb’s post? LOL…you really go by everything you read with those links, etc. Do you know the difference between the two? Did you ever work in an IT environment?

Please, just answer that one question.

crazylegs
#8. February 20th, 2008, at 2:34 PM.

@ hellno - fyi today’s outage only affected BIS customers. BES customers are unaffected. Last week’s 3 hour outage affected both, but again, it was down for only 3 hours. No doubt this is very frustrating to BB users who depend on their devices but to say there is any other solution that is competitive is ignorant. Name one competitor technology that replicates what BB has all the way down to security, handset policies, remote management, ascendant mobility (i.e. PBX integration with the corporate PBX), etc. out of the box. There isn’t. You’re ignorant to believe there is.

The BB network has to have the NOC to enable the secure connection from device to NOC to BES. The BB system is built around security from the ground up, so the government customers can’t use anything else. Does WM, Palm, Symbian, or iPhone have all those security certifications RIm does? You have to have the NOC outside of the firewall as the verification point between device and corporate mail server, so this architecture is here to stay. RIM should at least double their NOCs for redundancy purposes and they should be taken to task with this issue.

Last week’s outage was due to a botched upgrade, we’ll see what today is about. Hopefully, these are just growing pains and they will learn and get better. If not, then the competition might catch up in a few years.

Aodhan
#9. February 20th, 2008, at 3:18 PM.

Yep, another outage. I’ve had my Blackberry 8300 for only 9 months. One more outage and I’m through with Blackberry. There are so many outages its embarrassing.

AF
#10. February 20th, 2008, at 3:23 PM.

hellno, are you in any way related to Glavatron over at BoyGenius?

bluehorseshoe
#11. February 20th, 2008, at 3:33 PM.

@ AF

He just may be Galvatron! LOL…

I can see hellno now…looking up all those blogs out there to see if there’s anything to dispute what crazylegs just posted above. He’s putting those links tucked away on the side, looking for more info as he contructs his ‘War & Peace’ reply for us all to read. This one is going to take some time however…it’s not a simple copy and paste anymore from previous comments. He’s actually being called out now by others regarding his lack of technical knowledge. It’s no longer opinion, it’s FACT.

hellno
#12. February 20th, 2008, at 3:38 PM.

Robb started this thread to discuss “Research in Motion is having another outage” It (like every other thread here at RIMarkable) have nothing to do with personal opinions of posters. If you want to dicuss the topic great, but the thread crapping, calling people out, and name calling is LAME. Each of RIM’s outages are bad for RIM and for us it’s customers, it’s too bad fanboys cannot handle others posting thoughts and having discussions which are both pro and con about topic’s Robb our host starts.

bluehorseshoe
#13. February 20th, 2008, at 3:45 PM.

@hellno

WRONG. You brought up something, knocking the entire service when in fact it was only BIS that is having the outage. You took the opportunity to knock something that you have very little knowledge of. So now the others that share this blog call you out on it and you have no reply except trying to back out by referring to the topic of the article, which again, you tried to take a shot at with your usual propoganda. This is why you’re so loved on this board.

bluehorseshoe
#14. February 20th, 2008, at 3:54 PM.

oh…and name calling?

What the hell is zealot, vealot, sheep, mental capacity, fanboy, powderpuff, and other retarded crap you paste in there?

You truely are insane. Robb should just cut your ass off so we all can move forward. I happen to use and like the BB. You claim iPhone dominance and how it will cure cancer. Go to an iPhone site then. You won’t see me there, so go have fun.

bluehorseshoe
#15. February 20th, 2008, at 4:07 PM.

…and just to prove you don’t know what the hell you’re talking about when it’s about YOU…

All this in the same paragraph (it’s unreal…LOL):

“It (like every other thread here at RIMarkable) have nothing to do with personal opinions of posters”.

- in other words, opinions AREN’T allowed when an article is posted.

…now here’s the lower half of the paragraph:

“it’s too bad fanboys cannot handle others posting thoughts and having discussions which are both pro and con about topic’s”

- so now you are saying that personal opinions CAN be shared, but in a respectful way, perhaps?

In one paragraph you managed to contradict yourself. You are confused and way over your head. It’s time to ride of into the sunset hellno…you’ve been one hell of an entertaininer, that I’ll give you.

hellno
#16. February 20th, 2008, at 4:09 PM.

In a attempt to keep any thread topic flowing around the daily thread crapping.

crazylegs,

Not that RIM’s BB security is absolute for many users out there. Are any of the alternatives as secure? Know there is only one RIM, but with RIM’s current rash of outages and it’s proprietary, single choke point, single NOC network it’s probably not the best idea to have absolute reliance in just one technology from one data provider. During the legal battle with NTP I remember a few lists of Blackberry alternatives out there, now that technology has had 2 years I’m sure there are more.

Quickly found one:

“Good Mobile Messaging from Good Technology™ provides Palm users with real-time, push email/PIM and the most end-users features available. IT managers benefit from Good Mobile Messaging deployments due to enhanced easy remote device management; enhanced security features such as AES encryption, FIPS 140-2 certification, remote password policy changes; and remote data kill capabilities. Good Mobile Messaging supports wireless access to IBM® Lotus® Notes® or Microsoft® Exchange”

What about Palm’s noted “uptime” solution of letting the corporate exchange server do the work without the need for the NOC and 3rd party support? Sending something encrypted and private to a mobile device is alot like sending encrypted and private data to a laptop located off the corporate intranet no?

With all egg’s in one basket RIM is showing it’s not doing a good enough job to deserve to providing failure prone service. Sure there is only one RIM but the outages are nothing to to FUD about and say “once every few months isn’t going to hurt. I just resort back to using my laptop and booting it up. Big deal.” It’s not a flame to know and discuss valid alternatives out there.

Mojo
#17. February 20th, 2008, at 4:12 PM.

The guy that runs this site must have the patience of Job because I would have banned both of you folks a long time ago.

bluehorseshoe
#18. February 20th, 2008, at 4:24 PM.

@ Mojo

I’d agree except when I commented on the VZW Pearl coming out and how I was going to move from TMO to VZW for the better coverage and tethering, BAM, I got a shot of hellno telling me I’m a vealot sheep who is downgrading to VZW and how they suck and how I’m making a bad decision and foolish for joining the others that adhere to VZW’s crippling, etc. blah blah blah.

At first I was offended like everyone else who comes across this person because they obviously don’t know how to communicate, but now it’s just flat out stupidity.

hellno
#19. February 20th, 2008, at 4:29 PM.

Again since bluehorseshoe has shown his desperate need for attention and yet again shares his temper tantrum with the rest of us here in the form of yet another thread crap taking personal shots at others…..

Try as hard as you can to have some respect for others here who want to post ideas on RIMarkable. No one here wants or needs ANY of your thread craps and abusive comments. Robb has asked for it to STOP either YOU will STOP or you will be STOPPED. Post about the thread topic not about your opinion of fellow posters

Robb started this thread to discuss “Research in Motion is having another outage” It (like every other thread here at RIMarkable) have nothing to do with anyones personal opinions of fellow posters but of the thread topic’s. If you want to discuss the topic great, but the thread crapping, calling people out, and name calling is LAME. Each of RIM’s outages are bad for RIM and for us it’s customers, it’s too bad fanboys cannot handle others posting thoughts and having discussions which are both pro and con about topic’s Robb our host starts.

Robb better start filtering everything at RIMarkable making sure everything posted is positive about RIM, NO more posts about past of future RIM network outages, heck even the moto/rim lawsuit isn’t on solid ground as it could bring on temper tantrums. Either that or some here are going to have to go on permanent vacation so the rest of us can continue discussing and posting like adults, instead of lowering ourselves to give special attention to those which can’t handle others opinions and have desperate needs to stoke their own ego’s.

hellno
#20. February 20th, 2008, at 4:38 PM.

Mojo as you can see bluehorseshoe doesn’t have the ability to take RESPONSIBILITY for his own posts always blaming someone else for his piss poor actions and lack of respect. Robb has asked him to stop his nonsense several times and all RIMarkable gets is a empty apology which is sadly a guarantee that bluehorseshoe will be back to his thread crapping and name calling in less than 24 hours.

I’m sure most of us would like to be able to have discussions which are both pro and con about topic’s Robb our host starts wIthout the anti-thread services of bluehorseshoe. His posts continue to show us all what special (needs) person he is.

Greg B
#21. February 20th, 2008, at 4:44 PM.

Hellno, The Good Messaging and Palm alternatives may be alternatives for corporate e-mail, but they are not BIS alternatives. I could be wrong, but as far as I know, this is no true alternative to BIS currently out there (and what I mean by that is, true push e-mail for a personal, noncorporate user). Maybe Sidekicks? I’m sure someday there will be, so you are right to emphasize that RIM should not get lazy about such things.
But by the way, calling someone out for going off topic and name calling… geez, talk about the pot calling the kettle black!

hellno
#22. February 20th, 2008, at 4:46 PM.

Whats really LAME is on topic discussions have been curtailed due to this BS. Why must RIMarkable continue to be used as bluehorseshoe’s thread crapping toilet?

#18. January 12th, 2008, at 10:01 AM.
“If I took a vacation from the board, he/she would still find a way to offend someone with their choice of words and delivery. I’m not the only one who has ripped this person, just the only one who continues having fun with this disadvantaged fool.”

Not that everyone missed what “fun” is to bluehorseshoe, I find his fun very disrespectful to RIMarkable, fellow posters and Robb. In a way bluehorseshoe’s actions are disrespectful to the RIM culture.

hellno
#24. February 20th, 2008, at 4:57 PM.

Greg,

Thank you!

Wouldn’t BIS be easier to find a alternative to? IMO There are several automatic “pull” type mobile email programs which are solid performers. IMO I’d think BES customers would have more needs (security, sync with corporate servers, etc etc) which I think Good and others like them could be valid alternatives to having full reliance in one company with a proprietary, single choke point, single NOC. I too agree and hope RIM doesn’t continue fooling around with the current network design. I agree with Robb pointing out competitors, perhaps RIM has gotten to big to be both a mobile device design and manufacture, and also responsible for running the server back end to support all those devices. Does Google, Yahoo run it’s server back end completely in house? Not saying SE Asia should be getting a call from RIM at any time to farm any staff out, but perhaps a company better suited to reliable servers should be contracted or tapped to handle the BES/BIS server back end?

bluehorseshoe
#25. February 20th, 2008, at 5:13 PM.

@ Robb,

Comon’. Traffic and clicks equal bling bling.

OK. For your sake, I’ll stop. I’m done entertaining and playing, but in all honesty, I wouldn’t be surprised if his fan club grows even larger. Let’s all be real here, and yes, I myself know, we’re not dealing with someone who has all their marbles when discussing the other half. It’s just soooo hard to resist. So yes, for the better of mankind, I will back off.

PS - I’m surprised it lasted as long as it did. I really can’t believe someone would keep repeating the same stuff, over and over and over…

FrozenCricket
#26. February 20th, 2008, at 6:01 PM.

wouldn’t it be funny if hellno and bluehorseshoe were one in the same? hellno….. hello bi-polar disorder!

crazylegs
#27. February 20th, 2008, at 6:19 PM.

hellno - missed the whole point and I now see that you really aren’t that knowledgeable about this topic. Seems that emotion drives your posts with all that name calling and such. Well, let’s talk about this like adults to see why your arguments don’t stand up:

1. I said the same features out of the box. When you pick up a blackberry, it’s ready to go with minimal set up. I know about Good (which was bought by Motorola and never to be heard from again), Seven, Visto, and others. But these are software add-ons, not ready to go out of the box. ALSO, these do not offer PBX integration like RIM now does in the enterprise through Ascendant Mobility.

2. Look at Good’s architecture and it’s basically the SAME as BB with a NOC as the “choke” point as you say. SO, your argument is moot. They would have the same issues.

3. For the average consumer, they are simply not going to go to the extra effort to find, install, and most importantly PAY for the Good, Seven, or Visto service. Once you get the BB and the data plan, with a fews minutes of setup you are good to go on email. Only us geeks would consider another offering because we enjoy the tinkering ostensibly.

4. If I am managing a huge fleet of devices for an enterprise, I’m not going to risk my ass with a software add-on. I am going to stick with the leader, especially if they let us integrate the corporate PBX so that my employees have one number that rings at the desk and on the BB, having all PBX functionality on the BB. This brings cell phone calls behind the firewall and lets enterprises control that corporate asset.

5. To change it up a bit - Google and Yahoo absolutely run their own servers in the data centers. Are you joking? Do you think Google would let some joker run their servers and give them access to the search algorithms and any other trade secrets on those servers. RIM is not too big too run both the back end and design devices. These are two separate functions. The outages have occurred during upgrades (according to RIM) so the basic architecture is sound and is extremely reliable.

The bottom line is that RIM has its faults and needs to grow up and fix them because the reliability question will only grow if more outages occur. Please note that when BIS goes down, the mainstream press never talks about it because every big hitter is on a BES. When BES goes down it is a huge deal. Hopefully, they can fix these issues and move forward.

AND the best news is that EACH time there IS an outage, RIM gets the system up and running in a very short time. In the grand scheme, they still can claim 99.9% up time, which is pretty darn good.

So, hellno, flame away at me! WORD!

bluehorseshoe
#28. February 20th, 2008, at 6:35 PM.

@ FrozenCricket

That would be funny, but I don’t cut and paste that often.

j.a.
#29. February 20th, 2008, at 7:02 PM.

Reporting from Venezuela… BIS down all the afternoon in Movilnet (CDMA) and Movistar (CDMA & GSM)

hellno
#30. February 20th, 2008, at 7:30 PM.

crazylegs,

RIMarkable is a place for RIM fans to post, discuss and learn from others opinions why so hostel? Why make a statement about others you have zero clue about? You don’t wish to be personally judged and talked at? Why should I have reason to flame what you posted? When you post with respect you get respect, bh is a special case he doesn’t and gets neither. Come on strive to be a better RIMarkable poster than him

I didn’t miss anything you asked and as posted most statements about alternatives are question, as none of them have been proven. What I do know is reliance on one and only one network which is far from mission critical is not a good idea either. Sure RIM might be the market leader but with they outages we both know IT’s from coast to coast are looking for and researching alternatives. I also do not believe for a second that there are not other companies with technologies out there which meet and exceed RIM’s current offerings in both secure and reliable data communications. Like RIM everything needs to start somewhere.

“For the average consumer, they are simply not going to” pay for RIM to push data and email at them when a smartphone or handheld computer can perform more reliable without paying for a service from a 3rd party. I do totally agree with you that there is only one RIM and RIM’s device and BIS/BES service make the Blackberry a very simple device with little to setup. But just as consumers have somehow set up their desktop and laptop computers to get on the net and get email I think consumers putting the same settings into a mobile device (or having a smart device even transfer those settings automatically) is a very real possibility.

crazylegs you have many good points I agree with. Again I just find fault with complete device reliance on a single provider as we’ve learned if the RIM servers and/or single NOC have a problem it’s no data time for every Blackberry out there. Some of this just proves RIM does what they do very well, but as we’ve found when things at RIM go bad, they can really go bad for many/all customers.

RIM can do better and as well as other technologies need to start proving themselves.

Felipe
#31. February 20th, 2008, at 7:37 PM.

Reporting from Santiago, Chile, problems all day long…

Admin
#32. February 20th, 2008, at 7:45 PM.

Make sure you subscribe to the bb-outage mail list if you want to be kept up to date on what’s going on with it. We also have 3 other lists dedicated to BB users.

On a side note, I like Palm’s new marketing where they use Uptime as a reason to switch over. How can they claim uptime when their devices are rebooting all the time?

hellno
#33. February 20th, 2008, at 8:03 PM.

Over at BG they are saying “around 80% of all Berry devices in the continental US” are affected by todays failure.

Posted by Joshua Karp on Feb 20, 2008 10:53 am
Blackberry email goes down…again
http://www.boygeniusreport.com/2008/02/20/blackberry-email-goes-downagain/

“This is getting a little bit ridiculous, folks. We here at BGR have trouble putting our Blackberries down. Seriously. It’s a problem. That said, we’re getting a little bit frustrated with RIM following the 3rd service interruption in 9 months, and the second in a week. It’s poor form. Following the notorious blackout last year, RIM promised that it would never happen again. Hollow promises, as around 80% of all Berry devices in the continental US have been affected by a service outage that began around 3 am this morning. No carrier has been spared, with reports of crippled devices from Verizon, T-Mobile, AT&T, Nextel, and Sprint. It seems that some users are finally getting back up online, but there’s been no official word from RIM regarding a projected time of resolution. Forgive us, but we’re beginning to question the reliability of our beloved BlackBerries. So sad, they’ve turned into Sidekicks.”

Jon
#34. February 20th, 2008, at 8:13 PM.

Didn’t effect me, not sure what the big deal is when it’s down a few hours. People need a life.

Allan Ross
#35. February 20th, 2008, at 9:46 PM.

Oh relax everyone. It’s technology. It sometimes has issues. It sometimes doesn’t work right. Get over it. Unless someone, somewhere can prove that this caused a life threatening or otherwise damaging situation then this is just an inconvenience nothing more. If you really need to communicate with someone use the phone. You remember - the other thing that your Blackberry does.

If you REALLY want to complain - talk to your cel service provider and ask for a credit for the time the system was down. They are the ones who you pay for the Blackberry service - that way they can lean on RIM. Hit them in the wallet and then they will listen.

‘Nuff said.

AF
#36. February 21st, 2008, at 8:36 AM.

Right on Jon!

crazylegs
#37. February 21st, 2008, at 8:55 AM.

@ hellno - ok, didn’t mean to be hostel, sorry, I was just really frustrated by the yesterday’s outage. feeling better today since RIM upped their subscriber forecast and the stock is up over 10 points in the pre-market. that’s one way to sweep outages under the rug!

jonabyte
#38. February 21st, 2008, at 10:38 AM.

I had to respond to some of these comments regarding a noc. It is not another noc that is needed, but a redundant data centre to switch on when the other one is down. The noc monitors the data centre.

Also, lets put this in perspective, if bb’s continue to go down once a month, mostly in the middle of the night then I will live with it.
My power goes out almost once a month due to weather, but I am not going to cancel it because I need it, just like I need my bb.

Jon
#39. February 21st, 2008, at 10:45 AM.

Exactly jonabyte. Sometimes our cars break down, computers reboot, drives crash, nothing is 100% except death & taxes. Use the phone to call or jump on a PC to check your email. Crazy people.

hellno
#40. February 21st, 2008, at 5:44 PM.

I don’t think it’s too crazy. Again the US Government is RIM’s biggest customer. Think of all the security, first responders…. RIM is at least perceived at mission critical with mutiple system failures, and some knowledge of RIM’s system, the truth is RIM’s systems is hardy mission critical. It’s easy for some to kickback in the easy chair and write these outages off as “big deal”, “read a book”, “get a life”, “relax”, “get over it”, but big picture it’s both naive and short sighted, it’s sad to think (or a feather in the hat of technology) but at least some could be put in danger if their perceived mission critical Blackberry fails to keep them with timely data. Especially unacceptable about RIM’s situation is not only have they known about their networks problematic design for years now, and have not done anything to keep the problems from happening, but senior members of the RIM team have told the press tall tales saying “outages won’t happen again” to money is not a problem and they will do what ever they need to do to make sure there is layers of redundancy to the network.

bluehorseshoe
#41. February 21st, 2008, at 6:04 PM.

The government has satellites falling out of the sky, devices that guide missiles and miss targets, I think they can live with a couple of hours of downtime.

hellno
#42. February 21st, 2008, at 7:12 PM.

I’m not so sure it’s up to any of us to try and smooth over a outage, making excuses why RIM’s system failures are ok. IF Balsillie was being truthful back 5/07 wouldn’t the system redundancy work to the network be at least well on it’s way, if not complete? (We know the “will never happen again” part didn’t quite live up to being true) If Balsillie claims the RIM Blackberry system is “imperative to global and public safety” and “Its part of a broad, broad system of capability. Its shifting to mission critical in every sense of the word,” who are we to disagree with him?

http://www.eweek.com/c/a/Mobile-and-Wireless/Balsillie-BlackBerry-Shutdown-Will-Never-Happen-Again/

Balsillie: BlackBerry Shutdown Will Never Happen Again
By Wayne Rash
2007-05-11

Research in Motion co-CEO Jim Balsillie said the BlackBerry shutdown last month was caused by a process error and the problem has been “unambiguously solved.”
ORLANDO, Fla.—In a rare one-on-one interview with eWEEK, Research in Motion co-CEO Jim Balsillie said that the event that shut down e-mail for BlackBerrys in the United States for hours last month was due to “a process thing,” and that steps had been taken to ensure that it could never happen again.

Balsillie said that the improbable combination of events, which included the failure of a minor software upgrade to a caching subsystem, the failure of the failover system and the subsequent overloading of a second system has been fixed.

“It was a process error that we had thats been fixed. It shouldnt have happened, and it wont happen again,” Balsillie said. “It wasnt a corruption of any form of the infrastructure, and thats very important.”

“Were clearly putting a lot more fault tolerance into the system, a lot more capacity. Were having domain failover architectures; were having business continuity solutions experts, so from that component piece of the infrastructure, thats not going to happen again.”

Explaining that the problem that caused the blackout was totally avoidable, Balsillie said that the company is broadening, strengthening and “fault tolerating” the system. “Its a global and public safety imperative,” he said, adding that there is no constraint on budget or resources for this work.

Balsillie did note, however, that its the responsibility of an enterprise to make sure they have continuity plans for times when important communications paths, including the BlackBerry e-mail, are out of order. He pointed out that RIM was working with customers immediately upon learning of the blackout.

“We had literally hundreds of our top customers on open bridges with ongoing collaboration and communications. So those that were affected had ongoing communications,” he said about RIMs support efforts.

Balsillie said that the critical public safety portions of RIMs customer base were brought back on line immediately. “Then the consumer portion was brought back, also quickly, but subsequently,” he said.

The question of a failover data center had been discussed after the blackout, especially by government managers who were concerned about losing a vital communications link. Balsillie said that now there is a failover center, but he will not disclose its location.

He said that the same process problem that caused the blackout also delayed the failover, but he said that RIM was still able to get critical users back on line almost immediately. “There is another hub going in the U.S. across the fault line,” Balsillie added.

“There are also architecture failovers and dual homing plans for key secret service, government and security forces,” Balsillie said. “We can view this as a mistake or we can view it as an inoculation. Its unambiguously solved.”

Click here for more on what caused last months BlackBerry failure.

Balsillie noted that the U.S. government is RIMs biggest customer, which is one reason hes taking the issue of the blackout so seriously. He said that BlackBerry devices are used across the whole range of government organizations from intelligence agencies to the military to law enforcement.

“Its part of a broad, broad system of capability. Its shifting to mission critical in every sense of the word,” he said.

On other topics, Balsillie said that he thinks that telephony integration is the “coolest” thing hes seen at the RIM symposium.

“It was considered in many respects unsolvable, but its so obvious and so powerful,” he said, “its not unlike when we did e-mail. People said why would I want e-mail on my belt, but it changed everything. Once the telephony is synchronized, it totally changes the collaboration world and once thats interrelated with your other workflow and messaging, it changes everything.”

Balsillie also hailed the popularity of navigation for the BlackBerry, noting that when people are mobile, knowing where they are can be very important. “After messaging and talking, its the most horizontal application. By definition, mobile people have location needs,” he said.

I also love this part…. Someone get Lowell McAdam on the horn and let him know “Balsillie also hailed the popularity of navigation for the BlackBerry, noting that when people are mobile, knowing where they are can be very important. “After messaging and talking, its the most horizontal application. By definition, mobile people have location needs,” he said.” Neither RIM nor it’s customers think crippling of our blackberry’s built in autonomous GPS is a smart idea. Even with the RIM network in failure mode we should still have the ability to know where we our with our built in autonomous GPS

Phreqd
#43. February 21st, 2008, at 10:51 PM.

For everyone that’s got their knickers in a twist over this, please move on to another platform. Obviously the ‘berry isn’t for you because it’s not perfect. I don’t hear the Government complaining or switching to another device. It didn’t even make the papers here in the D.C. area and this is where the Feds have the most employees with Blackberrys. Get a life and live without emails for a few hours. Or check your computer if it’s so damned important. If your whole email-life is tied up in a smartphone, you’ve got problems putting all your eggs in one basket.

Jon
#45. February 22nd, 2008, at 7:21 PM.

US government dependent on a Canadian company for mission critical communications? How sad.