http://www.eweek.com/c/a/Mobile-and-Wireless/Balsillie-BlackBerry-Shutdown-Will-Never-Happen-Again/

Balsillie: BlackBerry Shutdown Will Never Happen Again
By Wayne Rash
2007-05-11

Research in Motion co-CEO Jim Balsillie said the BlackBerry shutdown last month was caused by a process error and the problem has been “unambiguously solved.”
ORLANDO, Fla.—In a rare one-on-one interview with eWEEK, Research in Motion co-CEO Jim Balsillie said that the event that shut down e-mail for BlackBerrys in the United States for hours last month was due to “a process thing,” and that steps had been taken to ensure that it could never happen again.

Balsillie said that the improbable combination of events, which included the failure of a minor software upgrade to a caching subsystem, the failure of the failover system and the subsequent overloading of a second system has been fixed.

“It was a process error that we had thats been fixed. It shouldnt have happened, and it wont happen again,” Balsillie said. “It wasnt a corruption of any form of the infrastructure, and thats very important.”

“Were clearly putting a lot more fault tolerance into the system, a lot more capacity. Were having domain failover architectures; were having business continuity solutions experts, so from that component piece of the infrastructure, thats not going to happen again.”

Explaining that the problem that caused the blackout was totally avoidable, Balsillie said that the company is broadening, strengthening and “fault tolerating” the system. “Its a global and public safety imperative,” he said, adding that there is no constraint on budget or resources for this work.

Balsillie did note, however, that its the responsibility of an enterprise to make sure they have continuity plans for times when important communications paths, including the BlackBerry e-mail, are out of order. He pointed out that RIM was working with customers immediately upon learning of the blackout.

“We had literally hundreds of our top customers on open bridges with ongoing collaboration and communications. So those that were affected had ongoing communications,” he said about RIMs support efforts.

Balsillie said that the critical public safety portions of RIMs customer base were brought back on line immediately. “Then the consumer portion was brought back, also quickly, but subsequently,” he said.

The question of a failover data center had been discussed after the blackout, especially by government managers who were concerned about losing a vital communications link. Balsillie said that now there is a failover center, but he will not disclose its location.

He said that the same process problem that caused the blackout also delayed the failover, but he said that RIM was still able to get critical users back on line almost immediately. “There is another hub going in the U.S. across the fault line,” Balsillie added.

“There are also architecture failovers and dual homing plans for key secret service, government and security forces,” Balsillie said. “We can view this as a mistake or we can view it as an inoculation. Its unambiguously solved.”

Click here for more on what caused last months BlackBerry failure.

Balsillie noted that the U.S. government is RIMs biggest customer, which is one reason hes taking the issue of the blackout so seriously. He said that BlackBerry devices are used across the whole range of government organizations from intelligence agencies to the military to law enforcement.

“Its part of a broad, broad system of capability. Its shifting to mission critical in every sense of the word,” he said.

On other topics, Balsillie said that he thinks that telephony integration is the “coolest” thing hes seen at the RIM symposium.

“It was considered in many respects unsolvable, but its so obvious and so powerful,” he said, “its not unlike when we did e-mail. People said why would I want e-mail on my belt, but it changed everything. Once the telephony is synchronized, it totally changes the collaboration world and once thats interrelated with your other workflow and messaging, it changes everything.”

Balsillie also hailed the popularity of navigation for the BlackBerry, noting that when people are mobile, knowing where they are can be very important. “After messaging and talking, its the most horizontal application. By definition, mobile people have location needs,” he said.

I also love this part…. Someone get Lowell McAdam on the horn and let him know “Balsillie also hailed the popularity of navigation for the BlackBerry, noting that when people are mobile, knowing where they are can be very important. “After messaging and talking, its the most horizontal application. By definition, mobile people have location needs,” he said.” Neither RIM nor it’s customers think crippling of our blackberry’s built in autonomous GPS is a smart idea. Even with the RIM network in failure mode we should still have the ability to know where we our with our built in autonomous GPS

Also, lets put this in perspective, if bb’s continue to go down once a month, mostly in the middle of the night then I will live with it.
My power goes out almost once a month due to weather, but I am not going to cancel it because I need it, just like I need my bb.