IT Jobs
Microsoft server crash nearly causes 800-plane pile-up
Failure to restart system caused data overload.
By Matthew Broersma | Techworld
Published: 16:42 GMT, 21 September 04
A major breakdown in Southern California's air traffic control system last week was partly due to a "design anomaly" in the way Microsoft Windows servers were integrated into the system, according to a report in the Los Angeles Times.
The radio system shutdown, which lasted more than three hours, left 800 planes in the air without contact to air traffic control, and led to at least five cases where planes came too close to one another, according to comments by the Federal Aviation Administration reported in the LA Times and The New York Times. Air traffic controllers were reduced to using personal mobile phones to pass on warnings to controllers at other facilities, and watched close calls without being able to alert pilots, according to the LA Times report.
The failure was ultimately down to a combination of human error and a design glitch in the Windows servers brought in over the past three years to replace the radio system's original Unix servers, according to the FAA.
The servers are timed to shut down after 49.7 days of use in order to prevent a data overload, a union official told the LA Times. To avoid this automatic shutdown, technicians are required to restart the system manually every 30 days. An improperly trained employee failed to reset the system, leading it to shut down without warning, the official said. Backup systems failed because of a software failure, according to a report in The New York Times.
The contract for designing the system, called Voice Switching and Control System (VSCS), was awarded to Harris Corporation in 1992 and the system was installed in the late 1990s, initially using Unix servers, according to Harris. In 2001, the company completed testing of the VSCS Control Subsystem Upgrade (VCSU), which replaced the original servers with off-the-shelf Dell hardware running Microsoft Windows 2000 Advanced Server. The upgrade was installed in California last year, according to the FAA.
Soon after installation, however, the FAA discovered that the system design could lead to a radio system shutdown, and put the maintenance procedure into place as a workaround, the LA Times said. The FAA reportedly said it has been working on a permanent fix but has only eliminated the problem in Seattle. The FAA is now planning to institute a second workaround - an alert that will warn controllers well before the software shuts down.
The shutdown is intended to keep the system from becoming overloaded with data and potentially giving controllers wrong information about flights, according to a software analyst cited by the LA Times.
Microsoft told Techworld it was aware of the reports but was not immediately able to comment.
Add your commentComments
P.M.R | Published: 02:57 GMT, 25 September 2009
#1.) Windows is a machine for play, if you can afford the large software cost and need for a massive quantity of hardware resources. #2.) Windows IS bloatware! Debug Descriptors and other debug applications/software, layers upon layers upon layers of kernels and files... #3.) Owned and Operated by greedy, money grubbing, moral lacking, and arrogant SUITS whom only speak about how to raise their stocks and "What vacation is the most talked about and expensive this year?"
Michael | Published: 01:08 GMT, 18 September 2009
There is a reason NASA, Norad, and other such do not run on Microsoft! I make my living on Microsoft servers, but for any truly mission critical system I turn back to UNIX, VAX, or Mainframe. Microsoft is simply too buggy to trust with peoples lives. Old Engineer -- no doubt! Any idiot who trust to a manual process like "have people reboot every 30 days or the critical system will go down" needs to be fired. That is not engineering, that is pulling one out the rear end.
Jeremy | Published: 21:57 GMT, 24 August 2009
I thought in Microsofts Terms of service they give us when we install thier Operating systems says that thier software is not to be used for life critical systems including Air Traffic Control, so why is California using it?
Chris | Published: 06:35 GMT, 10 March 2009
For a Linux Systems Engineer, the reboot of systems to make it more stable, does not make sense. Our Linux servers' uptime are typically from months to years. If no hardware failure, hardware upgrade and no power failure, I cannot see why a server should ever be shut ever. And when you have hardware redundancy, even upgrading of hardware can not render a service useless. Also, after the installation of new software, no need to reboot - the system just carries on and on without interruption. As to the security side, there are more or less 60 viruses for Linux versus millions for Windows. Costwise? Sql, web server, open office etc. etc. are all included in Linux. Previously, we had servers from Novell - 8 in total, to serve 4000 employees very well. Now management decided to go for Microsoft servers - the last count was 168 servers and growing. Microsoft dictates on the amount of servers. So be very carefull when going for Microsoft - it can cost you an arm and a leg.
Northshore Process Service Central Office | Published: 14:07 GMT, 02 November 2008
Northshore Process Service HQ 1560 Sherman Ave., Ste. 301 Evanston, IL 60201 U.S.A. Tel: 847.373.8972 Fax: 866.554.2485 Email: info_nps@lawyer.com
Old Engineer | Published: 14:43 GMT, 18 July 2008
if its designed to shut down at 49.7 days, manually restarting at 30 is ignorant. If the restart cycle was set to 24 days for example, there would be at least TWO chances of avoiding the 49 days. Of course there must be a hundred ways of having the system scream out "Hey guys, I haven't been rebooted!".
Hugh | Published: 10:59 GMT, 11 July 2008
Diego said: "I've worked on both unix and windows and both have their place. Unix is only a super version of DOS ...". 1) Mate, if you think that UNIX is just a "super version of DOS", I can only conclude that the only "work" you have done with it is to have dusted the server. 2)You are quite correct in saying that both UNIX and windows have their place. The difference is that UNIX's place is not in the toilet.
Criminal | Published: 02:32 GMT, 11 July 2008
For a system such as this the contract should have included huge penalties for failures like this. Then if those companies had any sense they wouldn't risk it so long as the contract prohibited insurance on such failures. This would hopefully force them to use more stable off the shelf X86 solution like GNU/Linux, BSD, Solaris, or something else.
Ray | Published: 18:52 GMT, 01 July 2008
FAA need to move to Linux system
JAKUB SZYPULKA (real one) | Published: 13:36 GMT, 07 June 2008
I'D LIKE TO SAY THAT THE PREVIOUS COMMENTS HAVE NOT BEEN WRITTEN BY ME AND SHOULD NOT BE CONNECTED WITH MY PERSON. THANK YOU.