Microsoft server crash nearly causes 800-plane pile-up

Failure to restart system caused data overload.

A major breakdown in Southern California's air traffic control system last week was partly due to a "design anomaly" in the way Microsoft Windows servers were integrated into the system, according to a report in the Los Angeles Times.

The radio system shutdown, which lasted more than three hours, left 800 planes in the air without contact to air traffic control, and led to at least five cases where planes came too close to one another, according to comments by the Federal Aviation Administration reported in the LA Times and The New York Times. Air traffic controllers were reduced to using personal mobile phones to pass on warnings to controllers at other facilities, and watched close calls without being able to alert pilots, according to the LA Times report.

The failure was ultimately down to a combination of human error and a design glitch in the Windows servers brought in over the past three years to replace the radio system's original Unix servers, according to the FAA.

The servers are timed to shut down after 49.7 days of use in order to prevent a data overload, a union official told the LA Times. To avoid this automatic shutdown, technicians are required to restart the system manually every 30 days. An improperly trained employee failed to reset the system, leading it to shut down without warning, the official said. Backup systems failed because of a software failure, according to a report in The New York Times.

The contract for designing the system, called Voice Switching and Control System (VSCS), was awarded to Harris Corporation in 1992 and the system was installed in the late 1990s, initially using Unix servers, according to Harris. In 2001, the company completed testing of the VSCS Control Subsystem Upgrade (VCSU), which replaced the original servers with off-the-shelf Dell hardware running Microsoft Windows 2000 Advanced Server. The upgrade was installed in California last year, according to the FAA.

Soon after installation, however, the FAA discovered that the system design could lead to a radio system shutdown, and put the maintenance procedure into place as a workaround, the LA Times said. The FAA reportedly said it has been working on a permanent fix but has only eliminated the problem in Seattle. The FAA is now planning to institute a second workaround - an alert that will warn controllers well before the software shuts down.

The shutdown is intended to keep the system from becoming overloaded with data and potentially giving controllers wrong information about flights, according to a software analyst cited by the LA Times.

Microsoft told Techworld it was aware of the reports but was not immediately able to comment.



What are your views on this subject? Use the form below to post a comment on this article up to 500 characters.


Characters remaining: 500

Add your commentComments

P.M.R | Published: 02:57 GMT, 25 September 2009

#1.) Windows is a machine for play, if you can afford the large software cost and need for a massive quantity of hardware resources. #2.) Windows IS bloatware! Debug Descriptors and other debug applications/software, layers upon layers upon layers of kernels and files... #3.) Owned and Operated by greedy, money grubbing, moral lacking, and arrogant SUITS whom only speak about how to raise their stocks and "What vacation is the most talked about and expensive this year?"

Michael | Published: 01:08 GMT, 18 September 2009

There is a reason NASA, Norad, and other such do not run on Microsoft! I make my living on Microsoft servers, but for any truly mission critical system I turn back to UNIX, VAX, or Mainframe. Microsoft is simply too buggy to trust with peoples lives. Old Engineer -- no doubt! Any idiot who trust to a manual process like "have people reboot every 30 days or the critical system will go down" needs to be fired. That is not engineering, that is pulling one out the rear end.

Jeremy | Published: 21:57 GMT, 24 August 2009

I thought in Microsofts Terms of service they give us when we install thier Operating systems says that thier software is not to be used for life critical systems including Air Traffic Control, so why is California using it?

Chris | Published: 06:35 GMT, 10 March 2009

For a Linux Systems Engineer, the reboot of systems to make it more stable, does not make sense. Our Linux servers' uptime are typically from months to years. If no hardware failure, hardware upgrade and no power failure, I cannot see why a server should ever be shut ever. And when you have hardware redundancy, even upgrading of hardware can not render a service useless. Also, after the installation of new software, no need to reboot - the system just carries on and on without interruption. As to the security side, there are more or less 60 viruses for Linux versus millions for Windows. Costwise? Sql, web server, open office etc. etc. are all included in Linux. Previously, we had servers from Novell - 8 in total, to serve 4000 employees very well. Now management decided to go for Microsoft servers - the last count was 168 servers and growing. Microsoft dictates on the amount of servers. So be very carefull when going for Microsoft - it can cost you an arm and a leg.

Northshore Process Service Central Office | Published: 14:07 GMT, 02 November 2008

Northshore Process Service HQ 1560 Sherman Ave., Ste. 301 Evanston, IL 60201 U.S.A. Tel: 847.373.8972 Fax: 866.554.2485 Email: info_nps@lawyer.com

Old Engineer | Published: 14:43 GMT, 18 July 2008

if its designed to shut down at 49.7 days, manually restarting at 30 is ignorant. If the restart cycle was set to 24 days for example, there would be at least TWO chances of avoiding the 49 days. Of course there must be a hundred ways of having the system scream out "Hey guys, I haven't been rebooted!".

Hugh | Published: 10:59 GMT, 11 July 2008

Diego said: "I've worked on both unix and windows and both have their place. Unix is only a super version of DOS ...". 1) Mate, if you think that UNIX is just a "super version of DOS", I can only conclude that the only "work" you have done with it is to have dusted the server. 2)You are quite correct in saying that both UNIX and windows have their place. The difference is that UNIX's place is not in the toilet.

Criminal | Published: 02:32 GMT, 11 July 2008

For a system such as this the contract should have included huge penalties for failures like this. Then if those companies had any sense they wouldn't risk it so long as the contract prohibited insurance on such failures. This would hopefully force them to use more stable off the shelf X86 solution like GNU/Linux, BSD, Solaris, or something else.

Ray | Published: 18:52 GMT, 01 July 2008

FAA need to move to Linux system

JAKUB SZYPULKA (real one) | Published: 13:36 GMT, 07 June 2008

I'D LIKE TO SAY THAT THE PREVIOUS COMMENTS HAVE NOT BEEN WRITTEN BY ME AND SHOULD NOT BE CONNECTED WITH MY PERSON. THANK YOU.

Related Operating Systems news

Microsoft, Linux rivals mock Google Chrome OS

Operating system crippled by reliance on web access

Google releases Chrome OS to waiting world

Stresses speed and security of operating system.

NSA helped with work on Windows 7 security

Privacy organisations concerned about spooks' involvement.

Microsoft previews Office 2010 beta to developers

But pirated copies of the software are already leaked online

Related Operating Systems reviews

Jolicloud OS

Moblin 2.0 review

Ubuntu Netbook Remix



Email this article to a friend or colleague:


PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

Techworld White Papers

Database security: Preventing enterprise data leaks at the source

IDC discusses the growing internal threats to business information, the impact of government regulations on the protection of data, and how enterprises must adopt database security best practices...

Download Whitepaper

Service-oriented security

SOA has become an integral part of enterprise software by providing a framework to efficiently develop software as services that is easily sharable, reusable, and integrated. No where is the need more apparent than in the Identity Management space. Welcome to the age of Service-Oriented Security (SOS).

Download Whitepaper

Data protection prospective vendor checklist

Organisations need a way to map business needs against all these challenges in procuring a technical solution. To help, SANS has developed the following Prospective Vendor Checklist.

Download Whitepaper

Unlock the power of the mainframe

This whitepaper presents the notion of CICS as an integration hub based on a component-based, service-oriented architecture supporting Web services. Highlights will review the challenges and contrasted support for Web services natively in CICS.

Download Whitepaper

Techworld UK - Technology - Business

COLT White Paper

Are all VoIP services the same?

Questions to ask your service provider to ensure you get the VoIP service you need
With careful choice of partner, your business can have all the advantages of VoIP access - reduced costs, flexibility and simplicity - without the drawbacks.
This white paper is your guide to ensure you get right the VoIP service and details the pitfalls which businesses would do well to avoid.

Download white paper
BMC

Ride the express lane in the journey to speed ITIL adoption

Explore the challenges in making the journey to ITIL and the criteria for selecting consulting services
By following ITIL practices, your IT organisation will become more closely integrated with the business. We recommend making the journey to ITIL in a sequence of six incremental steps, the phases of which are driven through execution of a strategic transformational roadmap.

Download white paper

Webcast: IT Financial Management: Cost Optimisation for Efficiency and Agility.
On Demand Webcast
Join this webcast to learn about the techniques and technologies that can help you prove the value of IT to the business by understanding the true cost of today's IT services and those that will be necessary to deliver future success.

Register Today

Site Map

IDG Network

* *