Yahoo announces SOX compliance coming for Hadoop

Hadoop, the open source distributed computing platform for handling large volumes of data, is making its way toward enterprise-readiness but still needs capabilities for SOX (Sarbanes-Oxley) compliance, a Yahoo official said Tuesday at the Hadoop Summit 2010 conference.

Officials from Yahoo, which has driven development of Hadoop, lauded the platform during keynote presentations at the Santa Clara event. Hadoop's use at companies such as Microsoft, Netflix, Twitter and the New York Times was championed by Yahoo, which itself uses Hadoop for data analysis and to deliver user-customised content. Hadoop also filters out spam at Yahoo.

"We believe that Hadoop is now ready for mainstream enterprise use," said Blake Irving, chief product officer at Yahoo and a former Microsoft vice president. Afterward, however, Irving retreated a bit, saying Hadoop still needs Sarbanes-Oxley capabilities for it to be suitable for enterprises.

"Well, it's going to be enterprise-ready," Irving said. "One of the things that's super-important about enterprise-ready Hadoop is SOX compliance. So we're in the process of working on SOX compliance for Hadoop," to provide an auditable system for enterprises, said Irving. He did not have an estimated date for when Hadoop would offer SOX compliance. Hadoop is an Apache Software Foundation project driven by Yahoo.

While there are some enterprise use cases for Hadoop already, it still requires a level of sophistication to use it,  said Eric Baldeschwieler, vice president of Hadoop software at Yahoo.

Irving noted the release Tuesday of Kerberos security capabilities for Hadoop. Workflow management also was unveiled for platform. The two improvements were characterised as beta releases by Baldeschwieler. The next major release of Hadoop will feature federation capabilities for simplifying management of large-scale clusters,  Baldeschwieler said.

One advocate for Hadoop at Tuesday's event, Java founder James Gosling, acknowledged Hadoop's Java roots. "It's totally Java and it uses the fact that code can be very mobile," Gosling said.

"[Hadoop is] a pretty big deal. People have been using it successfully all over the place in a wide range of fields," he said.

Also at the event Tuesday, Yahoo cited a planned release of an HTML5-based mobile mail product for the Apple iPhone 4, which would support capabilities for multimedia. The product is due soon, according to Yahoo.