Internet now dominated by 'traffic superpowers'

Something extraordinary is happening to the Internet. According to one of the largest analyses of traffic yet undertaken, what the world calls ' the Internet' is rapidly turning into an entity that exists inside and between a tiny number of hosting superpowers.

According to Arbor Networks, which gathered the data for its 2009 Atlas Observatory Report (due to be posted online on Tuesday), the implications of this will be profound.

Most people will guess that Google is one of the new Internet superpowers - it accounts for 6 percent of all Internet traffic of every type - but how many ordinary Internet users have heard of other traffic notables such as Akamai, Limelight, BitGravity, Highwinds or Gravity?

And how many would have heard of a company called Carpathia Hosting? Its MegaUpload, MeaErotik, MegaClick and MegaVideo services have turned it into a company that now accounts for 1 percent of all Internet traffic, says Arbor, and this will doubtless grow.

The important takeaway is that few of these companies had even been heard of two years ago, and very few of them are big telcos.

To put all this into perspective, in 2007 Arbor found that the overwhelming majority of Internet traffic was accounted for by 30,000 entities, with fifty percent of traffic accounted for by around 10,000 companies.

Only two years later that same fifty percent now runs through only 150 top ‘content delivery networks' (CDNs), an astonishing consolidation made more remarkable by the fact that Internet traffic has grown significantly during that time.

"Up to 2007, The Internet meant connecting to lots of servers and data centres around the world," notes Arbor's chief scientist, Craig Labovitz. Now there are barely 100 companies that matter.

Traffic patterns tend to be hidden, mainly because the companies losing out - the traditional telcos and ISPs - don't exactly have an interest in advertising their waning status. The reason for their decline in importance is that Internet traffic is being driven by huge providers with access to content such as video.

"For 150 years, they [BT and other telcos] have had the same business model. Now everyone is trying to get away from being a dumb pipe."

Arbor's Atlas Internet Observatory report crunched traffic from 100 of the Internet's largest entities, accounting for 12 Terabytes of peak throughput, equivalent to about a quarter of the Internet's total at any one moment, said Labovitz.
The importance of this is not simply that a small number of companies will account for a lot of traffic, but that these companies are increasingly what the Internet actually is.

The Internet up to around 2007 was dominated by a hierarchy of companies, co-operating with one another to allow traffic to be passed from one to the other, regardless of size. The new Internet superpowers, in stark contrast, bypass a lot of this and use direct connections from one to the other. If a company is not part of this new core, it could find itself increasingly passed to the ‘long tail', a polite way of saying they will be shoved to the fringe.

Video, including video that runs over web/http, now accounts for an estimated 10 percent of all Internet traffic, and is one reason all these direct connections between large data centres are now necessary. IPv6 traffic remains tiny at only 0.03 percent of traffic, but is showing sudden and possibly rapid growth in recent months thanks to deployments by named hosters.

Interestingly, P2P is in rapid decline, falling from around 3 percent of all traffic in 2007 to only half a percent now. Again, downloaders appear to prefer direct connectivity for downloads, mostly through port 80 and the web.