Internet crashes worldwide as routers run out of memory

Ageing internet hardware struggling to keep up with internet growth, causing a series of crashes - with worse to come

The aging infrastructure underlying much of the internet was exposed this week when many commonly used routers ran out of memory space to store their routing tables, causing a spike in outages leaving users unable to access websites.

Hardware affected include the Cisco 6500- and 7600-series of routers and switches. The problem seems largely to have affected the US, where telecoms giant Verizon pushed an extra 15,000 routing destinations to its routers - only to tip some of them over the default 524,288 limit causing multiple hardware crashes on aging routing hardware.

As a result, users across the world were unable to access websites as the number of outages globally spiked from around 1,500 to 2,587, according to network engineer Andree Toonk, who tracks such problems. Verizon had to withdraw the routing information in response, although Ukraine was most affected, according to BGPmon.net.

It follows a similar hiatus when the same hardware was pushed over the 256,000 address mark. Telecoms companies had to either upgrade their hardware or replace it in response. Network engineers predict more outages as companies tip their routing hardware over the limit.

"The problem is real, and we still haven't seen the full effects, because most of the internet hasn't yet experienced the conditions that could cause problems for under-provisioned equipment," wrote Jim Cowie, chief scientist at Renesys, on a corporate blog.

Cowie believes that the problem will get worse before it gets better: "The real test, when large providers commonly believe that the internet contains 512K routes, and pass that along to all their customers as a consensus representation of internet structure, will start later this week, and will be felt nearly everywhere by the end of next week."

He continued: "Enterprises that rely on the internet for delivery of service should pay close attention to the latency and reachability of the paths to customers in the coming weeks, in order to identify affected service providers upstream and work around them while they perform appropriate upgrades to their infrastructure."

Cisco has issued a workaround for the problem, which it will be advising customers to roll out before introducing new routing information to the hardware.

Cisco's advisory explains the problem in plain language: "In March, 2014, the Classless Inter-Domain Routing (CIDR) Report, which provides statistics on the global Internet routing table, reported that the global internet routing table had passed 500,000 routes for the first time in internet history.

"Most platforms have more than enough space to support larger routing tables, but the default configurations might require adjustment. As the internet routing table approaches 512,000 routes, it can cause the Catalyst 6500 and 7600 3BXL-based modules to exceed the default routing TCAM allocations."