Capacity planning

Two news items caught my attention this week. One of them was a report of a court case in America where, if I understood correctly, internet providers were being asked to keep a copy of every email and web page that passed through their systems. This put me to thinking about what that would mean in storage requirements. You see, testing a program does not just mean checking that the right text, icons and graphics are displayed, and clicking an icon performs the required action. Non-functional testing is looking at reliability, speed, and capacity.

So I started to think what is the storage requirement if an internet provider held a copy of all their traffic.

For one small provider with just two 64Mbit links, assuming that those links are mainly for business users (that is, traffic mostly between 8am and 6pm) with the links at just 75% of their capacity, then in one week the total traffic would be 64 * 0.75 * 3600 seconds in an hour * 10 hours * 5 days in Megabits. That comes to 8640000 Megabits (864 Gigabytes) per week. Of course, if the provider has a few hundred or thousand individual subscribers, each downloading emails, software and music files, and browsing the web looking for maps, video clips, reading and contributing to blogs, and so on every night then that figure could easily rise to double or triple that original estimate of 864Gb per week. Taken over the course of a year it approaches something of the order of 100 TerraBytes required for the storage.

If that is the figure for one small internet provider, just stop to think what the storage requirements will be for some of the larger providers who count their customers in the hundreds of thousands.

As you can see, bandwidth capacity and storage are important points to consider in planning any new service, or when upgrading an existing one.