By Sterling “Chip” Camden
Contributing Writer, [GAS]
The term “cloud computing” has entered that phase of the buzzword life-cycle in which everyone is talking about it, few people are doing anything with it, and most people’s understanding of what it means is, at best, cloudy (sorry).
Those who do understand cloud computing usually think of massively parallel servers used to scale up to meet user demand. But Randy Hayes of CapCal had the opposite idea: why not use the cloud to simulate user demand instead of satisfying it?
CapCal’s Web Performance Testing product spawns Linux instances in Amazon’s Elastic Compute Cloud (EC2) to perform load testing of a web site by simulating any number of simultaneous users. The tester first records a script of whatever use-case scenario they want to test. The script can include normal pauses for “think time” so that it truly represents how a user would interact with the site, and can be recorded from the desktop without having to edit the script at all. The user then feeds that script into CapCal and specifies how many simultaneous users to simulate. CapCal starts up the requisite number of Linux instances in the cloud (about one agent for every 100 simulated users), which gradually ramp up to pour the HTTP requests onto the specified site. Here are some more details on the architecture.
A controller process monitors the agents and gathers data in real time. This includes all sorts of performance metrics on each and every request including load time, bandwidth required, and any errors encountered. A summary report classifies the percentage of response times that fall within various categories of acceptability. The detailed data can be downloaded and fed into Excel so you can do whatever analysis you please on the numbers.
The cloud-based architecture is fairly new, but CapCal has been doing this type of load testing using a distributed network since 2000. Amazon’s EC2 allows them to grow easily to any number of agents at minimal cost.
Pricing for the service is yet to be determined, but Randy told me that he’s thinking of simply adding on to what Amazon charges for the compute time. Of course, depending on the load you select, you might have bandwidth costs of your own to consider. We also discussed the idea of subscription-based pricing for sites that would like to conduct this sort of testing as part of their regular cycle of development-QA-release.
Speaking of the development cycle, CapCal also offers a free service called Agile Performance Testing (APT), which is a scaled-down version of their Web Performance Testing product. It runs 20 user simulations from a single agent on your local machine, so (hopefully) you can’t really stretch your site to the breaking point — but you can still gather the performance data to compare what your latest change did to the overall page load experience. Agile developers could use this in conjunction with unit tests to verify that nothing gets seriously broken with each iteration. Randy showed me a pretty impressive demo of this product today. He also demonstrated starting up agents in Amazon EC2, but we didn’t do any live testing using the cloud-based agents. I guess they’re still filing off the rough edges on that product.
To launch the service onto the public stage, CapCal is looking for a site that is willing to undergo testing at the 1 million simultaneous user level. Randy says he’s done the research, and nobody has ever load-tested a site at that capacity before, so he hopes to score a line item in the Guinness Book of World Records. I don’t know about you, but after enduring a million simultaneous user test, I’d be ready for a Guinness of a different sort. Do you think your site could handle 1 million simultaneous users?
Randy has generously given [GAS] permission to give away ten free “find your limits” site tests! So the first ten people who leave a comment requesting this freebie, have their own domain, and do not run their site on a free or shared hosting platform (VPS’s and dedicated servers are fine), will get the opportunity to bring their own server to its knees. Please note that if your site includes impression-based advertising (including Google Adsense) you may want to disable the ads for the test, in order to avoid violating their Terms of Service.