CapCal: Web Performance Testing From the Lab to the Cloud

Advertisement

By Sterling “Chip” Camden
Contributing Writer, [GAS]

Web Performance Testing The term “cloud computing” has entered that phase of the buzzword life-cycle in which everyone is talking about it, few people are doing anything with it, and most people’s understanding of what it means is, at best, cloudy (sorry).

Those who do understand cloud computing usually think of massively parallel servers used to scale up to meet user demand.  But Randy Hayes of CapCal had the opposite idea:  why not use the cloud to simulate user demand instead of satisfying it?

CapCal’s Web Performance Testing product spawns Linux instances in Amazon’s Elastic Compute Cloud (EC2) to perform load testing of a web site by simulating any number of simultaneous users.  The tester first records a script of whatever use-case scenario they want to test.  The script can include normal pauses for “think time” so that it truly represents how a user would interact with the site, and can be recorded from the desktop without having to edit the script at all.  The user then feeds that script into CapCal and specifies how many simultaneous users to simulate.  CapCal starts up the requisite number of Linux instances in the cloud (about one agent for every 100 simulated users), which gradually ramp up to pour the HTTP requests onto the specified site. Here are some more details on the architecture.

A controller process monitors the agents and gathers data in real time.  This includes all sorts of performance metrics on each and every request including load time, bandwidth required, and any errors encountered.  A summary report classifies the percentage of response times that fall within various categories of acceptability.  The detailed data can be downloaded and fed into Excel so you can do whatever analysis you please on the numbers.

The cloud-based architecture is fairly new, but CapCal has been doing this type of load testing using a distributed network since 2000.  Amazon’s EC2 allows them to grow easily to any number of agents at minimal cost.

Pricing for the service is yet to be determined, but Randy told me that he’s thinking of simply adding on to what Amazon charges for the compute time.  Of course, depending on the load you select, you might have bandwidth costs of your own to consider.  We also discussed the idea of subscription-based pricing for sites that would like to conduct this sort of testing as part of their regular cycle of development-QA-release.

Speaking of the development cycle, CapCal also offers a free service called Agile Performance Testing (APT), which is a scaled-down version of their Web Performance Testing product.  It runs 20 user simulations from a single agent on your local machine, so (hopefully) you can’t really stretch your site to the breaking point — but you can still gather the performance data to compare what your latest change did to the overall page load experience.  Agile developers could use this in conjunction with unit tests to verify that nothing gets seriously broken with each iteration.  Randy showed me a pretty impressive demo of this product today.  He also demonstrated starting up agents in Amazon EC2, but we didn’t do any live testing using the cloud-based agents.  I guess they’re still filing off the rough edges on that product.

To launch the service onto the public stage, CapCal is looking for a site that is willing to undergo testing at the 1 million simultaneous user level.  Randy says he’s done the research, and nobody has ever load-tested a site at that capacity before, so he hopes to score a line item in the Guinness Book of World Records.  I don’t know about you, but after enduring a million simultaneous user test, I’d be ready for a Guinness of a different sort. Do you think your site could handle 1 million simultaneous users? If so, then please contact us and we’ll put you in touch with Randy right away!

Randy has generously given [GAS] permission to give away ten free “find your limits” site tests! So the first ten people who leave a comment requesting this freebie, have their own domain, and do not run their site on a free or shared hosting platform (VPS’s and dedicated servers are fine), will get the opportunity to bring their own server to its knees. Please note that if your site includes impression-based advertising (including Google Adsense) you may want to disable the ads for the test, in order to avoid violating their Terms of Service.





Advertisement



30 Responses to CapCal: Web Performance Testing From the Lab to the Cloud

  1. This is so useful in so many ways. It will be especially applicable for site owners who regularly see huge traffic spikes.

  2. This is so useful in so many ways. It will be especially applicable for site owners who regularly see huge traffic spikes.

  3. Me too! Me too!

    I'm still not excited about the cloud computing model. It just seems so unpredictable, as if you don't worry about issues because "someone out there will take care of it".

    • That's an interesting concern. But I think that eventually computing power will become almost completely commoditized. We trust the power company to provide us power, so why not trust a computing company to provide horsepower? Yes, even the power company fails sometimes — but that doesn't mean that we should always generate our own power.

  4. Me too! Me too!

    I’m still not excited about the cloud computing model. It just seems so unpredictable, as if you don’t worry about issues because “someone out there will take care of it”.

    • That’s an interesting concern. But I think that eventually computing power will become almost completely commoditized. We trust the power company to provide us power, so why not trust a computing company to provide horsepower? Yes, even the power company fails sometimes — but that doesn’t mean that we should always generate our own power.

  5. Welcome CapCal to this new frontier! SOASTA has run 1000's of CloudTests (average to massive) for customers since Cloud Computing emerged as a platform for us late in 2007.

    The typical use-case we support (with major websites like Hallmark, TurboTax, Proctor&Gamble, BMW, etc) is a desire to hit volume goals – e.g. 10 Million users per day, 50 million downloads in 10 hours, etc. Customers' "simultaneous" objectives as part of these tests are often in the 10s of thousands (still large by traditional standards), but large sites like to see simultaneous users in the 200-500,000 user range. The only limit now to the size of test is the capacity of the system being tested – which, of course, is why we're testing (Pints of Guinness notwithstanding!)

    The point here is that this CAN and IS begin done today, and with cloud testing, customers can now demand of their dev and ops teams that web apps be tested to full production scale. Thanks cloud computing, and good luck CapCal.

    For musings on cloud testing.. .http://www.soasta.com/blog/

  6. Welcome CapCal to this new frontier! SOASTA has run 1000's of CloudTests (average to massive) for customers since Cloud Computing emerged as a platform for us late in 2007.

    The typical use-case we support (with major websites like Hallmark, TurboTax, Proctor&Gamble, BMW, etc) is a desire to hit volume goals – e.g. 10 Million users per day, 50 million downloads in 10 hours, etc. Customers' "simultaneous" objectives as part of these tests are often in the 10s of thousands (still large by traditional standards), but large sites like to see simultaneous users in the 200-500,000 user range. The only limit now to the size of test is the capacity of the system being tested – which, of course, is why we're testing (Pints of Guinness notwithstanding!)

    The point here is that this CAN and IS begin done today, and with cloud testing, customers can now demand of their dev and ops teams that web apps be tested to full production scale. Thanks cloud computing, and good luck CapCal.

    For musings on cloud testing.. .http://www.soasta.com/blog/

  7. Hey Brad,

    Nice to meet you and thanks for the comment! 2007 was a good time to jump into the cloud, I have to admit we are a little late but we treated the Internet itself like a cloud for many years and I can tell you Amazon EC2 is MUCH easier!

    The key, of course, is simultaneous virtual users and you have to admit that a million is a lot! You know that already because of the range of numbers you are testing in – before cloud computing, 200 – 500k simultaneous users was totally unheard of. So you guys are already breaking records compared to doing things the old way.

    Anyway, I thought it might be fun to ask Geeks Are Sexy here if they want to host a competition – we know we can both spin circles around "legacy" tools like LoadRunner, but how about a million user race? It would be a neutral third party that would host the site and we would test the same pages in the same order with the same think times. How long it takes to reach 1 million simultaneous sessions from the first page hit would be the measuring stick.

    What do you think?

    In any case, thanks again for the comment and best of luck to you as well! You guys have an awesome UI, I like it!

    Randy

    • Randy,

      Thanks for the compliments and the interesting proposal.

      You’re absolutely right that we’re exceeding known load testing limits with customers all the time, whether 70M Users in 10 hours, 500,000+ concurrent users, or (the real challenge) 5.7 TB of real-time test data analyzed to pinpoint production issues in tests of this magnitude.

      For our CloudTests, there’s little difference in provisioning 5, 500 or 5000+ test servers, so pressing “Run” on 1M concurrent users isn’t the issue. What matters are the customers’ environment, the methodology used to ramp the tests meaningfully, and the analytics’ capability to find issues. We’ll refrain from this science experiment and stick to meeting our customer’s requirements. Any records will fall naturally.

      Brad

  8. Hey Brad,

    Nice to meet you and thanks for the comment! 2007 was a good time to jump into the cloud, I have to admit we are a little late but we treated the Internet itself like a cloud for many years and I can tell you Amazon EC2 is MUCH easier!

    The key, of course, is simultaneous virtual users and you have to admit that a million is a lot! You know that already because of the range of numbers you are testing in – before cloud computing, 200 – 500k simultaneous users was totally unheard of. So you guys are already breaking records compared to doing things the old way.

    Anyway, I thought it might be fun to ask Geeks Are Sexy here if they want to host a competition – we know we can both spin circles around “legacy” tools like LoadRunner, but how about a million user race? It would be a neutral third party that would host the site and we would test the same pages in the same order with the same think times. How long it takes to reach 1 million simultaneous sessions from the first page hit would be the measuring stick.

    What do you think?

    In any case, thanks again for the comment and best of luck to you as well! You guys have an awesome UI, I like it!

    Randy

    • Randy,

      Thanks for the compliments and the interesting proposal.

      You’re absolutely right that we’re exceeding known load testing limits with customers all the time, whether 70M Users in 10 hours, 500,000+ concurrent users, or (the real challenge) 5.7 TB of real-time test data analyzed to pinpoint production issues in tests of this magnitude.

      For our CloudTests, there’s little difference in provisioning 5, 500 or 5000+ test servers, so pressing “Run” on 1M concurrent users isn’t the issue. What matters are the customers’ environment, the methodology used to ramp the tests meaningfully, and the analytics’ capability to find issues. We’ll refrain from this science experiment and stick to meeting our customer’s requirements. Any records will fall naturally.

      Brad

  9. Brad,

    Sorry to hear this, the publicity would do both of us some good – you don't think HP is sitting on their hands waiting to see their LoadRunner franchise evaporate do you?

    In any case, referring to this as a "science experiment" is not fair and you know it – it's a contest between two companies who offer cloud testing to see which one can generate the most load the fastest. I suspect somebody at SOASTA read some of the material on my blog about the CapCal architecture and decided the risk was too high for SOASTA. If that's the case, I don't blame you at all – if I had a Windows load agent I would never go against someone with a Linux agent, for example. It's not a fair contest, even though the advantages are many.

    So if you want to place a comment here that says "it happens all the time even though we haven't done it" and then refuse to show us the "beef" then it probably would have been better not to post at all. It looks both arrogant and chickensh*t, which I am sure is not your intention. Net result – CapCal challenged SOASTA to a load contest and SOASTA declined.

    I love the name CloudTest, so much so that I decided to name CapCal that in 2005 and went so far as to snatch up the cloudtest.com domain. Because of a goof with an email address at the registrar, I ended up letting it elapse without knowing it and it's had two owners since then. Oh well, you don't need the domain to use it as a trademark and I like it. I also like Capacity Calibration because we do more than cloud testing. So I'm happy with where the names ended up!

    Cheers

    Randy

  10. Brad,

    Sorry to hear this, the publicity would do both of us some good – you don’t think HP is sitting on their hands waiting to see their LoadRunner franchise evaporate do you?

    In any case, referring to this as a “science experiment” is not fair and you know it – it’s a contest between two companies who offer cloud testing to see which one can generate the most load the fastest. I suspect somebody at SOASTA read some of the material on my blog about the CapCal architecture and decided the risk was too high for SOASTA. If that’s the case, I don’t blame you at all – if I had a Windows load agent I would never go against someone with a Linux agent, for example. It’s not a fair contest, even though the advantages are many.

    So if you want to place a comment here that says “it happens all the time even though we haven’t done it” and then refuse to show us the “beef” then it probably would have been better not to post at all. It looks both arrogant and chickensh*t, which I am sure is not your intention. Net result – CapCal challenged SOASTA to a load contest and SOASTA declined.

    I love the name CloudTest, so much so that I decided to name CapCal that in 2005 and went so far as to snatch up the cloudtest.com domain. Because of a goof with an email address at the registrar, I ended up letting it elapse without knowing it and it’s had two owners since then. Oh well, you don’t need the domain to use it as a trademark and I like it. I also like Capacity Calibration because we do more than cloud testing. So I’m happy with where the names ended up!

    Cheers
    Randy