Progress This Week

This week my primary target has been the testing of the three languages against SQL Server.  So far I have only run the categories page and the purchase page.  The results that I have obtained are interesting and fairly conclusive.

As you might expect ASP.NET works much better with SQL Server (another Microsoft product) than with MySQL.  In fact the difference is enough that ASP.NET has a slight performance edge when using this DBMS.

JSP is still ticking away as per usual.  The good thing about JSP is that you pretty much don’t have to change anything in order to change DBMS.  This is a time save for me as I don’t have to recheck my code.  For the categories page JSP was pretty much the same as ASP.NET.  For the purchase page ASP.NET gained a small edge but not much.

PHP has again failed to keep up.  In real terms its performance is pretty good but it is simply not as good as its competitors.  PHP doesn’t seem to handle a heavy load well and I often find myself waiting much longer for the test cycles to complete.

The Purchase Page

Today’s work was primarily comprised of setting up the lab and running the purchase page.  The outcomes were pretty much as expected.  JSP held an easy lead while ASP.NET and PHP were very close to one another.  I was actually a little surprised at the tiny margin between ASP.NET and PHP – it might require a little bit of code reading to ensure everything is in order.

The obvious outcome that my project seems to be leading towards is a statement to the effect that in a high load environment like what I am simulating I would expect JSP to be the best option.

Categories Page Test Results

The test results that I obtained against my product categories page were somewhat revealing.  I tested each page with 1, 4 and 8 clients running 10000 requests.  With 8 clients attached JSP was fastest taking around 98 seconds.  ASP.NET came next at 160 seconds while PHP lagged behind taking a full 238 seconds.

What this tells me is that JSP can perform exceptionally well when working with a MySQL database.  Since this in conflict with my initial results I also learn that the configuration of the JSP code can make a very big difference.

In terms of averages JSP responds once every 10ms, ASP.NET every 16ms and PHP every 24ms.  A friend in statistics class determined that the standard error of my means is so small as to be negligible.

Threading model matters

Interestingly I tried PHP under both Apache and IIS.  In IIS I was using the non-thread safe version along with FastCGI which is in itself thread safe.  In Apache I had it installed as a module and was using the thread safe version (because you can’t use the other).  The same script had a much better average response time under IIS than under Apache.

Looking at the complete results rather than just a summary showed something rather interesting.  When running in thread safe mode the results were slow but consistent.  Running NTS it was much faster… sometimes… and much slower for other requests.  In fact as NTS it was quite ragged with a high value every few requests interspersed by very fast requests.

My first real outcome

After the tests that I carried out today I finally have some results to work with.  What I found was interesting but not wholly unexpected.  ASP.NET was the clear winner with a response time more than 2ms faster than PHP and JSP.  JSP comes next followed by a very narrow margin by PHP.  The difference between PHP and JSP is less than one millisecond but my results are consistent enough to show that it is not just down to chance.

The corresponding server resource usage I found to be rather interesting.  PHP used significantly less processor but thrashed the disk.  JSP used about the same processor as ASP but went easier on the disk.  ASP just pushed hard on all resources and achieved better performance as a result.