Product Page Problems

Despite a successful mornings work my afternoon did not go to plan.  I began to run tests against the products page to list products by category.  The results of this test leaned in the favour of PHP although I didn’t run them with more than one client.

This is slightly disturbing as I can see no good reason why PHP is outperforming its competitors all of a sudden.  The one small possibility that I see is that it is only doing one trip to the database server rather than two which is normally done with a prepare operation.

Another thing I noticed was that in this particular test series the minimum value never changed.  This was easily fixed once I found that a hardcoded value was interfering with this particular return value.

In attempting to understand why PHP was suddenly doing well I also ran the MySQL based variant of the page.  What I noticed is that it was a lot faster.  At first I concluded that it is due to MySQL using foreign key indexes but on closer inspection I found that SQL Server is also using an index on that particular foreign key.

Overall I am left at a loss as to why PHP would do better.  One theory is that it performs better when the inputs change rather than when the page returned is the same each time – this could indicate that the other two are doing query caching.  This theory is I think invalidated by the fact that the Purchase page is faster in ASP.NET and JSP.

I seem to also be running up against issues of database performance and tuning and will have to determine where to draw the line.

An Interesting Observation

When running the purchase page with SQL Server I noticed that disk usage was actually slightly lower for JSP and ASP.NET with eight clients rather than four.  This didn’t make sense until I realised that they are performing worse with eight than four.  Clearly somewhere just past four clients the server reaches its maximum ability to handle incoming requests.

Performance under DOS

DOS in this case stands for Denial of Service (not Disk Operating System).  While I haven’t pushed my prototypes to the point of failure I have run them well beyond the server CPU ability.

By using two threads on each of my 8 test clients I was able to simulate up to a load of 16 simultaneous requests.  What I found was that the performance pretty well stabilised at 8.  At this point the requests per second value is at its optimum and any additional requests do not alter the value (except maybe to drop it ever so slightly).

JSP and ASP.NET performed respectably giving me no trouble at all.  PHP on the other hand began to drop some requests (only one in a couple of thousand).  I would like to know why PHP didn’t work properly but I can’t see any way of finding out.

Anyway it seems that all three of the languages I am testing are fairly good at balancing the load and giving fair performance to all incoming requests.

Progress This Week

This week my primary target has been the testing of the three languages against SQL Server.  So far I have only run the categories page and the purchase page.  The results that I have obtained are interesting and fairly conclusive.

As you might expect ASP.NET works much better with SQL Server (another Microsoft product) than with MySQL.  In fact the difference is enough that ASP.NET has a slight performance edge when using this DBMS.

JSP is still ticking away as per usual.  The good thing about JSP is that you pretty much don’t have to change anything in order to change DBMS.  This is a time save for me as I don’t have to recheck my code.  For the categories page JSP was pretty much the same as ASP.NET.  For the purchase page ASP.NET gained a small edge but not much.

PHP has again failed to keep up.  In real terms its performance is pretty good but it is simply not as good as its competitors.  PHP doesn’t seem to handle a heavy load well and I often find myself waiting much longer for the test cycles to complete.

Things to do

First up on my list of things to do is to begin working with SQL Server and get some data on the comparative performance of JSP, PHP and ASP.NET against this DBMS.  Also valuable would be a comparison between this data and the performance data with MySQL.

One thing I do want to run is a test where the data is handled in a object oriented fashion.  Probably I will use the categories page but instead of emitting data directly in the reader loop I will use one or two other classes for the retrieval.  In my mind I see myself using clsCategories and clsCategory.  You would use clsCategories.getCategories() to retrieve a list of all the product categories as objects.

To improve my understanding of the impact of increasing the number of clients I would like to re-run the categories page tests but increase the clients one at a time to produce data that I can actually graph.

Methodology Progress

Having spent much of my project time performing tests I now have my methodology fairly sure in hand.  Basically I run tests, check that the data is consistent and then move on to the next test.  I always run tests in batches to ensure other factors are not having an effect on my results.

Because my methods are now far more refined I am gathering more complete data sets and am able to be more certain of my results.  Since this is the case I may have to re-run some of the tests that I ran early on to gather more complete, useful data.

Week plan

My week plan is basically to move on and get test results from the SQL Server powered pages.  What time isn’t spent doing this is intended to be spent writing my report.  Having spent a little time reviewing statistics I think that I might also go back and get a more complete data set to see what actually happens if the number of clients is upped one at a time.

One remaining issue from my previous batch of tests was that the JSP performance data for the product list page wasn’t consistent with common sense.  It seemed like some of the test clients perhaps didn’t really have a connection.

Data Attachment Glitch

Upon inspecting the data in the table at the end of the day I noticed that all of the data was the same.  This should not affect my results in any way since the data is contrived but I would prefer to better simulate reality where possible.

The glitch is caused by the fact that I designed two subsystems of my test application separately without taking the other into account.  To fix it will be as simple as altering the counter behaviour.  I will need to be careful however not to influence the timer behaviour at all.

Sick Day…

As noted in the post I just finished writing I spent the day gathering information on the Purchase page.  Anyone with any sense would immediately question why it took so long to carry it out.  The first reason is simply that this page takes longer to run.  The second is that I am sick and made more mistakes than usual.  Finally because I was sick I let it run longer than I might otherwise have done so that I could have a more restful day.

Despite being sick though I have achieved some important results.  I will continue to work as hard as I can over the coming week though I will be a bit limited till I shake my cold.

The Purchase Page

Today’s work was primarily comprised of setting up the lab and running the purchase page.  The outcomes were pretty much as expected.  JSP held an easy lead while ASP.NET and PHP were very close to one another.  I was actually a little surprised at the tiny margin between ASP.NET and PHP – it might require a little bit of code reading to ensure everything is in order.

The obvious outcome that my project seems to be leading towards is a statement to the effect that in a high load environment like what I am simulating I would expect JSP to be the best option.