A little too late…

In the course of writing my report I discovered that I had not thoroughly tested all of the options for PHP.  At the time what I had tested was considered sufficient.  With the benefit of hindsight I now consider that view incorrect.  That point aside there appears to be a lot of contention as to whether ISAPI or FastCGI is in fact the better way to run PHP.  There was even suggestion that it may vary depending on what a script is doing.

Another thing I never tested was how JSP runs under other competing products.  This is not so much of a problem since Tomcat is the primary environment under which it is run.

Despite being too late to gather significant data under any other environment it would be good to finish the project work by testing how PHP and JSP perform under each installation type.  The reason for doing this last is that it requires an alteration to the server state which could in turn affect the results.

A successful day

Yesterday morning I noted on the blog what pages I had tested and those that I had yet to test.  I am pleased to say that I gathered test data for each of the remaining items yesterday.  This mitigates my need to spend time programming and testing unless further tests are designed.

Since I don’t have any further work that requires the lab immediately I am going to spend today focused on the report.

Wrapping Up…

So far in my project I have gathered data for the following:

• The Categories page with MySQL and SQL Server
• The Products page with MySQL
• The Purchase page with MySQL and SQL Server
• An arbitrary page which makes heavy use of an array
• Web server performance with a HTML only request

The data that I have gathered so far has given me a fairly good idea of the performance implications of each language.  To complete the data gathering for my project I would like to run the following:

• The Products page with SQL Server
• The Item page (individual product view) with either SQL Server or MySQL
• The Categories page with a query string against MySQL and SQL Server

Once I have gathered this data the testing phase of my project should be complete.  With this data in hand I will be able to change focus to report writing and work on completing the project.

Remaining Tasks

I have collected a reasonable amount of data thus far in my project.  I intend to collect a little more before I change focus to the report.

One particular task that I need to design and run is a page which uses classes and objects to do something.  I have decided to combine this with my individual product page in the interest of saving time.

Otherwise I want to run the categories pages with an incoming query string.  Preferably I would run a more complex page but I don’t have one and am rapidly running out of available time for prototype building and testing.

Products Page with MySQL

I realised that I had not actually tested the products page with MySQL because the first time I ran the tests the timer was not functioning correctly.  So I spent some time today running these tests.

The outcomes were very close and slightly inexplicable.  It would seem that PHP works better when you are doing a cross table query.  Why on earth this would be the case I have no idea.  Anyway PHP and ASP.NET are pretty close while JSP as usual had a slightly better time.

On the basis of these results alone I wouldn’t be able to make any recommendation except that it appears that JSP makes better use of the CPU.

The Answer?

I have spent the morning trying to solve yesterdays conundrum as to why PHP does better on the products page.  The answer appears to lie in the Prepare() method called in ASP.NET and a similar method called in JSP.  Although Microsoft says that the Query method (in their PHP interface) “prepares” the statement it doesn’t seem to be calling a server side prepare.

I theorise that the server side preparation of the statement is taking longer because it is a comparatively complex query.  In addition it does not appear that PHP is in fact calling the server side prepare.  If this is the case the test pages are not identical and the test is invalid.

The solution appears to be in removing the call to the prepare statement.  Without this statement the ASP.NET and PHP pages run in pretty well identical times.  Go one step further and use simple dynamic SQL and you find that ASP.NET runs better than PHP as is normally expected.

For the moment I will ignore this page and attempt to focus my energy on pages where I will get useful data.  I am now going to run tests on the categories page but with an input query string.

Product Page Problems

Despite a successful mornings work my afternoon did not go to plan.  I began to run tests against the products page to list products by category.  The results of this test leaned in the favour of PHP although I didn’t run them with more than one client.

This is slightly disturbing as I can see no good reason why PHP is outperforming its competitors all of a sudden.  The one small possibility that I see is that it is only doing one trip to the database server rather than two which is normally done with a prepare operation.

Another thing I noticed was that in this particular test series the minimum value never changed.  This was easily fixed once I found that a hardcoded value was interfering with this particular return value.

In attempting to understand why PHP was suddenly doing well I also ran the MySQL based variant of the page.  What I noticed is that it was a lot faster.  At first I concluded that it is due to MySQL using foreign key indexes but on closer inspection I found that SQL Server is also using an index on that particular foreign key.

Overall I am left at a loss as to why PHP would do better.  One theory is that it performs better when the inputs change rather than when the page returned is the same each time – this could indicate that the other two are doing query caching.  This theory is I think invalidated by the fact that the Purchase page is faster in ASP.NET and JSP.

I seem to also be running up against issues of database performance and tuning and will have to determine where to draw the line.

An Interesting Observation

When running the purchase page with SQL Server I noticed that disk usage was actually slightly lower for JSP and ASP.NET with eight clients rather than four.  This didn’t make sense until I realised that they are performing worse with eight than four.  Clearly somewhere just past four clients the server reaches its maximum ability to handle incoming requests.

Performance under DOS

DOS in this case stands for Denial of Service (not Disk Operating System).  While I haven’t pushed my prototypes to the point of failure I have run them well beyond the server CPU ability.

By using two threads on each of my 8 test clients I was able to simulate up to a load of 16 simultaneous requests.  What I found was that the performance pretty well stabilised at 8.  At this point the requests per second value is at its optimum and any additional requests do not alter the value (except maybe to drop it ever so slightly).

JSP and ASP.NET performed respectably giving me no trouble at all.  PHP on the other hand began to drop some requests (only one in a couple of thousand).  I would like to know why PHP didn’t work properly but I can’t see any way of finding out.

Anyway it seems that all three of the languages I am testing are fairly good at balancing the load and giving fair performance to all incoming requests.

Progress This Week

This week my primary target has been the testing of the three languages against SQL Server.  So far I have only run the categories page and the purchase page.  The results that I have obtained are interesting and fairly conclusive.

As you might expect ASP.NET works much better with SQL Server (another Microsoft product) than with MySQL.  In fact the difference is enough that ASP.NET has a slight performance edge when using this DBMS.

JSP is still ticking away as per usual.  The good thing about JSP is that you pretty much don’t have to change anything in order to change DBMS.  This is a time save for me as I don’t have to recheck my code.  For the categories page JSP was pretty much the same as ASP.NET.  For the purchase page ASP.NET gained a small edge but not much.

PHP has again failed to keep up.  In real terms its performance is pretty good but it is simply not as good as its competitors.  PHP doesn’t seem to handle a heavy load well and I often find myself waiting much longer for the test cycles to complete.