Software performance testing

An integral part of my research would be the testing of the prototypes that I constructed.  Before carrying out this testing it would be important for me to gain an idea of what should characterise the tests in order to render their results as valid.  No doubt there is much information already published on the subject of software testing.

I discovered one particularly useful paper published by the University of Helsinki.  This paper discusses two methods of performance testing software applications.  The APA reference for the paper is as below…

Gan, X. (2006, September 26). Department of Computer Science. Retrieved May 29, 2009, from University of Helsinki: http://www.cs.helsinki.fi/u/paakki/gan.pdf

This paper discusses testing of fully functional business applications but the same principles should apply in the testing of cut down prototypes.  If nothing else there are some useful quotes that I can use in my proposal.  I’m not 100% sure of the accuracy of the reference because the main site is written in Finnish.

Gathering metrics

While it’s obvious that I will need to test my prototypes I am not sure exactly how I would do it.  There are existing services and programs capable of testing websites.  If I were to use these though I would have to have to first be certain of their reliability and accuracy.  Once such test service is found at www.websiteoptimization.com/services/analyze/.

While I could potentially use such a product or service I would be more certain of my results using a tool of my own making.  So long as my prototypes emit only html over http I could easily construct a simple timer program which issued requests and timed their latency.

However I produce my results I would need to perform quantitative analysis of the data to determine their accuracy.  The little I remember of statistics indicates to me that I would probably have to have hundreds if not thousands of samples to construct a valid average.

How would I go about the research?

As I have already specified the primary focus would have to be in building and testing multiple prototype applications.  I would need to produce one prototype for each of the web languages I intend to test.

While prototyping is my method in general there are many important considerations.  As Frank kindly pointed out performance is not just the statistics of how fast a site is producing html.  Equally important in performance is user experience.  Little things such as a site not having a loading bar may make the users feel they are waiting forever.

My target in carrying out the research would be the technical side of things rather than user interface.  To this end the prototypes would be constructed using only basic html such as <div> and <table>.  To produce a fair result all the prototypes would need to produce almost exactly the same output.

My Question and Title

My Question: Which dynamic server side language is most suited to database intensive applications?

My Title:  Database intensive web application development: which language to choose.

I would specify several languages. Targets would probably include ASP.NET, Ruby on Rails, PHP and JSP (or equivalent).

Initially the research would involve specifying the target. There are many versions and variants of each of these languages and I would have to identify which version I would investigate. Additionally there are other languages such as Cold Fusion which are not commonly used. Investigation into these may determine that they are worth testing.

The primary part of the research would be prototyping and performance testing. This would need to be highly controlled to ensure validity of results. Unfortunately there may not be common database platform to test against. In this case the DBMS performance would also have to be considered.

Anyway I have a pretty clear idea I want to test.

Area, Topic and Question

My area of interest is dynamic websites. I am interested in the various technologies and languages that can be used both on the server and inside the clients browser.

The topic I am going to focus on is the performance of various server side languages. There are a variety of commercially used and competitive languages. Each of these languages have beneficial features.

The specific question I want to ask is “which server side language is best”. Unfortunately it is not possible to answer this question, there can be no best language for all scenarios. Perhaps I need to be asking this question for a specific scenario. To answer the question I would have to test each of my chosen languages under a variety of different loads. The reason I want the answer is because there is so much conflicting information. Each language claims it it best, it would be good to know for myself.

Can anyone think of a better way of asking this question? I know it isn’t suitable yet.