Software performance testing

An integral part of my research would be the testing of the prototypes that I constructed.  Before carrying out this testing it would be important for me to gain an idea of what should characterise the tests in order to render their results as valid.  No doubt there is much information already published on the subject of software testing.

I discovered one particularly useful paper published by the University of Helsinki.  This paper discusses two methods of performance testing software applications.  The APA reference for the paper is as below…

Gan, X. (2006, September 26). Department of Computer Science. Retrieved May 29, 2009, from University of Helsinki: http://www.cs.helsinki.fi/u/paakki/gan.pdf

This paper discusses testing of fully functional business applications but the same principles should apply in the testing of cut down prototypes.  If nothing else there are some useful quotes that I can use in my proposal.  I’m not 100% sure of the accuracy of the reference because the main site is written in Finnish.

Gathering metrics

While it’s obvious that I will need to test my prototypes I am not sure exactly how I would do it.  There are existing services and programs capable of testing websites.  If I were to use these though I would have to have to first be certain of their reliability and accuracy.  Once such test service is found at www.websiteoptimization.com/services/analyze/.

While I could potentially use such a product or service I would be more certain of my results using a tool of my own making.  So long as my prototypes emit only html over http I could easily construct a simple timer program which issued requests and timed their latency.

However I produce my results I would need to perform quantitative analysis of the data to determine their accuracy.  The little I remember of statistics indicates to me that I would probably have to have hundreds if not thousands of samples to construct a valid average.

How would I go about the research?

As I have already specified the primary focus would have to be in building and testing multiple prototype applications.  I would need to produce one prototype for each of the web languages I intend to test.

While prototyping is my method in general there are many important considerations.  As Frank kindly pointed out performance is not just the statistics of how fast a site is producing html.  Equally important in performance is user experience.  Little things such as a site not having a loading bar may make the users feel they are waiting forever.

My target in carrying out the research would be the technical side of things rather than user interface.  To this end the prototypes would be constructed using only basic html such as <div> and <table>.  To produce a fair result all the prototypes would need to produce almost exactly the same output.

My Question and Title

My Question: Which dynamic server side language is most suited to database intensive applications?

My Title:  Database intensive web application development: which language to choose.

I would specify several languages. Targets would probably include ASP.NET, Ruby on Rails, PHP and JSP (or equivalent).

Initially the research would involve specifying the target. There are many versions and variants of each of these languages and I would have to identify which version I would investigate. Additionally there are other languages such as Cold Fusion which are not commonly used. Investigation into these may determine that they are worth testing.

The primary part of the research would be prototyping and performance testing. This would need to be highly controlled to ensure validity of results. Unfortunately there may not be common database platform to test against. In this case the DBMS performance would also have to be considered.

Anyway I have a pretty clear idea I want to test.

Area, Topic and Question

My area of interest is dynamic websites. I am interested in the various technologies and languages that can be used both on the server and inside the clients browser.

The topic I am going to focus on is the performance of various server side languages. There are a variety of commercially used and competitive languages. Each of these languages have beneficial features.

The specific question I want to ask is “which server side language is best”. Unfortunately it is not possible to answer this question, there can be no best language for all scenarios. Perhaps I need to be asking this question for a specific scenario. To answer the question I would have to test each of my chosen languages under a variety of different loads. The reason I want the answer is because there is so much conflicting information. Each language claims it it best, it would be good to know for myself.

Can anyone think of a better way of asking this question? I know it isn’t suitable yet.

Web browser as an application platform

http://research.sun.com/technical-reports/2008/smli_tr-2008-175.pdf

http://research.sun.com/techrep/2007/smli_tr-2007-168.pdf

Class Notes – 03-04-09

Research Approach

- how will you do you research?
- what do you want to know?
- what is your purpose?
- what is your research question?

choosing the most appropriate approach is very important to the success of your project

Approaches…

- Discourse analysis – analysing what people have said
- Surveys and questionnaires
– Answers: open ended, yes/no, multichoice, scaled
- Combining and/or comparing data and findings
- Meta-analysis – combine the results of statistical studies
- Case study – looking at and analysing and existing case
- Action research – being part of the research while still observing the result
- Develop something and test it

Reading abstracts

What was the purpose of the research

They wanted to develop a prototype to combine various webservices into a useful form for the user in a way that was easy for the user to understand

What did the researcher do?

Developed a prototyle to dynamically compose webservices and then tested the prototype in some simple scenarios

Source:

Sirin, E., Hendler, J., & Parsia, B. (2003). Semi-automatic Composition of Web Services. College Park, Maryland, USA.

Reading an academic paper

Who is the paper written by and do they seem credible to you?

The article is written by someone called Robert Steele. They seem credible to me because they are published by the University of Technology, Sydney.

How do you know it’s an academic article

It has the structure of an academic article, is worded like an academic article and is published by a University

What is the title and what does the title tell you?

The title is “A Web Services-based System for Ad-hoc Mobile Application Integration”. This title tells me two things – The article will be talking about mobile devices. The article will be talking about how web-services can be used on such devices.

Does the abstract tell you what it should?

Yes, the abstract does give much information about the content to follow. I think that it could perhaps have been written a little more simply. The article is about integrating cross platform device resident software with webservices on temporary network connections. The article will not talk about the what the authors did/found because it is presenting a proposal.

How many references do they have / are there any that look useful

There are 12 references provides but I don’t think any of them are particularly useful to me. They all seem to point to technical specifications of various standards discussed in the paper.

Where do you decide to stop and why?

I read all of the information in the paper. This was for two reasons: 1 becasuse the paper was relatively short and 2 because I wanted to find out exactly what it was that they were proposing which I couldn’t tell from just a light read.

What did I learn from reading the paper?

In short not very much. I learned a little about some of the technologies surrounding webservices such as those used for their discovery. This paper was mostly a listing of facts I was already aware of in support of their proposal.

Is there useful knowledge in the paper?

Since the paper is written as a proposal it doesn’t present a whole lot of information useful to me. It did however list and improve my awareness of some of the issues surrounding mobile application development.

Would you read another paper by these authors?

Probably not, it didn’t present much useful information and while they made a reasonable case for their proposal it wasn’t breaking any significant ground – it could have been presented in a much simpler fashion

A paragraph summarising the article

The paper begins by summarising the issues leading to the proposal – the current limited capabilities of mobile devices and the need for devices of many different platforms to be able to communicate. The paper then continues to summarise existing protocols and technologies which are relevant to the proposal. It proceeds by presenting the advantages such as cross platform communication and low resource consumption as well as the downsides such as the lack of stable, existing standards and the nature of XML as a languge (very wordy). The proposal is presented in full detail – basically the idea of a client (mobile device) talking to some kind of local area server (around 100m away tops) and discovering services available in the area (food for sale, shops, etc).

Webservices – Academic Articles

The article I read

The academic article I read was located at http://www-staff.it.uts.edu.au/~rsteele/mobilewebservices.pdf.  This article dealt with the topic of using web-service structures such as SOAP and XML to communicate information from access points in the vicinity to mobile devices.  There were several reasons why they were interested in this topic but the greatest seemed to be the idea of automatically notifying users of services available in the vicinity.

My thoughts

As a whole this article spent a large amount of time defining exactly what they mean and want to talk about.  To me it seemed to ramble on and overcomplicate the communication of what for a technical topic was actually quite simple.  While a more general article may have less clout than an academically published article it would certainly be easier to read.

Articles of interest (accessible)

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.12.1117&rep=rep1&type=pdf

Articles of interest (inaccessible)

http://portal.acm.org/citation.cfm?doid=944217.944234

http://portal.acm.org/citation.cfm?id=844357.844360

http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=955095

http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=982908