System Test
Time your website with
WebWait - from the creator of AjaxPatterns.orgFrom Ajax Patterns
There's an archived version of this pattern available, taken from the Ajax Pattern book draft, showing roughly how it appeared before the page became publicly editable.
Evidence: 1/3
Tags: Automated FunctionalTest Javascript PerformanceTest Regression Robot UnitTest TDD Test
Goal Story
Devi has completed the first cut of a new Ajaxian application and wants to build an automated test of the whole thing. With the help of a testing application, she records a typical usage so it can be played back later. The software takes note of the final state, which will be used for verification during automated playback.
Problem
How can you test an entire Ajax application?
Solution
Build automated tests to simulate user behaviour and verify the results. A system test is one which exercises a complete system in much the same way as a user would. The most common category of system test is a functional test - verifying the functionality - but system testing also encompass qualities such as performance and robustness. While they are often conducted manually, with users clicking and mousing around, it's highly preferable to automate as much as possible in order to promote continuous integration.
Automated system tests remain somewhat difficult, but tools are improving. They can be broadly categorised as follows:
- Browser Controller. A specialised "robot"-style testing tool will fire up the browser, then use the underlying operating system's API to create the effect of a user interacting with the browser. Several commercial controllers are able to record user actions for later playback. (caution: capture/replay is the less cost effective way of test automation)
- Embedded Browser Script. Run the application in a standard browser, and embed a Browser-Side Test on the page. The test can use some Javascript trickery to simulate user activity. Don't let the browser aspect put you off, because it's feasible to have a continuous integration tool like Ant control coordinate Browser-Side Tests, as that pattern points out.
- Browser Simulation. The application is executed by a model of a real browser. Typically, the simulation pulls a page from a URL somewhere and builds up a DOM to represent it. The DOM can then be manipulated, e.g. the simulation allows a button click operation to occur or a form to be submitted. While several frameworks like this exist, the important issue for Ajax developers is how well they simulate Javascript. Javascript interpreters are implemented in several languages, so it's possible for Browser Simulations to incorporate them in order to simulate a realistic Ajax interaction.
TODO Diagrams
The first two approaches involve a real browser, which is a more realistic test, but it does imply that the test must run on a machine capable of launching the browser, or somehow controlling it remotely and this may block the machine during test execution.
System tests require not just automation of user behaviour, but of result verification. As with other tests, results can be verified with a combination of assertions against the DOM and assertions against server-side state.
Tool Support
SWEA (Webius Software)
SWEA automates regression and functional testing for Web applications. The program creates an automation API for any Web application developed with HTML, DHTML or AJAX. The Web application becomes programmatically accessible from any .NET language. SW Explorer Automation Visual Designer records, replays test scripts and generates C# or VB.NET script code which can be used in NUnit or Visual Studio unit tests.
- Commercial from Webius.
InCisif.net (InCisif Software)
InCisif.net is an automation tool designed to implement client-side functional testing of web applications under Internet Explorer 6.x or 7.x, using the C#, VB.NET or IronPython language with Visual Studio 2003, 2005 or express editions.
Commercial from InCisif Software
ItsNat, Natural AJAX (Innowhere Software)
ItsNat is a Component based Java AJAX Web Framework with built-in functional web testing support. ItsNat simulates a Universal W3C Java Browser in the server, the client DOM tree is basically a clone of the server W3C DOM tree with automatic synchronization from server to client.
The server can fire W3C DOM events and send them to the browser simulating user actions, these actions are usually processed by the server again as in a normal AJAX application. As the Java test code is in the server too, this test code can check the results as GUI changes (checking the expected changes in the server DOM tree) and/or the expected business behavior (checking changed/added/removed data).
- Dual licensed: Affero GPL v3 / commercial license for closed source projects.
Squish/Web (froglogic)
Squish is an automated, functional GUI testing tool. It allows to record, edit and run web/ajax tests in different browsers (IE, Firefox, Safari, Konqueror, etc.) on different platforms without having to modify the test scripts. Supports different scripting languages for tests.
- Commercial from froglogic
WebAii and Automation Design Canvas (ArtOfTest, Inc.)
WebAii is a .NET automation library provided for free to the quality assurance and developer community. It is built for Ajax and rich web application automation. WebAii 1.1 Supports IE and Firefox. WebAii 2.0 adds support for Safari. More browsers are planned. Includes Visual Studio unit test templates for VB.NET and C# to help you create a new WebAii based test. Also supports a new approach to abstracting markup applications using TestRegions. Supports testing in NUnit and Visual Studio Team Test.
Automation Design Canvas is their premier product for testing modern web 2.0 applications. It is built from the ground up for modern web applications that heavily rely on Ajax! It is specialized for testing ASP.NET applications, and is extremely cost-effective and very easy to use. Automate scenarios you never thought possible!
- WebAii is commercial from ArtOfTest, Inc. Free license to use. Premier support is offered for the serious QA tester or developer.
- There is a trial version of Automation Design Canvas available for download. After the trial period you will need to purchase a license.
GhostTrain
Scriptaculous's GhostTrain is an Embedded Browser Script with recording capability. When you start the recording mode, a Scriptaculous test case begins to fill up inside the div. Each mouse and key action becomes a new line of the test case. Later on, you can add your own assertions about the application state. GhostTrain is a very promising work-in-progress.
Watir
Watir follows the Browser Controller approach, enabling a Ruby programmer to script IE behaviour and interrogate the DOM at any stage. You can easily integrate Watir commands into a standard test case. Example syntax:
ie = Watir::IE.start("http://ajaxpatterns.org")
ie.button(:name, "Search").click
WatiN
WatiN follows the Browser Controller approach based on Watir. WatiN enables a .Net programmer to automate web application testing with Internet Explorer and Firefox. The framework can be used in any kind of tester application like NUnit, MBUnit, VSTS test, FIT, FitNesse or your own winforms or console app. Support for AJAX is included.
open source
Selenium
Selenium is an open-source tool by Thoughtworks developers, an Embedded Browser Script framework that works in IE, Mozilla, Firefox, and Safari. A Javascript Selenium component is embedded inside the browser application under test, and it communicates with a server-side component. Interesting, there's flexibility on the location of the server-side component. It can be placed in the web server, to provide a quick summary in the web page being developed. Alternatively, it can be run as a separate server, where the server controls the browser (like a Browser Controller would), and the embedded browser component merely transmits server commands to the application.
HTTPUnit
HttpUnit is an open-source Java testing framework, built on JUnit, and specifically designed for testing web applications. It works as a Browser Simulator, converting an HTML page into a Java HTMLPage object which wraps a Java DOM model of the original HTML. HttpUnit simulates Javascript by delegating to the Rhino Javascript engine, although support is limited and the project seems to be asleep.
DojoToolkit
DOJO Toolkit is a JavaScript toolkit. It comes with a UnitTest Module that uses Rhino to let you run tests as a Browser Simulator from an Ant task. Given the project's momentum and focus on Ajax, it's likely to evolve into a very solid product.
HtmlUnit
HmltUnit is an open-source Java library that can be considered as "browser for Java programs". It works like a browser that you can drive from your java code to get pages, fill form, click link, ... HtmlUnit is able to simulate Javascript execution like Internet Explorer or like Firefox (according to the chosen configuration). Simulation is already quite good and AJAX is fully supported.
Canoo WebTest
Canoo WebTest is an open-source (Apache license) tool to test web applications. Tests are specified (and not programmed) as Ant scripts and executed using HtmlUnit as underlying browser (therefore no display is needed). It provides high level abstraction allowing to write short, easy to maintain tests. Contained steps are not limited to HTML pages but allow to test xml, pdf, and Excel documents as well as Applets or Emails. Additionally the whole power of Ant can be used and it can be easily extended for application specific needs. Because WebTest is based on Ant it integrates directly in continuous integration tools like Cruise Control. Particularity of WebTest are the generated reports. Unlike other tools that just produce "JUnit like" reports (which are most of the time not enough to find a failure cause), WebTest results allow to follow the whole test execution and contain all received answers from the server(s).
Refactoring Illustration
The Ajax Patterns Reader is refactored here to include Scriptaculous's GhostTrain, discussed above. You can hit Escape anywhere in the document to open it up, and then use the buttons to record a test case, which can then be played back.
To hook up GhostTrain, the scriptaculous library files must be included (quite a lot to download, but keep in mind they don't need to be included in production):
<script type="text/javascript" src="/run/Lib/js/sc/prototype.js"></script> <script type="text/javascript" src="/run/Lib/js/sc/controls.js"></script> <script type="text/javascript" src="/run/Lib/js/sc/effects.js"></script> <script type="text/javascript" src="unittest.js"></script> <script type="text/javascript" src="ghosttrain.js"></script>
The only other requirement is to register the GhostTrain object as a keypress handler. This also serves to start up GhostTrain if it's not yet running.
window.onload = function() {
Event.observe(document, 'keypress', GhostTrain.intercept, true);
...
}
Related Patterns
Browser-Side Test
System Test frameworks that work as Embedded Browser Scripts are a form of Browser-Side Test.
Simulation Service
For integration tests, you can configure the server to run against a Simulation Service. That way, you'll have tight control over server behaviour, ensuring the final state is deterministic.
Want to Know More?
A Summary of Functional Testing Frameworks
Acknowledgements
Thanks to Scriptaculous developers, Jon Tirsen and Thomas Fuchs, for clarifying some issues on GhostTrain and their general testing framework.
