|
Home · All Namespaces · All Classes · Grouped Classes · Modules · Functions | |
The QtUiTest framework is a tool for automatic and semi-automatic Ui/System level testing of Qt based applications and libraries, specifically applications developed for embedded/handheld devices. QtUiTest provides functionality commonly found in system test frameworks but with a special attention for creating maintainable cross-platform tests.
Contents:
QtUiTest is, by design, quite similar to the QTestLib API that is used for Unit level testing. In contrast to QTestLib however, QtUiTest based tests are written in QtScript. Please refer to the QTestLib Manual if you are looking for a Unit test framework.
QtUiTest is a scripting test language plus some basic test execution tools and as such does not provide a complete Integrated Development Environment (IDE) to write and execute tests. Tests can be written with any standard editor or development IDE. Tests can be executed from the command line or in continuous integration test systems.
QtUiTest aims to provide a test scripting language that is:
QtUiTest scripts are written from a User perspective. When a User uses a GUI application all he/she does is apply a few concepts that form the basis of using a GUI application. For instance, a button can be clicked on and then 'the text on the button' will happen.
It is irrelevant to the user that the actual UI implementation uses a button. The UI could just as well have been an html page with a hyperlink in it with a similar text as on the button. In both cases, the user will 'select' the text to initiate a certain action from the application. So, today the implementation can be a button, tommorrow it can be a hyperlink, the day after tomorrow it can be something that hasn't been invented yet: the principle remains the same.
So conceptually a user does not 'click' keys, but rather uses the concept of a button to 'select' an action or hits keys on a real or virtual keyboard to 'enter' text. The user wants something to happen and has an understanding of what needs to be done to "get there".
On a new system a user first needs to learn the User Manual. The more the system behaves in a way the user already knows, and the more the system provides functionality in a way the user can predict, the more "Intuitive" and "Usable" the system is experienced by the user.
System testing is, in our view, very much related to these fundamental issues and just like a real user, QtUiTest understands the UI concepts that form the foundation of a UI application and uses a simple syntax to enable the tester to describe it's Use Cases.
Next to being able to write tests in a language that is as close as possible to the user experience it is also important to write maintainable code. Automating tests is well known to be major investment, and tests being broken for every small UI change, or tests that need to be modified for each slightly different device (or Operating System) are a major problem. QtUiTest is designed in such a way that the script will result in good readable and understandable code which reduces the need for additional documentation, as well as tests that are platform agnostic. For example:
function create_contact()
{
var contact_name = "John Doe";
var contact_email = "johndoe@hotstuff.com";
var contact_company = "HotStuff";
select( "Contacts", launcherMenu() ); // select and start the Contacts application from the Grid Menu
select( "New contact", optionsMenu() ) ; // Open the Options menu and select "New contact"
waitForTitle( "New Contact" ); // Make sure that we are in the "New Contact" dialog
select( "Contact", tabBar() ); // Go to the "Contact" tab
enter( contact_name, "Name" ) ; // enter a text in a field labeled "Name"
compare( contact_email, "Emails" ) ; // verify that the text in a field labeled "Emails" is 'contact_email'
if (contact_company != undefined) { // depending on the testdata ...
setChecked( true, "Business contact" ); // check a checkbox
enter( contact_company, "Business contact/Company" ) ; // and enter text in the "Company" field
} else {
setChecked( false, "Business contact" ); // or else uncheck the checkbox labeled "Business contact"
}
select( "Back", softMenu() ); // select "Back" from the soft menu
verify( getList().contains( contact_name ) ); // verify that the contacts list displays the new contact
}
As simple as this example may be, it shows many of the powerfull features of QtUiTest:
For an introduction to the QtUiTest scripting grammar please read the QtUiTest Scripting Language primer and for details about writing scriptable Qt applications go to the QtScript documentation.
QtUiTest consists of a number of libraries and plugins that run on the System Under Test (SUT) i.e. on the device side plus a script executer that runs on the desktop side. This code is compiled when Qt Extended is configured with the '-add-module qtuitest' configuration option.
The code is located:
When developing/maintaining source code it is crucially important that all development attributes are kept together as close as possible and are stored in a version control system. To make this process as simple as possible all available tests for our libraries, applications, etc are integrated into the source tree. As a convention, all tests are put under "tests" subdirectories and can contain a combination of Unit, Integration, System and Performance tests.
For example:
src/applications/addressbook
contains all source code for the addressbook application, and
src/applications/addressbook/tests/sys_addressbook
contains a System test for the addressbook application, and similarly
src/libraries/qtopia
contains a great number of subdirectories such as tst_qtimezone and sys_contextkeymanager which all contain a Unit/System test.
A crucial component in testing is the maintenance of test data. Test data is, depending on the situation, either kept separate for a specific test or needs to be shared between multiple tests.
Code and data that is shared between multiple tests can be found in:
tests/* which is a directory specific for testing. It contains helper scripts, shared data, etc.
QtUiTest is designed with performance testing in mind, and should not have a big impact on device performance. Of course, a device running without QtUiTest is likely to be slightly faster, which is at least erring towards the safe side.
QtUiTest consists of a few extra libraries and plugins that are installed on a device, but are not required on a shipped product. Installation of these extra libraries is optional, and by default will not be installed (or build).
The QtUiTest plugins provide unrestricted access to all parts of the System Under Test and as such create a security breach. To prevent security breaches on end-user devices the SUT must be compiled AND ran with extra commands to enable QtUiTest. Without these commands the QtUiTest code is not installed, or accessible, on the device.
To be able to use QtUiTest only Qt (Extended) is required. The test framework creates a small hook that accesses the object tree maintained by QApplication, and it will use a few basic classes such as the network classes to communicate with the script runner on the desktop (host) machine.
QtUiTest can handle information from a great deal of Qt's Widget classes although by default only a subset of all Widgets are supported, typically those that can be found on small embedded devices. The remaining QWidget derived classes that aren't standard supported can be added via custom plugins.
To emulate keyboard and mouse events the test system uses low level classes in Qt such as QWsKeyboardEvent and QEventFilter that form the core of Qt on target systems and as such should not impose a restriction to the applicability of QtUiTest in many configurations.
To be able to run System tests with QtUiTest three steps must be taken:
The difference between the two possible configurations (with and without the qtuitest module) is minimal from code impact point of view and it is therefore unlikely that a System tested with qtuitest included will have a different behaviour from one that is run without qtuitest.
To be able to run System tests the QtUiTest code needs to be compiled and included in the image. For example:
cd <my-build-dir>
<my-depot-dir>/configure -add-module qtuitest [other-config-options]
bin/qbuild
bin/qbuild image
Note that a System tests will fail when the SUT isn't configured with the qtuitest module.
The desktop (host) machine and device need to be set up appropriately so that an ethernet (socket) connection can be established between the desktop and the device, i.e. "ping <your-device-ip>" or something similar should work. The system administrative tasks involved are beyond the scope of this document.
Once the System is build with QtUiTest enabled and the image is installed (which could mean 'flashed on a device') the last step is to activate the system test mode.
Typically the "Test Mode" parameter is saved into a configuration file that is read once by the SUT upon startup, and applied afterwards. Depending on the SUT the configuration must be edited manually or can be modified via a settings application.
For Qt Extended the configuration file can be modified via "Settings->Startup Flags", checking the "Test mode" checkbox and then restarting Qt Extended.
Typically this setting needs to be changed only once and the action only needs to be repeated when the device is fully flashed with a new version.
Note that a System test will fail when it can't connect to the device in which case it will give you a hint to activate the "Test mode".
See also QtUiTest Security.
Each System Test is stored in its own file ending with .js and should have a project file qbuild.pro that tells the build system how to build the test.
Within the test script, a testcase object is created containing several test functions. Within each test function, the QtUiTest API scripting commands may be used, as well as standard commands provided by QtScript.
Helper functions and global variables can be written outside of the testcase object.
For example:
testcase = {
testFunction1: function() {
startApplication("Dog Walker");
select("Walk the dog...", optionsMenu());
compare( getText(), "Dog was walked." );
},
testFunction2: function() {
var i = 1;
helper(i);
}
}
function helper(num) {
verify( num > 0 );
...
}
Tests can be put anywhere in a directory tree but as a convention each test is stored in a subdirectory under the application or library that it is testing. When using qbuild as the build system this has the added advantage that 'qbuild test' and/or 'make test' will execute all tests for the library/application that is being developed.
For example, the 'tests' directory structure for the Qtopia library is something like this:
src/libraries/qtopia/tests/sys_contextkeymanager/qbuild.pro
src/libraries/qtopia/tests/sys_contextkeymanager/sys_contextkeymanager.js
src/libraries/qtopia/tests/tst_qcontent/qbuild.pro
src/libraries/qtopia/tests/tst_qcontent/tst_qcontent.cpp
In this example the Qtopia library is tested by two tests: a System test (sys_contextkeymanager) and a Unit test (tst_qcontent). The actual testcode is in the .js and .cpp files, and the qbuild.pro files are required by the build system.
The project files for System tests are usually very simple. The qbuild.pro for sys_contextkeymanager has the following contents:
CONFIG+=systemtest
TARGET=sys_contextkeymanager
SOURCES+=sys_contextkeymanager.js
Project files for Unit tests may have a few extra lines. For instance qbuild.pro for the tst_content Unit Test has the following content:
TEMPLATE=app
CONFIG+=qtopia unittest
TARGET=tst_qcontent
HEADERS*=my_special_header.h
SOURCES*=tst_qcontent.cpp
Please refer to the QTestLib Manual for more detailed instructions regarding Unit test project files.
Writing a Manual System test is typically the first step in many test automation efforts. Unless the test can be recorded and instantly results in a series of understandable steps that can be re-run immediately it is far better to first put the test steps on paper and run through the steps manually to check that the idea is executable (and repeatable) before time is invested in automating all steps.
The downside about Manual tests is that the test is usually 'somewhere else', i.e. in a text document or spreadsheet that isn't necessarily saved in the same version control system as the source code it tests, or in a format that can be easily merged by the version control system.
QtUiTest integrates the Manual test process into the Automation process and makes it possible to seemlessly transition from a full Manual to a Semi-Automatic to a Full-Automatic test. All work is done in the same (ascii) text file, and thus the intermediate steps are all recorded in the version control system for easy retrieval, merging and tracking.
A Manual test is written by using the prompt() command:
my_manual_test: function()
{
prompt( "* Do something\n"+
"* Do something else\n"+
"* Verify that the result is as expected");
}
When a System test is executed it can be a combination of fully automated and semi-automated test functions. The automated tests simply execute without user intervention and result in a PASS/FAIL/etc. A Manual test will be executed until a prompt is detected at which time a Dialog is shown on the Desktop (host) machine containing the specified prompt text. In the dialog, the stars (*) will be replaced with "1:", "2:", etc to indicate the steps that need to be taken in sequence. The advantage of using stars (*) in the code is that lines can be inserted/removed without having to renumber the remaining steps until the end of the prompt.
The Dialog will also contain "Pass" and "Fail" buttons as well as a "Record" button that puts the test system into recording mode. The Pass and Fail buttons obviously must be used to indicate the result of the manual step.
It is perfectly fine, and to be expected, that prompts() will be combined with automated steps. For instance, one could launch an application and then select a few menu options before arriving at the section of the test that requires manual intervention.
For instance:
startApplication("Dog Walker");
select("Walk the dog...", optionsMenu());
prompt( "* Verify that the dog is having a good time");
A test can be 'build' by invoking qbuild, the new optimized build system for Qt Extended. For backwards compatibility make is also supported so, wherever this documentation tells you to use qbuild, you can also use make.
The result of the build depends on whether the test is a Unit or System test:
After a successfull build the result for both Unit and System testing is an 'executable' that has the same name as the test (or actually what is defined as the TARGET in the qbuild.pro file) and that can be ran from the command line just like any other executable.
For example:
cd src/libraries/qtopia/tests/sys_contextkeymanager
qbuild
./sys_contextkeymanager
Note that the system will always attempt to build a test if it hasn't been build yet, so building is in most cases an implicit step that is done automatically as part of the test execution.
System tests are executed by a QtUiTest Script Execution engine:
qsystemtestrunner <scriptname> <options>
which runs on the desktop machine (host environment).
When a System test is 'build' a small wrapper script is created that will take care of calling the script runner script with the appropriate parameters, so generally speaking there will never be a need to call qsystemtestrunner directly. For example:
cd sys_dogwalker
./sys_dogwalker -foobar
will call qsystemtestrunner with sys_dogwalker and -foobar as a parameter.
Alternatively tests can be executed from anywhere within the source tree, in which case all tests within scope are build and executed.
cd src/libraries/qtopia
qbuild test
This mechanism has the added advantage that it will run multiple tests (both System and Unit tests) with one command.
When running a System Test, the test will attempt to either run a new instance of Qt Extended, or connect to an existing instance. This behaviour and other system test specific behaviour can be controlled using the following command-line arguments:
Example, running the sys_dogwalker.js test against an already running Qt Extended instance on the local machine:
cd sys_dogwalker
./sys_dogwalker
Example, running the sys_dogwalker.js test against an existing Qt Extended instance on a device located at 10.10.10.20:
cd sys_dogwalker
./sys_dogwalker -authost 10.10.10.20
Example, running the walking_the_dog testfunction in sys_dogwalker.js, instructing the test to learn new and changed data:
cd sys_dogwalker
./sys_dogwalker -authost 10.10.10.20 -learn walking_the_dog
When running tests using the alternative make test or qbuild test call conventions, it is also possible to add additional parameters to the command. But because the parameters need to be passed up the chain to the actual script a small deviation to the syntax is required, and all parameters must be assigned to ARGS as a double quoted string:
cd sys_dogwalker
<build_dir>/bin/qbuild test ARGS="-authost 10.10.10.20 -learn walking_the_dog"
For a full list of possible command line options please run a test with the -help command, e.g.
cd sys_dogwalker
./sys_dogwalker -help
or alternatively
cd sys_dogwalker
<build_dir>/bin/qbuild test ARGS="-help"
Note that the qbuild test syntax allows to run multiple tests in one go, e.g.
cd libraries/qtopia
<build_dir>/bin/qbuild test ARGS="-authost 10.10.10.20 -learn"
will run all available tests for the qtopia library on a device with IP address 10.10.10.20 and it will learn all new and changed testdata.
QtUiTest tests can be executed from the command line of a desktop (Host) machine and as such are perfectly suited to be executed in Continuous Integration Test Systems. Testdata, as well as binaries, that are required for the test-execution will be installed on the fly on a device when required. The output format of the test, the testresults, can be printed to the screen or saved into a file in a number of formats (such as XML) to allow easy post processing by test-reporting and defect-tracking systems.
Please refer to the QtUiTest Tutorial for a detailed example about writing Qt Extended System tests.
To write a System Test, create a new file containing a test script, which defines a "testcase" object with one or more test functions.
For example:
testcase = {
testFunction1: function() {
...
},
testFunction2: function() {
...
}
}
The testfunctions can be written with a combination of standard QtScript and special QtUiTest commands. In most cases what will be used from QtScript are a few constructs such as a for ... loop, if ... then .. else and variable assignments. The remaining 99% of the code will most likely be composed of the QtUiTest commands which are grouped below.
The QtUiTest framework is designed around a script execution engine that is executed on the desktop (host machine) and a set of QtUiTest plugins that are loaded by the System Under Test (SUT). The plugins are running in the SUT process space, i.e. may be running on a device, and communicate with the System Test script execution engine via a socket based communication protocol.
Everything that a test can do is a query that is send to the SUT, and the script runner will be blocked until a positive or negative reply is received from the SUT. The code that runs on the device side is kept as minimalistic as possible and is written with performance (both in terms of speed as well as size) in mind.
The set of queries that can be send to the SUT are designed in such a way that the overhead introduced by QtUiTest is minimal but at the same time, does not limit the posibilities of a tester. In other words, things that will be done a lot require only a single turnaround and are completely handled by the QtUiTest plugins whereas other, less used, features can be implemented (for instance in a helper script) by using more low level queries.
Many of the QUiTest queries use so called 'query paths' to identify an application and/or widget. A complete queryPath specifies both the application and the object to access and adheres to the format: application:object.
In cases where the System Under Test consists of a system of multiple applications with one application (the server) managing the behaviour of the other applications QtUiTest will connect to the server and will communicate to the other applications via the server. In such cases the server would be accessible via e.g. qpe: (where qpe is the name of the server) and qpe/addressbook: would be used to send a query to the addressbook binary that is controlled by the server.
The application part is optional. If it isn't specified the query will automatically be sent to the current application. For a single application solution, such as a Qt desktop application, this will always be the one and only application under test, and for multi application systems, such as Qt Extended, this will be the application that currently has input focus.
A Label is a (usually static) text as can be seen on the screen and that is associated with an editable field. The Label and Edit field form a buddy pair and clicking on the Label will set the input focus on the Edit field. The buddy relationship can be hardcoded but from User perspective the expectancy usually is that a Label is shown just in front of the editable field or, in exceptional cases, just above the edit field (in a Left-to-Right language). Sometimes the Label is a buddy of itself, such as is the case for i.e. a Radio Button in which case clicking on the Button text will set the input focus to the Radio button and may select/de-select the button.
For example, the code below will enter the text "dog walker" in a text field next to a Label with text "Occupation".
enter( "dog walker", "Occupation" );
In words the example says: "enter the text 'dog walker' into a field labeled 'Occupation'.
A special case of a Label is an empty string. This will resolve to the widget that currently has focus, or (for touchscreen devices) the first field in the dialog. Note that for tests that should run both on keyboard and touchscreen devices it may be risky to rely on this feature. Always explicitely specifying a Label is the most safe and portable thing to do.
Using a Label to identify a field is the preferred mechanism for identifying fields. It is reliable, easy to write, easy to understand and easy to debug.
In exceptional cases where multiple widgets have the same label text, or a widget has no Label the Labeltext form of the query path is ambiguous and an explicit Signature is required to identify the widget.
A Signature is a full description of the Object/Widget and all its parents and is organised much like a directory path and may be implemented as a string of class names followed by a tag that uniquely identifies the object instances.
A special function is available that can return a signature to access a 'difficult' object field.
enter( "dog walker", signature("Label",1));
The signature() function will return the 1st editable field BELOW the field that is associated with label "Label" and can be used in cases where the Label field is visible and associated with an editable field but the editable field just below (or above) is not paired to a buddy label.
Some testcases require the existence of non-volatile test data, such as screen snapshots. To simplify the process of data gathering, the test system supports the concept of a learn mode. When a test is run in learn mode, certain actions will result in the generation of test data, which will be presented to the user to manually verify once, then stored for use in subsequent test runs. A system test is run in learn mode by supplying the -learn or -learn-all command line options.
The system test framework will store generated test data in the testdata subdirectory of the directory containing the test script.
NOTE: It is up to the tester to ensure that new learned elements are added to a version control system.
Within a system test it is often necessary to start and close applications. The system test framework provides several functions for this purpose:
It is possible to access files on the System Under Test (even if this is another device) from within a system test.
A frequently used mechanism when performing system tests is to modify the current date and/or time of the System Under Test (SUT) to simulate or test certain behavior. QtUiTest helps out with a number of functions that are targeted towards manipulating the Time.
A special case is setting the date/time in a virtual environment. In these situations the SUT usually runs as a 'normal' user in a virual environment on a desktop machine and as such might not have root permission to alter the date and time. QtUiTest resolves this problem by 'overriding' the standard setsystemtime calls so that the system time is altered for the SUT but not for the desktop machine itself.
The same mechanism is also used in performance testing, hence starting a performanc test, then changing the system time and then stopping the performance test will still result in an accurate measurement of the elapsed time.
The system test framework provides powerful methods of obtaining information on particular objects and widgets in the system.
The system test framework provides several ways of entering text into widgets and for navigating through applications.
These functions are input method independant and will have the same syntax for both Keyboard and Mouse/Touchscreen based devices and are therefore the preferred commands to use in tests.
The system test framework provides several ways of generating arbitrary key events.
Note that Keypad simulation is a less preferred technique that should be used in exceptional cases only since it makes the test specific for keyboard based devices. Other functions such as select() are the preferred mechanism to write tests because these will work both for Keyboard and Mouse/Touchscreen based devices.
The system test framework provides methods to simulate input from a mouse or touchscreen.
Note that Mouse simulation is a less preferred technique that should be used in exceptional cases only since it makes the test specific for mouse/touchscreen based devices. Other functions such as select() are the preferred machanism to write tests because these will work both for Keyboard and Mouse/Touchscreen based devices.
The system test framework can be used to provide a simulated AT modem and hence simulate incoming phone calls.
Some functionality does not fall into the aforementioned categories:
QtUiTest's high-level commands require knowledge of the widgets they interact with. For example, select() must know how to select items from both a combobox and a list view.
When testing an application which uses custom widgets, some additional steps may be required in order to use high-level QtUiTest commands. The logic for interacting with custom widgets must be implemented either in the application under test or in a plugin.
See Creating a QtUiTest Test Widget for more details.
For a complete description of the QtUiTest API please refer to the provided documentation:
| QSystemTest API | a complete reference of commands for Qt applications. |
| QtopiaSystemTest API | a complete reference of commands for Qt Extended applications. |
| QtopiaSystemTestModem | an object that can be used to simulate an AT modem. |
| Copyright © 2009 Trolltech | Trademarks | Qt Extended 4.4.3 |