Home · All Namespaces · All Classes · Grouped Classes · Modules · Functions

[Contents] [Next: Chapter 2]

Chapter 1: Writing a System Test

In this first chapter we will demonstrate how to write a simple system test, and how to execute it.

Setting up the environment

Tests are saved in directories, one directory per test. As a convention we keep the name of the directory the same as the test name although this isn't a requirement. Tests are further stored in a 'tests' directory immediately under the application or library that is being tested. So let's create our first test directory, and we'll be writing a test for the addressbook application.

    cd src/applications/addressbook
    mkdir -p tests/sys_demo

Note that we write and save tests in the source tree, and we can execute them from the source as well as the build tree.

The tests subdirectories have a special meaning when used in combination with QBuild, the Qt Extended build system. Anything below tests is ignored from a normal build. In other words, all code is compiled when 'configure && qbuild && qbuild image' is executed except for the testcases beneath tests.

To run a testcase we do a qbuild test but more about that a little later.

Writing the Test

New system tests are defined by creating a new QtScript file (named ending with .js by convention), containing a testcase object with one or more test functions: So cd into tests/my_demo and create a file named sys_demo.js with the following content:

        testcase = {
            testFunction1: function() {
                print( "Hello World" );
            },
            testFunction2: function() {
                print( "Testfunction 2" );
            }
        }

The example above shows the creation of a testcase object containing two test functions, testFunction1 and testFunction2. Obviously not much exciting is going to happen when we run this test. But we'll get to that a little later.

The second bit that needs to be done is tell the build system about our new project. For this we need to create a project file named qbuild.pro. The contents of the file is as follows:

        CONFIG+=systemtest
        TARGET=sys_demo
        SOURCES+=sys_demo.js

Running the test for the first time

Now that we have created a little test it's about time to run it for which we use QBuild again. qbuild is a binary that is created when configure is ran on the Qt Extended source tree and will end up in the bin directory of the build tree. Whenever we want to run tests, or re-compile parts of Qt Extended for that matter, it is important to call the qbuild binary from the build tree. The simplest mechanism to make that happen is to add the bin directory to our PATH variable:

        export PATH=~/build/qtopia/demo/bin:$PATH

(assuming that ~/build/qtopia/demo is where the code has been build.

Once the PATH is set correctly we can simply run the test as follows:

        cd <your_src_location>/src/applications/addressbook/tests/sys_demo
        qbuild test

The test will be built and executed and results in something like this:

    1 echo '#!/bin/sh' > <your_build_dir>/src/applications/addressbook/tests/sys_demo/sys_demo
    2 echo 'exec <your_build_dir>/bin/qtuitestrunner <your_src_location>/src/applications/addressbook/tests/sys_demo/sys_demo.js "$@"' >> <your_build_dir>/src/applications/addressbook/tests/sys_demo/sys_demo
    3 chmod 755 <your_build_dir>/src/applications/addressbook/tests/sys_demo/sys_demo
    4 export QPEHOME=$(mktemp -d /tmp/qtopia_maketest_XXXXXX); <your_build_dir>/src/applications/addressbook/tests/sys_demo/sys_demo $ARGS; rm -rf $QPEHOME
    5 ********* Start testing of sys_demo *********
    6 Config: Using QTest library 4.4.2, Qt 4.4.2
    7 PASS   : sys_demo::initTestCase()
    8 QDEBUG : sys_demo::testFunction1() Hello World
    9 PASS   : sys_demo::testFunction1()
    10 QDEBUG : sys_demo::testFunction2() Testfunction 2
    11 PASS   : sys_demo::testFunction2()
    12 PASS   : sys_demo::cleanupTestCase()
    13 Totals: 4 passed, 0 failed, 0 skipped
    14 ********* Finished testing of sys_demo *********

In the above snippet line numbers have been added for clarity. Lines 1-4 are a bit of magic to create a subdirectory for the test in the build tree, and to create an 'executable' for the test. Once the executable is created, it is executed and the results are printed to stdout (or to a file if we use extra command line parameters).

Line 5-14 are the result of the test being execute. Line 5 simply marks the start of the test. Line 6 shows the version numbers of Qt and Qtest that are used. Line 7 is the result of a special function named initTestCase. Note that we haven't written an initTestCase yet, so this function is empty (and shouldn't fail). Once the test is initialized (with initTestCase) the testrunner will start executing all testfunctions, in principle in the order in which they are discovered in the file. Line 8 is the result of the print statement we did in testFunction1. Note also that the output clearly marks this information as being a part of sys_demo::testFunction1. Line 9 is the final test result for testFunction1. Since we did nothing else but a print statement it PASSes the test. Line 10 and 11 show the results for testFunction2. Line 12 shows the result for the counterpart to initTestCase: after all testfunctions have been executed the system takes a moment to clean up any garbage that has been created during test execution. Just like it's init counterpart is cleanupTestCase always called, and called only once, at the end of a test. Since we haven't actually defined anything yet the cleanup is empty and we'd expect it to PASS. Line 13 shows the accumulated totals for the test and Line 14 finally marks the end of the test.

In this example we have called qbuild test from the source tree, but we could have done the same, and with the same result from the build tree. This is because we're calling qbuild which knows everything about the actual configuration we're testing, as well as where to find it.

Special functions

As with QTestLib unit tests, special reserved functions include the four init/cleanup functions. These behave identically to their unit test counterparts but will be explained here for completeness.

Let's re-write our testcase a bit, but this time we use the special functions.

            testcase = {
                initTestCase: function() {
                    print( "Init complete test" );
                },
                cleanupTestCase: function() {
                    print( "Cleanup complete test" );
                },
                init: function() {
                    print( "Init test function" );
                },
                cleanup: function() {
                    print( "Cleanup test function" );
                },
                testFunction1: function() {
                    print( "Hello World" );
                },
                testFunction2: function() {
                    print( "Testfunction 2" );
                }
            }

When we run qbuild test again we get the following output:

    ********* Start testing of sys_demo *********
    Config: Using QTest library 4.4.2, Qt 4.4.2
    QDEBUG : sys_demo::initTestCase() Init complete test
    PASS   : sys_demo::initTestCase()
    QDEBUG : sys_demo::testFunction1() Init test function
    QDEBUG : sys_demo::testFunction1() Hello World
    QDEBUG : sys_demo::testFunction1() Cleanup test function
    PASS   : sys_demo::testFunction1()
    QDEBUG : sys_demo::testFunction2() Init test function
    QDEBUG : sys_demo::testFunction2() Testfunction 2
    QDEBUG : sys_demo::testFunction2() Cleanup test function
    PASS   : sys_demo::testFunction2()
    QDEBUG : sys_demo::cleanupTestCase() Cleanup complete test
    PASS   : sys_demo::cleanupTestCase()
    Totals: 4 passed, 0 failed, 0 skipped
    ********* Finished testing of sys_demo *********

Interacting with the System Under Test

System tests do not have direct access to the code under test. Instead, the system testrunner connects to Qt Extended, and sends and receives information via a communication protocol. This effectively controls access to the tested system, reducing the impact of tests on results and emulating standard user behaviour.

Suppose we wish to create a simple test which launches an application; we first need to ensure that Qt Extended has finished loading successfully, since we can't be sure of the state of the system we have connected to. This is accomplished by using the waitForQtopiaStart() function in initTestCase():

            initTestCase: function()
            {
                waitForQtopiaStart();
            }

Since initTestCase() is called before any test functions are executed, this will pause test execution until it receives confirmation that Qt Extended has started. Now we're ready to start the application:

            testFunction1: function()
            {
                startApplication( "Contacts" );
            }

The startApplication() method will attempt to start the specified application (in this case, Contacts). If the specified application cannot be started, testFunction will generate a test failure and abort the test.

Once the application has started, we can begin testing. While it is possible to execute as many actions in a test function as desired, the recommendation is to dedicate each test function to a single use case. The test function should then attempt to realistically simulate each step of the use case, and verify that the expected result occurs. For example, assume we need to test creating a new contact; a simple test function could look like this:

            creating_a_contact: function()
            {
                // Start the application
                startApplication( "Contacts" );

                // Open the options menu and choose "New contact"
                select( "New contact", optionsMenu() );

                // Enter some details in the "Name" and "Emails" fields
                enter( "Frank Grimes", "Name" );
                enter( "frank@example.com", "Emails" );

                // Select 'Back' from the softkey menu to commit changes
                select( "Back", softMenu() );

                // We should now be at the contacts list.
                // Verify that we can select the contact we just created.
                select( "Frank Grimes" );

                // We should now be viewing the contact.
                // Move to "Details" tab.
                select( "Details", tabBar() );

                // Now verify that the details screen contains
                // the expected details.
                var text = getText();
                verify( text.contains("Frank Grimes") );
                verify( text.contains("frank@example.com") );
            }

This test function will start Contacts, navigate through several screens, input data, and verify that the data was successfully saved as a new contact. It is important to note that this test has not explicitly simulated any key or mouse events (although this can be done). This means that the same test can be used as-is on both a keyboard and touchscreen device. This is possible because the implementation of select() and enter() know how to select items and enter data on both keyboard and touchscreen devices.

The next chapter gives more detail on how navigation works.

[Contents] [Next: Chapter 2]