Tuesday, August 26, 2008

AntSys -- lightweight and powerful system test framwork


Why write AntSys

I was looking for a well-know and easy to use open source tool for our system testing.
My requirements are:

  • Test Model, provides the concept of test suite, test case and test result

  • Test Method: launch command line application, check the outputs. which means should be easy to
    * configure parameters
    * setup environment variables
    * able to redirect stand input, output, and error
    * get back the process exit code

  • Test Assertions: should built-in with the following assertions:
    * check if output file is created as expect
    * check the content of output file against expected file.
    * check process exit code
    * check the content of system.ouput.
    * check the content of system.error.

  • Test Result:It should generate XML and HTML report

  • Test Language/API: the test language should provide rich API to help users write test case:
    such as create/delete directory, copy files, and even xml processing.

  • Extensibility: the test lanauge can be extended by end users.

  • Although the requirements are quite generic, I can't find a well-known OSS project. Most companies are
    using their own home-grow system test framework.
    Let's create a OSS tool to solve this problem then!

    What is AntSys

    AntSys is a lightweight and powerful system test framwork built on top of Ant.
    It is designed to be easy to work and user can create system/function test cases in xml file without writing single line of code.

    AntSys Tasks

    AntSys provides the following Ant tasks to write a system test:
  • tastsuite: is used to aggregate tests into groups of test cases that should be run together.

  • testcase: is a set of conditions under which a tester will determine if a requirement or use case upon an application is partially or fully satisfied.

  • assertfileexist: check if the output file is exist as expected.
  • assertfileequal: check the content of output file is the same as expected test result.
  • assertfilegrep: check the content of output file contains expected string defined in regular expression
  • assertpropertyequals: check the ant property against expected value. This is offened used after 'exec' task to check the process exit code.

  • Example test script

    Here is one example of test script to test echo command.

    <project name="antsys-systemtest" basedir="." default="systemtest">
    <target name="systemtest">
    <!-- this test suite is using AntSys to test echo shell commands -->
    <testsuite name="CommandTestSuite" failonerror="false" todir="./report" htmlreport="true">
    <!-- test echo command output -->
    <testcase name="echo_test">
    <!-- create working dir for testing -->
    <mkdir dir="work"/>
    <!-- use echo to create a test file -->
    <exec vmlauncher="false" executable="echo" output="work/test.dat">
    <arg value="testmessage"/>
    <!-- check the file exists or not -->
    <assertfileexist file="./work/test.dat" errormsg="should generate test.dat" />
    <assertfilegrep file="./work/test.dat" regexp="^test" errormsg="should start with test" />
    <assertfileequal expectedfile="./work/test.dat" actualfile="./work/test.dat"/>
    <delete dir="work"/>