Minix Man Pages

Man Page or Keyword Search:
Man All Sections 1 - General Commands 2 - System Calls 3 - Subroutines 4 - Special Files 5 - File Formats 6 - Games 7 - Macros and Conventions 8 - Maintenance Commands 9 - Kernel Interface n - New Commands Minix Minix 2.0.0 Minix 3.1.5 Minix 3.1.6 Minix 3.1.7 Minix 3.1.8 Minix 3.2.0 Minix 3.2.1 Minix 3.3.0 X11R7 default i386 Architecture
Apropos Keyword Search (all sections) html pdf ascii Output format

home | help

---

KYUA-ATF-INTERFACE(1)     BSD General Commands Manual    KYUA-ATF-INTERFACE(1)

NAME
     atf-interface -- Description of the ATF test program interface

DESCRIPTION
     The interface of ATF test programs is the interface of the test programs
     linked to the atf-c, atf-c++ and atf-sh libraries provided by ATF.

     The ATF interface can be understood as the mechanisms used by test
     programs to communicate with the runtime engine as well as the
     assumptions that test programs and test cases can make while running.

     A test case is the most basic part of a test suite.  A test case is
     supposed to reproduce one, and only one, scenario.  For example: if the
     item under test was a function, the test case would provide a single set
     of input parameters to the function and check its output; If the item
     under test was a binary, the test case would provide a single set of
     arguments to the program and check its behavior.

   Test case parts
     Test cases have three parts:

     Head       Programmatically defines metadata properties.  The head must
                not perform any other thing than defining such properties.  In
                particular, no testing whatsoever can happen in the head.
                (Ideally the definition of metadata properties would not
                happen programmatically.)

     Body       The actual test case which performs any desired testing and
                reports a result.  The body is executed by the runtime engine
                in a deterministic way; see the isolation section below.

     Cleanup    An optional cleanup routine.  Note that the runtime engine
                will attempt to clean up the work directory automatically, so
                this routine should only be provided in cases where the test
                modifies system-wide state not known by the runtime engine.
                The cleanup part is executed in the same directory as the
                body.  However, the body and the cleanup parts do not share
                the same process space; the only way to pass data around from
                the body to the cleanup is by means of files in the work
                directory.

   Metadata properties
     The following test case metadata properties must be exported in the test
     case list for every test case:

     ident
         Single-word string.  The name of the test case.  Must be unique
         within the test program.

     The following test case metadata properties may be exported in the test
     case list for every test case:

     descr
         Multi-word string.  A textual description for the test case.
         Usually, providing a descriptive identifier is better than providing
         a textual description.

     has.cleanup
         Boolean.  Whether the test case defines a cleanup routine or not.

     require.arch
         Whitespace separated list of the architectures required by the test
         case.  If defined, the test case is skipped unless the host
         architecture matches any of the values defined in this property.

     require.config
         Whitespace separated list of configuration variable names.  The list
         of configuration variables that must be defined.  The test is skipped
         if any of these is missing.

     require.files
         Whitespace separated list of absolute paths to installed files.  If
         any of these files is not found, the test case is skipped.

     require.machine
         Whitespace separated list of the machine types required by the test
         case.  If defined, the test case is skipped unless the host machine
         type matches any of the values defined in this property.

     require.progs
         Whitespace separated list of program names (either absolute names or
         base names).  If any of these programs is not found, the test case is
         skipped.

     require.user
         Either 'root' or 'unprivileged'.  If 'root', the test case must be
         run as the superuser or otherwise it is skipped.  If 'unprivileged',
         the test case must be run as an unprivileged user or else it is
         skipped.

     timeout
         Integer.  The amount of seconds the test case can run for before it
         is killed by the runtime engine.

   Configuration properties
     The following properties may be defined by the runtime engine and are
     propagated to the test cases:

     unprivileged-user
         String, optional.  Specifies the name of the user under which tests
         that set 'require.user=unprivileged' are executed.

   Results
     A test case must always report a result by creating the results file
     specified through the -r flag.  For convenience when running test cases
     without the runtime engine, this file may point to /dev/stdout or
     /dev/stderr in which case the file must not be created (because the
     creation will fail).

     Aside from creating the results file, the process in which the test case
     runs must terminate in particular ways for the test result to be
     considered valid.

     If the test case fails to create the test result, if the test result is
     created but contains an invalid syntax, or if the termination status of
     the process does not match the requirements of the test result, the
     runtime engine marks the test case as 'broken'.  Note that the 'broken'
     state is decided by the runtime engine; a test case cannot report itself
     as 'broken'.

     The general syntax for the results file is as follows:

           <status>[[(int)]: reason]

     The following results are allowed:

     expected_death
         The process is expected to terminate either due to a clean call to
         exit(3) or due to the reception of a signal.  The contents of the
         file are 'expected_death: <reason>0  ."' Example: 'expected_death:
         Calling libdofoo breaks due to bug xyz'.

     expected_exit
         The process is expected to terminate cleanly.  The contents of the
         file are 'expected_exit: <reason>' if the exit code is irrelevant or
         'expected_exit(<exitcode>): <reason>' if the process must terminate
         with a given exit code.  Examples: 'expected_exit: Calling bar exits
         but it should not' or 'expected_exit(123): Calling bar exits with an
         unexpected code'.

     expected_failure
         The process must exit cleanly with an EXIT_SUCCESS exit code.  The
         contents of the file are 'expected_failure: <reason>0' Example:
         'expected_failure: 2 + 2 = 3'.

     expected_signal
         The process is expected to terminate due to the reception of a
         signal.  The contents of the file are 'expected_signal: <reason>' if
         the signal number is irrelevant or 'expected_signal(<signalno>):
         <reason>' if the process must terminate due to a particular signal.
         Examples: 'expected_signal: Calling bar crashes' or
         'expected_signal(1): Calling bar kills ourselves due to unhandled
         SIGHUP'.

     expected_timeout
         The process is expected to hang for longer than its timeout metadata
         property.  Only the runtime engine can control this situation because
         the runtime engine is the one implementing the timeout functionality.

     failed
         The process must exit cleanly with an EXIT_FAILURE exit code.  The
         contents of the file are 'failed: <reason>0  ."' Example: 'failed:
         Failed on purpose0  ."'

     passed
         The process must exit cleanly with an EXIT_SUCCESS exit code.  The
         contents of the file are 'passed0  ."'

     skipped
         The process must exit cleanly with an EXIT_SUCCESS exit code.  The
         contents of the file are 'skipped: <reason>0  ."' Example: 'skipped:
         Skipped because the foo is not present0  ."'

   Isolation
     The runtime engine attempts to isolate test cases from other test cases
     in the same test program and from the rest of the system by performing
     what is called test case isolation.

     Whenever the user runs a test program binary by hand (i.e. not through
     kyua(1)), the test program will print a warning message stating that test
     case isolation does not work and therefore the program may cause side-
     effects and/or report invalid values.

     The runtime engine must set the __RUNNING_INSIDE_ATF_RUN environment
     variable to the magic value 'internal-yes-value' to tell the test
     programs that they are being run with isolation enabled.

     The test case isolation performs the following:

     Process space
         Each test case body and cleanup routines are executed in independent
         processes.  Corollary: the test case can do whatever it wants to the
         current process (such as modifying global variables) without having
         to undo such changes.

     Process group
         The test case body and cleanup are executed in their own process
         groups.  Should they spawn any children, such children should
         maintain the same process group.  This is done to allow the runtime
         engine to kill the whole process subtree once the test case finishes
         (or if the test case hangs).

     Work directory
         The test case body and its cleanup (if any) are executed in a
         temporary directory automatically created by the runtime engine.
         This temporary directory is shared among the body and cleanup parts
         of a single test case but is completely separate from the temporary
         directories of other tests.  Corollary: the test case body and
         cleanup routines can write to their current directory without
         bothering to clean any files and/or directories they create.  The
         runtime engine takes care to recursively delete the temporary
         directories after the execution of a test case.  Any file systems
         mounted within the temporary directory will be unmounted if possible.

     Home directory
         The HOME environment variable is set to the absolute path of the work
         directory.

     Umask
         The value of the umask is set to 0022.

     Environment
         The LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY,
         LC_NUMERIC and LC_TIME variables are unset.  The TZ variable is set
         to 'UTC'.

     Process limits
         The maximum soft core size limit is raised to its corresponding hard
         limit.  This is a simple, best-effort attempt at allowing test cases
         to dump core for further diagnostic purposes.

   Test programs
     A test program is, simply put, a collection of related test cases.  The
     test program can be seen as a command-line dispatcher for the test cases.
     A test program must provide one or more test cases.  If it does not
     contain any test case, the runtime system will report it as invalid.

     Test programs expose their list of test cases in a machine parseable
     format.  The runtime engine obtains the list of test cases to know what
     tests to run and to know how to set up the environment of each test prior
     execution.  The test program must not do any test when asked to dump its
     test case list.

     The generic syntax to obtain the list of test cases included in a test
     program is:

           <test-program> -l

     The list of test cases follows the following format:

           LIST ::= HEADER NEWLINE TEST_CASES

           HEADER ::= 'Content-Type: application/X-atf-tp; version="1"'
           NEWLINE ::= '\n'
           TEST_CASES ::= TEST_CASE | TEST_CASE NEWLINE TEST_CASES

           TEST_CASE ::= IDENT_PROPERTY PROPERTIES
           IDENT_PROPERTY ::= 'ident' DELIM STRING NEWLINE
           DELIM ::= ': '

           PROPERTIES ::= PROPERTY | PROPERTY PROPERTIES
           PROPERTY ::= PROPERTY_NAME DELIM STRING NEWLINE
           PROPERTY_NAME ::= (see below)

     An example:

           Content-Type: application/X-atf-tp; version="1"

           ident: addition
           descr: Tests that the addition function works

           ident: subtraction
           descr: Tests that the subtraction function works

           ident: remove
           descr: Tests removing files
           require.root: true
           timeout: 50
           has.cleanup: true

     The syntax to run a test case body part is:

           <test-program> [-r resfile] [-s srcdir] [-v var=value]* <test-case>[:body]

     This must run the test case body "as is", without any attempt of
     isolating it from the rest of the system.  It is the responsibility of
     the runtime engine to do such isolation.

     The runtime engine always passes the path of a nonexistent file to -r,
     which must be created by the test case; and always passes an absolute
     path to the -s flag pointing to the directory containing the test program
     executable.

     The runtime engine shall pass any configuration variables it wants
     through the -v flag, and these can be later inspected by the test case at
     will.

     A note to users: if you run the test case by hand (not through kyua(1)
     nor atf-run(1)) from the command line, none of the isolation features
     described in the isolation section apply.  This means that the test case
     can (and probably will) write to the current directory and leave garbage
     behind.  Also, given that the test case is executed without e.g. clearing
     the environment, the results of the test case may differ from those
     obtained when running the test case inside the runtime engine.  Only use
     this for debugging purposes (i.e. to run the test case code under GDB).

     The syntax to run a test case cleanup part is:

           <test-program> [-s srcdir] [-v var=value]* <test-case>:cleanup

     This can only be performed if and only if the test case sets the
     has.cleanup property to true.  Otherwise the behavior of executing the
     cleanup part is undefined.

     The same rules for -s and -v apply as to when running the body.

     The cleanup part must be executed in the same directory as the body but
     in a separate process space.  The only way for test cases to transfer
     state (if any) from the body to the cleanup routine is by means of files
     in the current directory.

     The cleanup part does not have to worry about deleting temporary files
     created in the current directory.  The runtime engine does this
     automatically.

SEE ALSO
     kyua-test(1), kyuafile(5)

BSD                            September 9, 2012                           BSD

---

NAME | DESCRIPTION | SEE ALSO

Want to link to this manual page? Use this URL: https://man.minix3.org/?query=kyua-atf-interface&sektion=7&manpath=Minix