This is /home/jacob/projects/GNU/dejagnu/doc/dejagnu.info, produced by makeinfo version 4.8 from /home/jacob/projects/GNU/dejagnu/doc/dejagnu.texi. Copyright (C) 1992-2020 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". INFO-DIR-SECTION Software development START-INFO-DIR-ENTRY * DejaGnu: (dejagnu). The GNU testing framework. END-INFO-DIR-ENTRY  File: dejagnu.info, Node: Top, Next: Introduction, Up: (dir) DejaGnu ******* * Menu: * Introduction:: * Running tests:: * Running other DejaGnu commands:: * Customizing DejaGnu:: * Extending DejaGnu:: * Unit testing:: * Built-in Procedures:: * GNU Free Documentation License:: * Concept Index:: * Procedure Index:: * Variable Index:: Introduction * What is DejaGnu?:: * New in this release: Release Notes * Design goals:: * A POSIX conforming test framework: A POSIX Conforming Test Framework. * Installation:: Running tests * Running 'make check': Make Check. * Running runtest: Runtest. * Output files: Output Files. Running other DejaGnu commands * Invoking dejagnu:: Command line options for the launcher itself. * Invoking dejagnu help:: Reading man pages for dejagnu subcommands. * Invoking dejagnu report card:: Summarizing test results from many tools. Customizing DejaGnu * Global configuration file:: * Local configuration file:: * Board configuration file:: * Remote host testing:: * Configuration file values:: Extending DejaGnu * Adding a new testsuite:: * Adding a new tool:: * Adding a new target:: * Adding a new board:: * Board file values:: * Writing a test case:: * Debugging a test case:: * Adding a test case to a testsuite:: * Test case special variables: Test case variables. Unit testing * What is unit testing?:: Unit testing and system testing. * Running unit tests:: * DejaGnu unit test protocol:: DejaGnu native unit testing protocol. * C unit testing API:: * C++ unit testing API:: Reference * Built-in Procedures:: Indices * Concept Index:: * Procedure Index:: * Variable Index::  File: dejagnu.info, Node: Introduction, Next: Running tests, Prev: Top, Up: Top 1 Introduction ************** * Menu: * What is DejaGnu?:: * New in this release: Release Notes. * Design goals:: * A POSIX compliant test framework: A POSIX Conforming Test Framework. * Installation::  File: dejagnu.info, Node: What is DejaGnu?, Next: Release Notes, Up: Introduction 1.1 What is DejaGnu? ==================== DejaGnu is a framework for testing other programs, providing a single front-end for all tests. You can think of it as a library of Tcl procedures to help with writing a test harness. A _test harness_ is the infrastructure that is created to test a specific program or tool. Each program can have multiple testsuites, all supported by a single test harness. DejaGnu is written in Expect, which in turn uses Tcl, the Tool command language. There is more information on Tcl at the Tcl/Tk web site (http://www.tcl.tk) and the Expect web site (http://expect.nist.gov). Julia Menapace first coined the term _DejaGnu_ to describe an earlier testing framework she wrote at Cygnus Support for testing GDB. When we replaced it with the Expect-based framework, it was like DejaGnu all over again. More importantly, it was also named after my daughter, Deja Snow Savoye, who was a toddler during DejaGnu's beginnings. DejaGnu offers several advantages for testing: * The flexibility and consistency of the DejaGnu framework make it easy to write tests for any program, with either batch-oriented, or interactive programs. * DejaGnu provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test for `GDB' can run from any supported host system on any supported target system. DejaGnu runs tests on many single board computers, whose operating software ranges from a simple boot monitor to a real-time OS. * All tests have the same output format. This makes it easy to integrate testing into other software development processes. DejaGnu's output is designed to be parsed by other filtering script and it is also human readable. * Using Tcl and Expect, it's easy to create wrappers for existing testsuites. By incorporating existing tests under DejaGnu, it's easier to have a single set of report analyse programs.. Running tests requires two things: the testing framework and the testsuites themselves. Tests are usually written in Expect using Tcl, but you can also use a Tcl script to run a testsuite that is not based on Expect. Expect script filenames conventionally use `.exp' as a suffix. For example, the main implementation of the DejaGnu test driver is in the file `runtest.exp'.  File: dejagnu.info, Node: Release Notes, Next: Design goals, Prev: What is DejaGnu?, Up: Introduction 1.2 New in this release ======================= The following major, user-visible changes have been introduced since version 1.5.3. 1. Support for target communication via SSH has been added. 2. A large number of very old config and baseboard files have been removed. If you need to resurrect these, you can get them from version 1.5.3. If you can show that a board is still in use, it can be put back in the distribution. 3. The `--status' command line option is now the default. This means that any error in the testsuite Tcl scripts will cause runtest to abort with exit status code 2. The `--status' option has been removed from the documentation, but will continue to be accepted for backward compatibility. 4. `runtest' now exits with exit code 0 if the testsuite "passed", 1 if something unexpected happened (eg, FAIL, XPASS or UNRESOLVED), and 2 if an exception is raised by the Tcl interpreter. 5. `runtest' now exits with the standard exit codes of programs that are terminated by the SIGINT, SIGTERM and SIGQUIT signals. 6. The user-visible utility procedures `absolute', `psource' and `slay' have been removed. If a testsuite uses any of these procedures, a copy of the procedure should be made and placed in the lib directory of the testsuite. 7. Support was added for testing the D compiler. 8. `~/.dejagnurc' is now loaded last, not first. This allows the user to have the ability to override anything in their environment (even the `site.exp' file specified by `$DEJAGNU'). 9. The user-visible utility procedure `unsetenv' is *deprecated* and will be removed in the next release. If a testsuite uses this procedure, a copy should be made and placed in the lib directory of the testsuite.  File: dejagnu.info, Node: Design goals, Next: A POSIX Conforming Test Framework, Prev: Release Notes, Up: Introduction 1.3 Design goals ================ DejaGnu grew out of the internal needs of Cygnus Solutions (formerly Cygnus Support). Cygnus maintained and enhanced a variety of free programs in many different environments and needed a testing tool that: * was useful to developers while fixing bugs; * automated running many tests during a software release process; * was portable among a variety of host computers; * supported a cross-development environment; * permitted testing of interactive programs like `GDB'; and * permitted testing of batch-oriented programs like `GCC'. Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing. But all the requirements are important. For instance, it is imperative to make sure that `GDB' works as well when cross-debugging as it does in a native configuration. Probably the greatest challenge was testing in a cross-development environment. Most cross-development environments are customized by each developer. Even when buying packaged boards from vendors there are many differences. The communication interfaces vary from a serial line to Ethernet. DejaGnu was designed with a modular communication setup, so that each kind of communication can be added as required and supported thereafter. Once a communication procedure is written, any test can use it. Currently DejaGnu can use `ssh', `rsh', `rlogin', `telnet', `tip', and `kermit' for remote communications.  File: dejagnu.info, Node: A POSIX Conforming Test Framework, Next: Installation, Prev: Design goals, Up: Introduction 1.4 A POSIX compliant test framework ==================================== DejaGnu conforms to the POSIX 1003.3 standard for test frameworks. Rob Savoye was a member of that committee. POSIX standard 1003.3 defines what a testing framework needs to provide to create a POSIX compliant testsuite. This standard is primarily oriented to checking POSIX conformance, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework. TET was created by Unisoft for a consortium comprised of X/Open, Unix International and the Open Software Foundation. The POSIX documentation refers to "assertions". An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is day or night. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written. As there is no testsuite to verify that testing frameworks are POSIX 1003.3 compliant, this is done by repeatedly reading the standard and experimenting. One of the main things POSIX 1003.3 does specify is the set of allowed output messages and their definitions. Four messages are supported for a required feature of POSIX conforming systems and a fifth for a conditional feature. DejaGnu supports all five output messages. In this sense a testsuite that uses exactly these messages can be considered POSIX compliant. These definitions specify the output of a test case: PASS A test has succeeded. That is, it demonstrated that the assertion is true. FAIL A test has not succeeded - the assertion is false. The _FAIL_ message is based on this test case only. Other messages are used to indicate a failure of the framework. As with _PASS_, POSIX tests must return _FAIL_ rather than _XFAIL_ even if a failure was expected. XFAIL POSIX 1003.3 does not incorporate the notion of expected failures, so _PASS_, instead of _XPASS_, must also be returned for test cases which were expected to fail and did not. This means that _PASS_ is in some sense more ambiguous than if _XPASS_ is also used. UNRESOLVED A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion. This outcome requires a human to go over results to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to _PASS_ or _FAIL_ before a test run can be considered finished. Note that for POSIX, each assertion must produce a test result code. If the test isn't actually run, it must produce _UNRESOLVED_ rather than just leaving that test out of the output. This means that you have to be careful when writing tests to not carelessly use Tcl commands like _return_--if you alter the flow of control of the Tcl code you must insure that every test still produces some result code. Here are some of the ways a test may wind up _UNRESOLVED_: * Execution of a test is interrupted. * A test does not produce a clear result. This is usually because there was an _ERROR_ from DejaGnu while processing the test, or because there were three or more _WARNING_ messages. Any _WARNING_ or _ERROR_ messages can invalidate the output of the test. This usually requires a human to examine the output to determine what really happened - and to improve the test case. * A test depends on a previous test, which has failed. * The test was set up incorrectly. * A test script aborts due to a Tcl error. In this case, the DejaGnu framework inserts an _UNRESOLVED_ result as a placeholder for an unknown number of tests that were not run because the script crashed. UNTESTED A test was not run. This is a placeholder used when there is no real test case yet. UNSUPPORTED There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented. DejaGnu also uses this message when a testing environment (often a "bare board" target) lacks basic support for compiling or running the test case. For example, a test for the system subroutine _gethostname_ would never work on a target board running only a boot monitor. DejaGnu uses the same output procedures to produce these messages for all testsuites and these procedures are already known to conform to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3, you must avoid the _setup_xfail_ procedure as described in the _PASS_ section above and you must be careful to return _UNRESOLVED_ where appropriate, as described in the _UNRESOLVED_ section above.  File: dejagnu.info, Node: Installation, Prev: A POSIX Conforming Test Framework, Up: Introduction 1.5 Installation ================ Refer to the `INSTALL' in the source distribution for detailed installation instructions. Note that there is no compilation step as with many other GNU packages, as DejaGnu consists of interpreted code only. Save for its own small testsuite, the DejaGnu distribution does not include any testsuites. Testsuites for the various GNU development tools are included with those packages. After configuring the top-level DejaGnu directory, unpack and configure the test directories for the tools you want to test; then, in each test directory, run _make check_ to build auxiliary programs required by some of the tests, and run the test suites.  File: dejagnu.info, Node: Running tests, Next: Running other DejaGnu commands, Prev: Introduction, Up: Top 2 Running tests *************** There are two ways to execute a testsuite. The most common way is when there is existing support in the `Makefile' of the tool being tested. This usually consists of a _check_ target. The other way is to execute the `runtest' program directly. To run `runtest' directly from the command line requires either all of the correct command line options, or a *Note Local configuration file:: must be set up correctly. * Menu: * Running 'make check': Make Check. * Running runtest: Runtest. * Output files: Output Files.  File: dejagnu.info, Node: Make Check, Next: Runtest, Up: Running tests 2.1 Running 'make check' ======================== To run tests from an existing collection, first use `configure' as usual to set up the build directory. Then type `make check'. If the _check_ target exists, it usually saves you some trouble. For instance, it can set up any auxiliary programs or other files needed by the tests. The most common file the _check_ target depends on is the `site.exp' file. The `site.exp' contains various variables that DejaGnu uses to determine the configuration of the program being tested. Once you have run _make check_ to build any auxiliary files, you can invoke the test driver `runtest' directly to repeat the tests. You will also have to execute `runtest' directly for test collections with no _check_ target in the `Makefile'. GNU Automake has built-in support for DejaGnu. To add DejaGnu support to your generated `Makefile.in', just add the keyword `dejagnu' to the AUTOMAKE_OPTIONS variable in `Makefile.am'. This will ensure that the generated `Makefile.in' has a `check' target that invokes DejaGnu correctly. *Note DejaGnu Tests: (automake)Tests.  File: dejagnu.info, Node: Runtest, Next: Output Files, Prev: Make Check, Up: Running tests 2.2 Running runtest =================== `runtest' is the test driver for DejaGnu. You can specify two kinds of things on the `runtest' command line: command line options, and Tcl variables that are passed to the test scripts. The options are listed alphabetically below. `runtest' returns one of the following exit codes: 0 if all tests passed including expected failures and unsupported tests. 1 if any test failed, passed unexpectedly, or was unresolved. 2 if Expect encountered any error in the test scripts. * Menu: * Output States:: * Invoking runtest:: * Common Options: Common Operations.  File: dejagnu.info, Node: Output States, Next: Invoking runtest, Up: Runtest 2.2.1 Output States ------------------- `runtest' flags the outcome of each test as one of these cases. See *Note A POSIX Conforming Test Framework:: for a discussion of how POSIX specifies the meanings of these cases. PASS The most desirable outcome: the test was expected to succeed and did succeed. XPASS A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure. FAIL A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug. XFAIL A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is _UNSUPPORTED_ instead. UNRESOLVED Output from a test requires manual inspection; the testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected. UNTESTED A test case is not yet complete, and in particular cannot yet produce a _PASS_ or _FAIL_. You can also use this outcome in dummy "tests" that note explicitly the absence of a real test case for a particular property. UNSUPPORTED A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine. `runtest' may also display the following messages: ERROR Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. POSIX testsuites should not emit this message; use _UNSUPPORTED_, _UNTESTED_, or _UNRESOLVED_ instead, as appropriate. WARNING Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests. NOTE An informational message about the test case.  File: dejagnu.info, Node: Invoking runtest, Next: Common Operations, Prev: Output States, Up: Runtest 2.2.2 Invoking runtest ---------------------- This is the full set of command line options that `runtest' recognizes. Option names may be abbreviated to the shortest unique string. `-a', `--all' Display all test output. By default, _runtest_ shows only the output of tests that produce unexpected results; that is, tests with status _FAIL_ (unexpected failure), _XPASS_ (unexpected success), or _ERROR_ (a severe error in the test case itself). Specify `--all' to see output for tests with status _PASS_ (success, as expected) _XFAIL_ (failure, as expected), or _WARNING_ (minor error in the test case itself). `--build [triplet]' _triplet_ is a system triplet of the form _cpu-manufacturer-os_. This is the type of machine DejaGnu and the tools to be tested are built on. For a normal cross environment this is the same as the host, but for a Canadian cross, they are different. `-D0', `-D1' Start the internal Tcl debugger. The Tcl debugger supports breakpoints, single stepping, and other common debugging activities. See the document Debugger for Tcl Applications (http://expect.sourceforge.net/doc/tcl-debug.ps) by Don Libes. If you specify _-D1_, the _expect_ shell stops at a breakpoint as soon as DejaGnu invokes it. If you specify _-D0_, DejaGnu starts as usual, but you can enter the debugger by sending an interrupt (e.g. by typing -). `--debug' Turns on the Expect internal debugging output. Debugging output is displayed as part of the _runtest_ output, and logged to a file called `dbg.log'. The extra debugging output does _not_ appear on standard output, unless the verbose level is greater than 2 (for instance, to see debug output immediately, specify `--debug -v -v'). The debugging output shows all attempts at matching the test output of the tool with the scripted patterns describing expected output. The output generated with `--strace' also goes into `dbg.log'. `--global_init [name]' Use _name_ as the global init file instead of `site.exp' in _libdir_. The default is, of course, `site.exp'. Note that this option accepts a relative file name, interpreted starting at _libdir_, so a file in a subdirectory may be used. This is probably less useful for most sites, but is orthogonal with the `--local_init' option and may be useful in large testing labs. `--help' Prints out a short summary of the _runtest_ options, then exits (even if you specify other options). `--host [triplet]' _triplet_ is a system triplet of the form _cpu-manufactuer-os_. Use this option to override the default string recorded by your configuration's choice of host. This choice does not change how anything is actually configured unless -build is also specified; it affects _only_ DejaGnu procedures that compare the host string with particular values. The procedures _ishost_, _istarget_, _isnative_, and _setup_xfail_ are affected by `--host'. In this usage, _host_ refers to the machine that the tests are to be run on, which may not be the same as the _build_ machine. If `--build' is also specified, then `--host' refers to the machine that the tests will be run on, not the machine DejaGnu is run on. `--host_board [name]' The host board to use. `--ignore [tests(s)] ' The name(s) of specific tests to ignore. `--local_init [name]' Use _name_ as the testsuite local init file instead of `site.exp' in the current directory and in _objdir_. The default is, of course, `site.exp'. Note that this option accepts a relative file name, so a file in a subdirectory may be used. `--log_dialog' Emit Expect output to stdout. The Expect output is usually only written to the `.log' file. By enabling this option, they are also printed to standard output. `--mail [address(es)]' Send test results to one or more email addresses. `--objdir [path]' Use _path_ as the top directory containing any auxiliary compiled test code. The default is '.'. Use this option to locate pre-compiled test code. You can normally prepare any auxiliary files needed with _make_. `--outdir [path]' Write log files in directory `path'. The default is '.', the directory where you start _runtest_. This option affects only the summary (`.sum') and the detailed log files (`.log'). The DejaGnu debug log `dbg.log' always appears (when requested) in the local directory. `--reboot [name]' Reboot the target board when `runtest' starts. When running tests on a separate target board, it is safer to reboot the target to be certain of its state. However, when developing test scripts, rebooting can take a lot of time. `--srcdir [path]' Use `path' as the top directory for test scripts to run. _runtest_ looks in this directory for any subdirectory whose name begins with the toolname (specified with `--tool'). For instance, with `--tool gdb', _runtest_ uses tests in subdirectories `gdb.*' (with the usual shell-like filename expansion). If you do not use `--srcdir', _runtest_ looks for test directories under the current working directory. `--strace [n]' Turn on internal tracing for _expect_, to n levels deep. By adjusting the level, you can control the extent to which your output expands multi-level Tcl statements. This allows you to ignore some levels of _case_ or _if_ statements. Each procedure call or control structure counts as one "level". The output is recorded in the same file, `dbg.log', used for output from `--debug'. `--target [triplet]' Use this option to override the default setting (native testing). _triplet_ is a system triplet of the form _cpu-manufacturer-os_. This option changes the configuration `runtest' uses for the default tool names, and other setup information. `--target_board [name(s)]' The list of target boards to run tests on. `--tool [name(s)]' Specifies which testsuite to run, and what initialization module to use. `--tool' is used _only_ for these two purposes. It is _not_ used to name the executable program to test. Executable tool names (and paths) are recorded in `site.exp' and you can override them by specifying Tcl variables on the command line. For example, including `--tool' gcc on the command line runs tests from all test subdirectories whose names match `gcc.*', and uses one of the initialization modules named `config/*-gcc.exp'. To specify the name of the compiler (perhaps as an alternative path to what _runtest_ would use by default), use _GCC=path-to-gcc_ on the _runtest_ command line. `--tool_exec [name]' The path to the tool executable to test. `--tool_opts [options]' A list of additional options to pass to the tool. `-v', `--verbose' Turns on more output. Repeating this option increases the amount of output displayed. Level one (_-v_) is simply test output. Level two (_-v -v_) shows messages on options, configuration, and process control. Verbose messages appear in the detailed (`*.log') log file, but not in the summary (`*.sum') log file. `-V', `--version' Prints out the version numbers of DejaGnu, Expect, and Tcl. `-x', `--xml' Generate XML output. The output file is named after the tool with an .xml extension. `testfile'.exp[=arg(s)] Specify the names of testsuites to run. By default, _runtest_ runs all tests for the tool, but you can restrict it to particular testsuites by giving the names of the _.exp_ scripts that control them. _testsuite_.exp cannot include directory names, only plain filenames. `arg(s)' specifies a subset of test cases to run. For compiler or assembler tests, which often use a single _.exp_ script covering many different test case files, this option allows you to further restrict the tests by listing particular test cases. For larger testsuites such as that included in GCC, this can save a lot of time. Some tools support wildcards here, but this varies from tool to tool. Typically the wildcards _?_, _*_, and _[chars]_ are recognized. `tclvar'=value You can define Tcl variables for use by your test scripts in the same style used with _make_ for environment variables. For example, _runtest GDB=gdb.old_ defines a variable called `GDB'; when your scripts refer to `$GDB' in this run, they use the value _gdb.old_. The default Tcl variables used for most tools are defined in the main DejaGnu _Makefile_; their values are captured in the `site.exp' file.  File: dejagnu.info, Node: Common Operations, Prev: Invoking runtest, Up: Runtest 2.2.3 Common Options -------------------- Typically, you don't need to use any command line options. The `--tool' option is only required when there is more than one testsuite in the same directory. The default options are in the local `site.exp' file, created by `make site.exp'. For example, if the directory `gdb/testsuite' contains a collection of DejaGnu tests for GDB, you can run them like this: $ cd gdb/testsuite $ runtest --tool gdb The test output follows, then ends with: === gdb Summary === # of expected passes 508 # of expected failures 103 /usr/latest/bin/gdb version 4.14.4 -nx You can use the option `--srcdir' to point to some other directory containing a collection of tests: $ runtest --srcdir /devo/gdb/testsuite By default, `runtest' prints only the names of the tests it runs, output from any tests that have unexpected results, and a summary showing how many tests passed and how many failed. To display output from all tests (whether or not they behave as expected), use the `-a' (all) option. For more verbose output about processes being run, communication, and so on, use `-v' (verbose). To see even more output, use multiple `-v' options. See *Note Invoking runtest:: for a more detailed explanation of each `runtest' option.  File: dejagnu.info, Node: Output Files, Prev: Runtest, Up: Running tests 2.3 Output files ================ DejaGnu always writes two kinds of output files. Summary output is written to the `.sum' file, and detailed output is written to the `.log' file. The tool name determines the prefix for these files. For example, after running with `--tool gdb', the output files will be called `gdb.sum' and `gdb.log'. For troubleshooting, a debug log file that logs the operation of Expect is available. Each of these will be described in turn. * Menu: * Summary log file:: * Detailed log file:: * Debug log file::  File: dejagnu.info, Node: Summary log file, Next: Detailed log file, Up: Output Files 2.3.1 Summary log file ---------------------- DejaGnu always produces a summary (`.sum') output file. This summary lists the names of all test files run. For each test file, one line of output from each `pass' command (showing status _PASS_ or _XPASS_) or `fail' command (status _FAIL_ or _XFAIL_), trailing summary statistics that count passing and failing tests (expected and unexpected), the full pathname of the tool tested, and the version number of the tool. All possible outcomes, and all errors, are always reflected in the summary output file, regardless of whether or not you specify `--all'. If any of your tests use the procedures `unresolved', `unsupported', or `untested', the summary output also tabulates the corresponding outcomes. For example, after running `runtest --tool binutils' a summary log file will be written to `binutils.sum'. Normally, DejaGnu writes this file in your current working directory. Use the `--outdir' option to select a different output directory. *Sample summary log* Test Run By bje on Sat Nov 14 21:04:30 AEDT 2015 === gdb tests === Running ./gdb.t00/echo.exp ... PASS: Echo test Running ./gdb.all/help.exp ... PASS: help add-symbol-file PASS: help aliases PASS: help breakpoint "bre" abbreviation FAIL: help run "r" abbreviation Running ./gdb.t10/crossload.exp ... PASS: m68k-elf (elf-big) explicit format; loaded XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed C types === gdb Summary === # of expected passes 5 # of expected failures 1 # of unexpected failures 1 /usr/latest/bin/gdb version 4.6.5 -q  File: dejagnu.info, Node: Detailed log file, Next: Debug log file, Prev: Summary log file, Up: Output Files 2.3.2 Detailed log file ----------------------- DejaGnu also saves a detailed log file (`.log'), showing any output generated by test cases as well as the summary output. For example, after running `runtest --tool binutils', a detailed log file will be written to `binutils.log'. Normally, DejaGnu writes this file in your current working directory. Use the `--outdir' option to select a different output directory. *Sample detailed log for g++ tests* Test Run By bje on Sat Nov 14 21:07:23 AEDT 2015 === g++ tests === Running ./g++.other/t01-1.exp ... PASS: operate delete Running ./g++.other/t01-2.exp ... FAIL: i960 bug EOF p0000646.C: In function `int warn_return_1 ()': p0000646.C:109: warning: control reaches end of non-void function p0000646.C: In function `int warn_return_arg (int)': p0000646.C:117: warning: control reaches end of non-void function p0000646.C: In function `int warn_return_sum (int, int)': p0000646.C:125: warning: control reaches end of non-void function p0000646.C: In function `struct foo warn_return_foo ()': p0000646.C:132: warning: control reaches end of non-void function Running ./g++.other/t01-4.exp ... FAIL: abort 900403_04.C:8: zero width for bit-field `foo' Running ./g++.other/t01-3.exp ... FAIL: segment violation 900519_12.C:9: parse error before `;' 900519_12.C:12: Segmentation violation /usr/latest/bin/gcc: Internal compiler error: program cc1plus got fatal signal === g++ Summary === # of expected passes 1 # of expected failures 3 /usr/latest/bin/g++ version cygnus-2.0.1  File: dejagnu.info, Node: Debug log file, Prev: Detailed log file, Up: Output Files 2.3.3 Debug log file -------------------- The `runtest' option `--debug' creates a file showing the output from Expect in debugging mode. The `dbg.log' file is created in the current directory. The log file shows the string sent to the tool being tested by each `send' command and the pattern it compares with the tool output by each `expect' command. The log messages begin with a message of the form: expect: does {tool output} (spawn_id n) match pattern {expected pattern}? For every unsuccessful match, Expect issues a _no_ after this message. If other patterns are specified for the same Expect command, they are reflected also, but without the first part of the message (_expect... match pattern_). When Expect finds a match, the log for the successful match ends with _yes_, followed by a record of the Expect variables set to describe a successful match. *Example debug log file for a GDB test* send: sent {break gdbme.c:34\n} to spawn id 6 expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? no {.*\(gdb\) $}? no expect: does {} (spawn_id 0) match pattern {return} ? no {\(y or n\) }? no {buffer_full}? no {virtual}? no {memory}? no {exhausted}? no {Undefined}? no {command}? no break gdbme.c:34 Breakpoint 8 at 0x23d8: file gdbme.c, line 34. (gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern {Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes expect: set expect_out(0,start) {18} expect: set expect_out(0,end) {71} expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } epect: set expect_out(spawn_id) {6} expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } PASS: 70 0 breakpoint line number in file This example exhibits three properties of Expect and DejaGnu that might be surprising at first glance: * Empty output for the first attempted match. The first set of attempted matches shown ran against the output _{}_ -- that is, no output. Expect begins attempting to match the patterns supplied immediately; often, the first pass is against incomplete output (or completely before all output, as in this case). * Interspersed tool output. The beginning of the log entry for the second attempted match may be hard to spot: this is because the prompt _{(gdb) }_ appears on the same line, just before the _expect:_ that marks the beginning of the log entry. * Fail-safe patterns. Many of the patterns tested are fail-safe patterns provided by GDB testing utilities, to reduce possible indeterminacy. It is useful to anticipate potential variations caused by extreme system conditions (GDB might issue the message _virtual memory exhausted_ in rare circumstances), or by changes in the tested program (_Undefined command_ is the likeliest outcome if the name of a tested command changes). The pattern _{return}_ is a particularly interesting fail-safe to notice; it checks for an unexpected prompt. This may happen, for example, if the tested tool can filter output through a pager. These fail-safe patterns (like the debugging log itself) are primarily useful while developing test scripts. Use the `error' procedure to make the actions for fail-safe patterns produce messages starting with _ERROR_ on standard output, and in the detailed log file.  File: dejagnu.info, Node: Running other DejaGnu commands, Next: Customizing DejaGnu, Prev: Running tests, Up: Top 3 Running other DejaGnu commands ******************************** DejaGnu now features auxiliary commands not directly related to running tests, but somehow related to the broader purpose of testing. These commands are run via the `dejagnu' multiplex launcher, which locates an appropriate script and the required interpreter and then runs the requested command. * Menu: * Invoking dejagnu:: Command line options for the launcher itself. * Invoking dejagnu help:: Reading man pages for dejagnu subcommands. * Invoking dejagnu report card:: Summarizing test results from many tools.  File: dejagnu.info, Node: Invoking dejagnu, Next: Invoking dejagnu help, Prev: Running other DejaGnu commands, Up: Running other DejaGnu commands 3.1 Invoking `dejagnu' ====================== The `dejagnu' launcher is primarily designed to pass most options on to the scripts that it runs, but does process the `--help' and `--version' options entirely internally, while also recognizing the `--verbose' option. `dejagnu' [options...] `dejagnu' --help `dejagnu' --version Note that the command names may contain multiple words. In this case, the command can be given as separate arguments to `dejagnu' or combined with dashes (`-'); both forms are equivalent. All words of the command name must appear before any options. The search for a command terminates when an option is found. Note that the first valid command found is used. A longer command name can be shadowed by a shorter command name that happens to be a prefix of the longer name, if the command name is given as multiple arguments. The equivalent form with the longer command name combined using dashes into a single argument will correctly refer to the otherwise shadowed command. The `dejagnu' launcher can also be run using symbolic links, provided that the shell places the name under which `dejagnu' was invoked in `$0'. Any dash-separated words after "dejagnu" in the name of such a link are taken to be the leading words of a command name. The `dejagnu' launcher supports alternate implementations depending upon available interpreters. Options for the `dejagnu' launcher itself cannot be abbreviated, since the launcher has no way to know which abbreviations are unique and which would be ambiguous to the invoked command. `--help' Print a help message instead of running a command. `-V', `--version' Print a version banner for the launcher itself including the version of DejaGnu. Any command given is ignored. `-v', `--verbose' Emit additional output describing the inner workings of the `dejagnu' launcher. This option is also passed on to the invoked command. All arguments after the command name are passed to the invoked command.  File: dejagnu.info, Node: Invoking dejagnu help, Next: Invoking dejagnu report card, Prev: Invoking dejagnu, Up: Running other DejaGnu commands 3.2 Invoking `dejagnu help' =========================== The `dejagnu help' tool displays long-form documentation for DejaGnu auxiliary commands that are invoked using the `dejagnu' launcher. `dejagnu help' [options...] Again, command names may contain multiple words. This command forms an operand by joining all words in the command name using dashes (`-') and prepending `dejagnu-'. This is then used as the name of a manual page and passed to the `man' command. If the manual page is in a particular directory relative to the script implementing this command, a full file name is produced, otherwise, `man' performs its normal search. The `--verbose' option causes additional output describing the inner workings of the `dejagnu help' command to be produced. The `--path', `-w', and `-W' options are passed to `man'.  File: dejagnu.info, Node: Invoking dejagnu report card, Prev: Invoking dejagnu help, Up: Running other DejaGnu commands 3.3 Invoking `dejagnu report card' ================================== The `dejagnu report card' tool produces a tabular summary of the results from test runs by reading the summary files that DejaGnu produces. `dejagnu report card' [