Contents
This page describes several things we should do to improve the testing and quality assurance of Vesta.
Improve Unit Tests
Many of the packages that make up Vesta include programs under the "tests" sub-binding in their result. The exact nature of these programs varies:
Some are self-testing exercisers that test some part of the system, can be run with no arguments or user input, check their own results, and indicate success or failure with their exit status. (TestIntSeq and TestFPStream are two such examples.) These could easily be used to automatically run tests, but currently there is no automation to run them.
Others are test programs require direct user involvement, either providing inputs to direct the tests (e.g. TestFdStreams requires one or more input data files) or checking results for correctness (e.g. TestText prints the results of a variety of operations but doesn't check them for correctness, and TestUniqueId simply prints a number of fingerprints which should be unique). In some cases, both input and checking of results is required (e.g. TestParseImports takes models as inputs and prints the result of parsing their imports). These test programs should be improved to be able to operate with no help from the user (like the first kind of test). It would be fine to allow for the possibility of user input, but it should be optional rather than required.
Finally there are programs which would be better described as debug utilities. (The TestVDirSurrogate and TestShortId programs are two such examples.) These should probably be re-classified as made part of a "debug" result binding.
If all the test programs are improved to require no user input, check their own results, and indicate success or failure with their exit status then we can automate running them. If we separate the debug utilities from the self-checking tests, this will make it easier to automate running them.
Automate Running Unit Tests
If we make the improvements suggested above, then every program in the "tests" sub-binding of a package build result should be a self-checking test program which can be run with no input. We can then write a very simple script to run these test programs. Perhaps something like:
testdir=/tmp/tests.$$ mkdir $testdir vmake -shipfrom tests -shipto $testdir for t in $testdir/*; do $t || exit 1 done rm -rf $testdir
To ensure that these tests get run when making changes, this kind of automation should ideally become a pre-vcheckin trigger. Of course not all packages will have such tests, so we'll need some way to determine whether this should be done before checkin. One way we could do this would be to pick a specific file name for such a shell script to test a package and run it before checkin if it exists.
Improve Integration Testing
Our main method for testing the whole Vesta system is the test-vesta.pl script. While it does a reasonably good job of testing some aspects of the system (building, weeding, cache server fault tolerance), it has some significant limitations which should be addressed:
- It is mostly a single-threaded test. While some work is done in the background, the main process runs a loop of evaluations and weeding. It does not perform multiple evaluations in parallel.
- It is currently a single machine test. If it were to access the servers simultaneously from multiple client hosts is could do a better job of exercising the servers and (if enough client hosts are used) testing conditions of high activity.
- It only tests a single repository and does not exercise important kinds of cross-repository interaction. The repository under test is used as a replication destination, but not as a replication source. Mastership transfer and remote checkout/checkin are not tested.
- Types of operations which occur during real-world use are not exercised:
- Failed or canceled evaluations (e.g. simulating a user pressing control-C)
- The typical checkout-edit-build-edit-build-checkin process (i.e. a series of changes, each recorded in a snapshot and built)
- Updating model imports with vupdate
- A bunch of other more obscure cases aren't tested (e.g. restarting and completing a failed partial weed).
There have been at least two other Vesta testing systems which have been developed. Under Compaq during the initial evaluation of Vesta before the Alpha development group began using it, a multi-client stress test was developed. A different multi-client testing system is currently in use at Intel. Unfortunately, in their current form both of these may contain:
- Potentially sensitive information that can't be publicly released, and
- Company/site/OS/installation-specific idiosyncrasies which make them unsuitable for use for general-purpose testing in a heterogeneous environment.
However, they should both be reviewed for portions which could be incorporated into test-vesta.pl.
Make Integration Testing Setup Easier
Though we probably can't make it entirely automated (especially for testing that uses multiple hosts), we should try to make the setup required to run test-vesta.pl easier. Ideally any developer who wishes to work on Vesta should be able to download the script and use it to test their changes on a host or set of hosts running any combination of OSes and CPU architectures that Vesta supports.