Up

Testing

Testing frameworks


Need to support predefined squences of steps. Each step is a command (function/program).

It can be tested with various arguments; the spec of these is not part of the step def.

So: need to test each step with a variety of arg values; and need to test sequences with steps missing and out of order.

For each step, there is an argument space, and the valid arguments to the step constitute some sub-space.

Similarly, for each sequence, there is some number (possibly just one) of valid sequences, and some number of other invalid sequences. Again, a valid sub-space.

So a language for defining tests should be a constraint language. Should be able to declare discrete and continuous dimensions (for arguments).

In general, since the spaces themselves are not unbounded, need to declare both the space of all values, and the subspace of valid values.

However, certain spaces are common enough that they can be predefined. E.g. "string". Actually, these would simply be the universes for the C types of the parameters of the API functions. For command-line arguments, we must allow "string" as the universe.

The purpose of the "valid value" constraint definition is merely to indicate the range (space) of values for which we expect the test to pass.

Hmmm. we need to be able to define disjunctive subspaces; because it may be that both

  foo X
  bar 1
and
  foo Z
  bar 3
should pass, but that neither
  foo Z
  bar 1
nor
  foo X
  bar 3
should pass. So we can't simply define that
  foo [XZ]
  bar [13]
is the space of validity.

Another issue: need a way to define points (like checkpoints) at which the environment/context MUST be re-initialized, and also, points at which it MUST NOT be re-initialized.

Model the Use Cases.

Rather than (or in addition to) specific complete sequences, define sequence rules, i.e. A -> B. Build the graph from those. Construct valid (and invalid) sequences from that/those.

Petri Nets!

...which are a specialization of state graphs, and so can be modeled with state transition tables. Loops should be detected and extracted as sub-graphs (subroutines), so as to permit controlled-depth reiteration.


Ant

a cool replacement for Make, written in Java and using XML config files.