Documentation Center

  • 評価版
  • 製品アップデート

prog::test

Automatic comparing of calculation results

Use only in the MuPAD Notebook Interface.

This functionality does not run in MATLAB.

Syntax

prog::test(stmt, res | TrapError = errnr, <Timeout = sec>, <message>, options)
prog::test(stmt)

Description

prog::test works in two different modes: interactive and inside of test files.

In interactive mode a single call of prog::test can be used to compare two MuPAD® statements.

The call prog::test(stmt, res) evaluates both arguments stmt and res. When the evaluation leads to exactly the same MuPAD object and no Enhancement was requested, nothing is printed and prog::test returns the void object null().

If the results are different, the test fails and a message is printed.

The additional arguments are described in the following part for using prog::test in test files.

Another mode is using prog::test inside of test files. A test file must start with prog::testinit . This function initializes the test file. Then you can write several tests using prog::test. The last statement in a test file must be prog::testexit(). You also can specify the name of the tested procedure by using print(Unquoted, "testname") after prog::testinit. This name does not affect the tested procedure itself. It only appears in the test reports generated by your test script.

The tests can be arbitrary MuPAD statements and prog::test statements. However, most of the functionality should be executed as argument of prog::test. Only initialization of variables should be performed outside of prog::test statements in a test file, because prog::test traps every error (with the function traperror) and prints a specific error message.

    Note:   If an error occurs outside of prog::test, reading of the test file is interrupted.

If no error occurs (as should be the default case), the results are compared and a message is printed, if they are different.

Timing information can be collected and compared that consider only the evaluation time of the first argument stmt of prog::test (see prog::testinit).

If a test fails, for example, the two first arguments of prog::test lead to different MuPAD objects, or if an enhancement request was given, prog::test prints a message. This message lists the following pieces of information:

  1. The first line starts with the Error in test string and contains the name and a sequence number of the individual test.

  2. The next three lines contain the input, the expected result, and the result actually observed.

  3. For each of the options Priority, Enhancement, Message, Developers, and BugId, if the option has been set, a corresponding line will be printed. Note that Message can be set by simply providing a message string.

This information is followed by an empty blank line.

If only one argument is given, the argument is evaluated and compared with TRUE, i.e., prog::test(ex) is equivalent to prog::test(ex, TRUE).

When a test is initialized with prog::testinit and ended by prog::testexit, a short message is printed with the following format:

Info: 20 test, 1 error, runtime factor 1.7 (expected 2.0)

The message contains the number of all tests performed (20), the number of errors (1), and two time factors: The first time factor is the based on the actual time of the test and the second time factor is the expected value given by prog::testinit.

Examples

Example 1

prog::test can be called interactively:

prog::test(1 + 1, 2):
prog::test(is(2 > 1)):
prog::test(sin(PI), 0, "check sin"):

These tests checked all right. In the next tests wrong results are tested against, to demonstrate the messages given by prog::test:

prog::test(1 + 2, 2):
Error in test 4
Input: 1 + 2
Expected:  2
Got:       3
Near line: 1


prog::test(is(x > 1)):
Error in test 5
Input: is(1 < x)
Expected:  TRUE
Got:       UNKNOWN
Near line: 1


prog::test(sin(PI), PI, "check sin"):
Error in test 6
Input: sin(PI)
Expected:  PI
Got:       0
Message: check sin
Near line: 1


Example 2

A test file must contain calls to prog::testinit and prog::testexit. In the following file, we test a function defined in the same file, which is rather uncommon, obviously.

// test file "test.tst"
test:= (a, b) -> a^2 + 2*b^2 - a*b:
prog::testinit("test", 0.1):
print(Unquoted, "testname"):
prog::test(test(1, 4), 29, Message = "my first example"):
prog::test(test(3, -2), 24, "the second example"):
prog::test(error("test"), TrapError = 1028):
prog::testexit():

The first statement is only a comment. The second line contains an initialization of a test procedure called test. Then the test is initialized with prog::testinit.

After that three tests are performed: The first test is right, the second expected result is wrong, and the third test produces an error, but the expected result is this error, the error number returned by traperror is 1028 (user call of error).

The whole test takes nearly no time:

read("test.tst")
Info: memory limit is 256 MB
Error in test 2
Input: test(3, -2)
Expected:  24
Got:       23
Message: the second example
Near line: 4

Info: time used outside of 'prog::test' takes 100%
Info: 3 tests, 1 error, runtime factor  0.0  (expected  0.1)
Info: CPU time:  1.1 s
Info: Memory allocation 9026800 bytes [prog::testexit]

Example 3

Most of the options accepted by prog::test are more or less directly placed in the output:

prog::test(1+1, 1, Baseline, Message(2)="well ...",
  Priority=Low, BugId="123-456")
Baseline Error in test 7
Input: 1 + 1
Expected:  1
Got:       2
Priority: Low
Message: well ...
BugId: 123-456
Near line: 2

Example 4

To test that a certain call does not take longer than a specified number of seconds, use the option Timeout:

prog::test(prog::wait(5.0), null(), Timeout = 2)

Example 5

In most cases, the actual and the expected result are simply compared for equality. Sometimes, however, this is not desirable, especially for floating-point results:

prog::test(float(PI), 3.1415926535897932385)
Error in test 8
Input: float(PI)
Expected:  3.141592654
Got:       3.141592654
Near line: 1

The problem here is that there are many floating-point values which are not identical, yet are displayed as such (unless you increase DIGITS far enough to see the difference). Using the option Method, you can provide a function to compare the values:

prog::test(float(PI), 3.1415926535897932385, Method = `~=`)

Example 6

When implementing symbolic algorithms, there are often multiple correct and acceptable answers. In some cases, getting any of a certain set of solutions is fine. In these cases, using Method = _in is a reasonable way of writing tests (_in is the functional form of the in operator):

prog::test(int(ln(ln(a*x)^(1/2)), x),
  {
    x*ln(ln(a*x)^(1/2)) - Li(a*x)/(2*a),
    x*ln(ln(a*x))/2 - Li(a*x)/(2*a)
  },
  Method = _in,
  Timeout = 20)

Sometimes, however, while multiple results are acceptable, you are actually targeting for one particular output. For these cases, you can use Enhancement to set the golden goal:

prog::test((x^2+2*x+1)/(x+1),
  (x^2+2*x+1)/(x+1),
  Enhancement = x+1)
Enhancement request: 11
Input: (x^2 + 2*x + 1)/(x + 1)
Got:       (x^2 + 2*x + 1)/(x + 1)
Requested: x + 1
Near line: 3

If the enhancement request ever is fulfilled, the output changes:

prog::test(normal((x^2+2*x+1)/(x+1)),
  (x^2+2*x+1)/(x+1),
  Enhancement = x+1)
Enhancement done: 12
Input: normal((x^2 + 2*x + 1)/(x + 1))
Got:       x + 1
Requested: x + 1
Near line: 3

Note that a test with an enhancement request is, first and foremost, still an ordinary test and behaves as such:

prog::test((x^2+x+1)/(x+1),
  (x^2+2*x+1)/(x+1),
  Enhancement = x+1)
Error in test 13
Input: (x^2 + x + 1)/(x + 1)
Expected:  (x^2 + 2*x + 1)/(x + 1)
Got:       (x^2 + x + 1)/(x + 1)
Near line: 3

Example 7

Certain calls are expected to give warnings to the user:

numeric::quadrature(sin(1/x), x=0..1)
Warning: Precision goal is not achieved after 10000 function calls. Increase 'MaxCalls' and try again for a more accurate result. [numeric::quadrature]

Placing such calls into an unadorned call to prog::test causes the test to fail, because by default, prog::test expects the calls not to emit any warnings:

prog::test(
  bool(numeric::quadrature(sin(1/x), x=0..1) < 1),
  TRUE)
Error in test 14
Input: bool(numeric::quadrature(sin(1/x), x = 0..1) < 1)
Expected:  TRUE
Got:       TRUE
Expected warnings: []
Got warnings:      [message("symbolic:numeric:PrecisionGoalNotReached", 10000)]
Near line: 3
Used Time: 0.615 (0.946*prog::ntime())

To check for this warning, add ExpectedWarnings with the list shown above:

prog::test(
  bool(numeric::quadrature(sin(1/x), x=0..1) < 1), TRUE,
  ExpectedWarnings = ["Precision goal is not achieved after 10000" .
  " function calls. Increase 'MaxCalls' and try again for a more accurate result."])

It is possible to abbreviate the excepted warnings (and make tests robust against changes in places where they are expected) by using regular expression matching for the expected warnings (see strmatch for details on pattern matching):

prog::test(
  bool(numeric::quadrature(sin(1/x), x=0..1) < 1), TRUE,
  ExpectedWarnings = ["Precision goal is not achieved .* for a more accurate result."])

Parameters

stmt

A MuPAD statement to test

res

A MuPAD expression or statement that determines the expected result.

message

A message (a string) that is displayed if the test fails – see option Message below

Options

TrapError

Option, specified as TrapError = errnr

Expect the test to throw an error. errnr must be the integer expected from the call traperror(stmt) or a list of an integer and a string, as returned by getlasterror().

Method

Option, specified as Method = comp

A method used to compare the actual and the expected result. Will be called with both expressions and must return TRUE or FALSE.

Timeout

Option, specified as Timeout = sec

A timeout for the evaluation of the tests. Both the actual and the expected result are evaluated with this time limit. If the computation takes too long, prog::test behaves as if the command had resulted in a timeout error (error number 1320).

Message

Option, specified as Message = message or Message(res1) = message

Append a message (a string) to the output of prog::test. If res1 is given, the message is given if the result of evaluating stmt is res1.

Baseline

Mark this test as failing in some sort of "baseline," to differentiate new bugs (stemming from new code developments, regression failures) from bugs already present in some specific earlier version. This affects the output of prog::test.

Enhancement

Option, specified as Enhancement = res1

Request some other output than the one currently tested for. Semantically, a call of the form prog::test(inp, out, Enhancement = out2) means "check that the call inp results in the same thing as the call out, but note that we'd actually prefer to see out2."

ExpectedWarnings

Option, specified as ExpectedWarnings = list

Gives a list of warnings the call should emit, as strings. Not emitting these warnings, or additional ones, is considered an error.

High, Low, Medium, Priority

Option, specified as Priority = Low | Medium | High

Denote the importance of this test. This will usually be a very subjective question and affects the output of prog::test only, to allow tools parsing the output displaying the problems of higher priority more prominently.

Developers

Option, specified as Developers = devnames

A string included in the output of prog::test, denoting the developers deemed responsible for the code tested. This is intended for post-processing tools.

BugId

Option, specified as BugId = bugid

Again, for the output of prog::test, include a reference to some bug tracking system. bugid can be any MuPAD object.

Return Values

prog::test returns the void object null().

See Also

MuPAD Functions

Was this topic helpful?