Building an mtestengine
Creating an mtestengine is as simple as the following code suggests:
mte = MTestRunner;
Similar to a struct we can now visualize the properties of this object:
mte
mte =
MTestRunner handle
Properties:
MainDir: 'e:\Temp\tp1c071d7c_df0d_4cef_af63_f7023d9bc749'
Recursive: 1
Publish: 0
Verbose: 0
IncludeCoverage: 1
TargetDir: 'e:\Temp\tp1c071d7c_df0d_4cef_af63_f7023d9bc749'
TestID: '_test'
Exclusions: {'.svn' '_tutorial' '_exclude'}
Template: 'default'
Tests: [1x1 MTest]
WrongTestDefs: {}
FunctionsRun: {}
It directly shows the default options for all fields. Of course we would like to set some of the properties. This can be done either when creating the object itself (with property value pairs) or afterwards in the same way as altering fields of a struct:
mte.TargetDir = fullfile(tempdir,'htmltest'); mte.MainDir = fileparts(which('MTestRunner')); mte.Verbose = true; mte.Template = 'oet';
or:
mte = MTestRunner(... 'TargetDir',fullfile(tempdir,'htmltest'),... 'MainDir',fileparts(which('mte_simple_test')),... 'Verbose',true,... 'Template','oet');
Properties (fieldnames) of the MTestRunner
The MTestRunner has a couple of properties that can be used to define the behaviour of the MTestRunner when running and publishing. The following table lists the properties of an MTestRunner object.
| Property Name | Class |
Description |
|---|---|---|
| TargetDir | char Default=cd |
Pathname of the directory where output (html) files must be generated. |
| MainDir | char Default=cd |
Root directory of the toolbox that must be analysed and tested. |
| Recursive | logical Default=true |
Flag to determine whether the mtestengine only lists tests in the maindir or also in all subdirs of the maindir. |
| Publish | logical Default=false |
Determines whether the publishable parts of a test get published. |
| Verbose | logical Default=false |
Flag that determines whether intermediate information must be printed in the command window. |
| IncludeCoverage | logical Default=true |
Flag to turn on / off the profile function during execution of the tests to obtain function coverage information. |
| TestID | cell Default={'_test'} |
The character arrays included in this cell determine which functionnames are identified as testdefinition files. |
| Exclusions | cell Default={'.svn','_tutorial'} |
Any function or pathname that includes one of the character arrays filling this cell is excluded from the list of testsdefinitions. |
| Template | char Default=default |
Name of the template that must be used to print the testresults of the toolbox. (OpenEarthTools has its own template named: "oet". Use this template when publising tests from OpenEarthTools). |
| Tests | mtest Default=[] |
mtest object with the loaded testdefinitions. The mtestengine takes this list whenever the run command is given. |
| WrongTestDefs | cell Default={} |
Files identified as tests according to the testid and exclusion keywords, but failed to load in an mtest object. This is probably due to a testdefinition that is not according to the convention used for the mtest toolbox. |
| FunctionsRun | mtestfunction Default=[] |
Array of mtestfunction objects with information on the functions that were executed during the tests (including coverage information). |
Setting properties is already adressed in the previous section.
Specifying tests to run
After creation of an mtestengine we can see that it does not contain any testdefinitions yet. The tests contains an empty test.
mte.Tests
ans =
MTest handle
Properties:
Name: []
FileName: []
FilePath: []
FunctionHeader: ''
FunctionName: ''
H1Line: []
Description: {}
Author: []
SeeAlso: {}
Publish: 1
MaxWidth: 600
MaxHeight: 600
StyleSheet: ''
Ignore: 0
IgnoreMessage: ''
Category: 'Unit'
TestResult: 0
Time: 0
Date: NaN
ProfilerInfo: []
FunctionCalls: []
StackTrace: []
Verbose: 1
manually add tests
Specifying tests can be done in two different ways. Of course we can add tests manually to the field:
mte.Tests(1) = MTest('mte_simple_test');
search for tests
The catalogueTests method (or function) searches for all testdefinitions that match the following requirements:
- Tests should be in the maindir (or one of the subdirs if mte.recursive = true)
- The filename should include one of the strings specified in the field testid of the mtestobject
- The filename should not include one of the strings specified in the field exclusion of the mtestobject
- mfiles that match the above mentionned criteria are converted to mtest objects. they must therefore match the layout criteria described in the tutorial on testdefinitions
By default testdefinition files include the following string in their filename:
mte.TestID
ans = _test
but we exclude files with one of these strings in the filename
mte.Exclusions
ans =
'.svn' '_tutorial' '_exclude'
After automatically searching for testdefinitions and adding them to the testengine ...
cataloguetests(mte);
Collecting tests
we can see that the content of the tests field has changed. This is of course due to the fact that we just collected all tests that match the description as stated above:
mte
mte =
MTestRunner handle
Properties:
MainDir: 'F:\OpenEarthTools\matlab\maintenance\1_automated_Testing\examples'
Recursive: 1
Publish: 0
Verbose: 1
IncludeCoverage: 1
TargetDir: 'e:\Temp\htmltest'
TestID: '_test'
Exclusions: {'.svn' '_tutorial' '_exclude'}
Template: 'oet'
Tests: [1x11 MTest]
WrongTestDefs: {'F:\OpenEarthTools\matlab\maintenance\1_automated_Testing\examples\mte_wrongdefinition_test.m' [1x1 MException]}
FunctionsRun: {}
Run all tests
To run all tests specified in the tests field of the mtestengine object there is a very simple function:
run(mte);
## start running tests ##
mte_wrongdefinition_test Could not be interpreted as a valid testdefinition.
1. mt_definitionwithdescription_newstyle_test (mt_definitionwithdescription_newstyle_test)
Test started: mt_definitionwithdescription_newstyle_test
Test finished: mt_definitionwithdescription_newstyle_test
2. name of the test (mte_concepttest_test)
Test started: name of the test
Error occurred: 1 is not 2
Test finished: name of the test
3. testname (mte_fulldefinition_test)
Test started: testname
Test finished: testname
4. mte_givecategory_test (mte_givecategory_test)
Test started: mte_givecategory_test
Test finished: mte_givecategory_test
5. name of the test (mte_givename_test)
Test started: name of the test
Test finished: name of the test
6. mte_headeronly_test (mte_headeronly_test)
Test started: mte_headeronly_test
Test finished: mte_headeronly_test
7. Ignore test (mte_ignore_test)
Test started: Ignore test
Ignored: ignore test example
Test finished: Ignore test
8. mte_simple_test (mte_simple_test)
Test started: mte_simple_test
Test finished: mte_simple_test
9. mte_simplest_test (mte_simplest_test)
Test started: mte_simplest_test
Test finished: mte_simplest_test
10. mte_testpublish_test (mte_testpublish_test)
Test started: mte_testpublish_test
Test finished: mte_testpublish_test
11. mte_versiononly_test (mte_versiononly_test)
Test started: mte_versiononly_test
Test finished: mte_versiononly_test
Each mtest object under the field "tests" contains a field that stores the testresult:
mte.Tests(2).TestResult
ans =
0
Run all tests and publish documentation and results
The run function runs all tests and publishes the description and vizualization of results as specified in the testdefinition files. TODO: It also publishes the coverage of the tests (which percentages (and lines) of which functions did we actually test). More information on this function can be found in its help documentation:
help MTestRunner.run
RUN Runs all tests (that are in the "Test" property)
This function executes the run function of all mtest objects in the Tests property.
TODO:
- include input argument to specify test numbers (instead of just all tests).
Syntax:
outobj = obj.run;
outobj = run(obj);
Input:
obj - an MTestRunner object.
Output:
outobj - The same MTestRunner object as the input argument obj. It is not
necessary to have an output, since the MTestRunner is of type handle.
This automatically adjusts all copies of the object that are in the
matlab memory. The one that is in the base workspace is therefore
automatically updated and does not need to be output of the function.
See also MTestRunner MTestRunner.MTestRunner MTestRunner.run MTest MTestFactory TeamCity
Publication templates
Publication of the mtestengine results is done acoording to a predefined format. This format is determined by the choice for a template. It is of course also possible to write your own template. At this moment there is not yet a tutorial on how to write such a template. The (sometimes hidden) functions of the MTestRunner object give a lot of information already.