Tutorial: HelloWorld, a first example to get started with rTest¶

The resources for this example are located in the ./tutorial/HelloWorld directory of the rTest folder.

The HelloWorld project¶

To understand how rTest works, you need to put yourself in the shoes of an FME script developer who is trying to check that his project conforms to the specifications.

Let's take the example of the FME project HelloWorld.fmw.

The objective of this project is to write information about a country from a Shapefile data set into a Spatialite file. The processing must be only interested in the country which contains the point whose coordinates are passed in parameter.

For example, if the coordinates passed as parameters are -100.0 (longitude) and 60.0 (latitude), then the result must contain the geometry and attributes of Canada.

Source format: ESRI Shapefile
Destination format: Spatialite
Parameters:
- --xy (longitude and latitude of the point separated by a space)
- --destSpatialiteFile (location of the result)

The image above shows the HelloWorld.fmw project that a developer could create to meet this request.

The produces a result but to validate its good functioning, two things are missing:

Identify test cases
Verify that the workspace produces the correct result for each test case

Approach¶

1. Identification of test cases¶

The identification of test cases should ideally come from the specifications. We can imagine the following test cases:

Point in Switzerland (positive coordinates)
Point in Argentina (negative coordinates)
Coordinates on a border between 2 countries (France and Switzerland)
Coordinates on a triple point (France, Switzerland, Italy)
Point at sea (0 0)
Numerical aberrant coordinates (1000 1000)
Alphanumeric coordinates (1x 45)
Coordinates with bad delimiter (4;45)
Lack of coordinates
…

For each case, it should be checked whether or not the FME processing fails, whether the country found is the correct one and check at least once that the geometry and all attributes have been loaded.

2. Writing the test script¶

The second step is to write a test scenario.

The document ./tutorial/HelloWorld.xml shows the formalization of the first three tests:

Point in a country with positive coordinates (Switzerland)
Point in a country with negative coordinates (Argentina)
Point on a border (Franco-Swiss)

Content¶

The header <scenario> is mandatory and usually fixed.

<scenario xmlns="http://www.veremes.com/rtest" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.veremes.com/rtest http://schemas.veremes.net/rtest/2.0/rtest.xsd">

The elements "project" and "description" are purely descriptive and even optional. They are used in the control reports.

<project>Hello World !</project>
<description>A simple test scenario to understand how rTest for FME does work</description>

The <rtest> elements contain information about the tests to be performed. Each <rtest> element is composed of one or more processes (<process> element) and one or more checks (<check>).

<rtest id="hw.t1" label="Point with positive coordinates">
  <processCollection>
    <process label="Run HelloWorld.fmw with a point in Switzerland">
...
    </process>
  </processCollection>
  <checkCollection>
    <check label="Number of features in the 'country' feature type">
..
    </check>
    <check label="Country name in the first record">
...
    </check>
  </checkCollection>
</rtest>

The processing (<process}) is described essentially by the location of the .fmw project and the parameters to be used to run the project according to the test case, including the xy parameter that defines the coordinates of the point.

<process label="Run HelloWorld.fmw with a point in Switzerland" deletebefore="./result/dbcountry.sqlite">
  <workspace>./HelloWorld.fmw</workspace>
  <parameterCollection>
    <parameter name="xy" value="6.6 46.6"/>
    <parameter name="destSpatialiteFile" value="./result/dbcountry.sqlite"/>
  </parameterCollection>
</process>

Note the presence of the xml attribute "deletebefore" which allows the destination dataset to be deleted before the processing is run. This attribute is not specified in the <process> of the second test, so the second processing will load an additional record into the dbcountry.sqlite file created in the first test.

The checks (<check>) require more information to be defined. The first check calculates the number of entities in the result set (call the featuresCount() test operator) and compares this observed value (comparator "eq" corresponding to equality) to the constant "1" which is the expected value. The second check is to look for the first value of the name field of the country entity type in the result dataset (./result/dbcountry.sqlite). The expected value here is Switzerland.

<checkCollection>
  <check label="Number of features in the 'country' feature type">
    <source type="datafile" format="SPATIALITE" dataset="./result/dbcountry.sqlite" request="featuresCount()" requestparams="country"/>
    <condition comparator="eq" expectedvalue="1"/>
  </check>
  <check label="Country name in the first record">
    <source type="datafile" format="SPATIALITE" dataset="result\dbcountry.sqlite" request="attributeValue()" requestparams="country,name,1"/>
    <condition comparator="eq" expectedvalue="Switzerland"/>
  </check>
</checkCollection>

3. Launching the control¶

A check is the execution of a test scenario. You can start the control in several ways:

From a command line

fme.exe ..\scenarioPlayer.fmw --scenarioxml "./HelloWorld.xml"

By running the HelloWorld.bat file

./HelloWorld.bat

Or simply on Windows from a double click on HelloWorld.bat

Several parameters can be used when calling scenarioPlayer.fmw, these are documented in the section Launching a control with scenarioPlayer.

Warning

FME Desktop 2019 or higher must be installed on your workstation. It may be necessary to specify the path to fme.exe on the command line or in HelloWorld.bat. If you have problems, check the processing log file in the ./rtlog directory or run the processing from the command line to view the messages in the console.

The results of the control¶

After running the control, the ./tutorial directory should contain three additional directories:

./result was created by the FME HelloWorld.fmw project, so it is independent of rTest
./rtreport has been created by scenarioPlayer, it contains the html reports generated during each control
./rtlog was created by scenarioPlayer, it contains the log files generated by scenarioPlayer and by the processes described in the test scenario, here HelloWorld.fmw

The generated report is therefore located in the ./rtreport directory. It has a name with a date mark such as HelloWorld_1627559274861.html, so the history of reports is preserved. The last report is also duplicated and renamed with a fixed value lastReport.html which makes it easier to consult in a browser.

The HTML report¶

The report is an HTML + JavaScript application that contains all the information related to the control that has just been performed:

control (date, project, user, FME version, link to log file...)
test (status)
processing (parameters, status, duration, memory status, link to each log file)
check (status, observed value, expected value...)

Clicking on an item (process or check) provides additional information about the process or check.

Note

The description of the different test status is available on the page Test status, processing and verification of the documentation.

Analysis¶

Analysis of the report shows that the first test (Point with positive coordinates) is satisfactory (status: Successful). The FME processing went well and both checks indicate that the results are well within expectations.

However, there is a problem with the second test (Point with negative coordinates). The processing went well but the second check failed (status: Failed).

When accessing the details of the problem we see that the expected value is "Argentina" while the observed value is "Argentina".

In this case, it is a simple input error in the test scenario. To correct this problem, you must :

Edit HelloWorld.xml
Modify the check:

		<condition comparator="eq" expectedvalue="Argentina"/>

Restart the test scenario execution
Refresh lastReport.html in the browser

The first two tests should be passed.

The third test (Point on a border) has a particular status which is neither "Success", nor "Failed" but "Error". This status is attributed when it is impossible to compare the expected value and the observed value (see Definition of the tests status). In our case we see that the workspace proceeded well but that the checking failed (status: Error)

The message indicates that the expected file ./result/border.sqlite does not exist. To understand what the problem is, it is a good idea to inspect the FME processing log file. This is accessible from the "View log file" link in the process section.

Indeed the processing did not produce any result although it ended correctly.

Hmmm... It seems that there is a problem in the HelloWorld.fmw project, the developer will have to review his copy.

Small hint if you want to fix HelloWorld.fmw : you have to add the test Touches in the transform SpatialFilter, in addition to Within...

Your turn ...¶

To create or modify a test scenario, we recommend using an XML editor such as XMLSpy or an online validation tool such as Truugo.

Warning

Be careful not to lock the files produced by the HelloWorld.fmw processing (dbcountry.sqlite and border.sqlite) by leaving them open in FME DataInspector or another tool, otherwise the file deletion will fail as well as all the control processing.

Add a test to check the value of the attribute "name" and "pop2005" when the point is located at coordinates (0.0 41.1). Expected result: "Spain" and "43397491". Remember to assign a unique identifier to the test.
Check the value of all the attributes of the record (name, pop2005,region, subregion ) in a single check.
Check the value of the geometry.
Treat the case of the point at sea (0,0). In this case you must first specify the expected result: processing failure or write an information in the log file.

Correction tutorial¶

You can download the HelloWorld_corrected.xml file which corrects the tutorial.

Note

If you want to run the .xml patch file, you need to modify the "HelloWorld.bat" file by changing the called scenario.xml.