Call mtc.stop after setverdict(fail), add reasons to most failures and
fail with verdict error for internal errors.
Change-Id: I9b618235939fa41160b9be6677b121963d3ec857
PARALLEL | VERDICTOP will log when the port is dying or when other
components will change to fail. This helped to find a timeout in the
SGSN tests where a function call message timed out.
Change-Id: I770ac964dc37e2752e7d35e493f707b091c739b0
'make clean' as generated by ttcn3_makefilegen removes all *.log files, which
of course cleans out expected-results.log, which should not happen. Since this
is a junit XML file, rename the suffix to .xml.
Change-Id: Ic334f6b758eef865e3a497aa430691a3ae696d25
Compare current test results to the expected results, and exit in error on
discrepancies.
Add compare-result.sh: (trivially) grep junit xml output to determine which
tests passed and which didn't, and compare against an expected-result.log,
another junit file from a previous run. Summarize and determine success.
Include an "xfail" feature: tests that are expected to fail are marked as
"xfail", unexpected failures as "FAIL".
In various subdirs, copy the current jenkins jobs' junit xml outputs as
expected-results.log, so that we will start getting useful output in both
jenkins runs and manual local runs.
In start-testsuite.sh, after running the tests, invoke the results comparison.
Due to the single-line parsing nature, the script so far does not distinguish
between error and failure. I doubt that we actually need to do that though.
Related: OS#3136
Change-Id: I87d62a8be73d73a5eeff61a842e7c27a0066079d
Daniel Willmann
Pau Espin
Alexander Couzens
Harald Welte