2017-03-28 12:30:28 +00:00
- non-existing suite dir
2020-05-11 16:30:58 +00:00
cnf -: DBG: Found main configuration file in [PATH]/selftest/suite_test/paths.conf which is [PATH]/selftest/suite_test/paths.conf
cnf -: DBG: MAIN CONFIG:
{'default_suites_conf_path': '[PATH]/selftest/suite_test/default-suites.conf',
'defaults_conf_path': '[PATH]/selftest/suite_test/defaults.conf',
'resource_conf_path': '[PATH]/selftest/suite_test/resources.conf',
2020-05-25 11:26:41 +00:00
'scenarios_dir': ['[PATH]/selftest/suite_test/scenarios'],
2020-05-11 16:30:58 +00:00
'state_dir': '[PATH]/selftest/suite_test/test_work/state_dir',
2020-05-25 11:26:41 +00:00
'suites_dir': ['[PATH]/selftest/suite_test/suitedirA',
'[PATH]/selftest/suite_test/suitedirB'],
2020-05-12 13:06:55 +00:00
'trial_dir': '[PATH]/selftest/suite_test/trial'}
2020-05-25 11:26:41 +00:00
--- -: ERR: RuntimeError: Suite not found: 'does_not_exist' in [[PATH]/selftest/suite_test/suitedirA, [PATH]/selftest/suite_test/suitedirB]
2017-03-28 12:30:28 +00:00
- no suite.conf
2017-06-06 17:44:32 +00:00
cnf empty_dir: DBG: reading suite.conf
2020-05-25 11:26:41 +00:00
cnf [PATH]/selftest/suite_test/suitedirA/empty_dir/suite.conf: ERR: FileNotFoundError: [Errno 2] No such file or directory: '[PATH]/selftest/suite_test/suitedirA/empty_dir/suite.conf' [empty_dir↪[PATH]/selftest/suite_test/suitedirA/empty_dir/suite.conf]
2017-03-28 12:30:28 +00:00
- valid suite dir
2017-06-06 17:44:32 +00:00
cnf test_suite: DBG: reading suite.conf
2020-06-12 15:54:55 +00:00
config:
suite:
test_suite:
test_timeout:
timeout: '1'
2017-03-28 12:30:28 +00:00
resources:
bts:
2017-09-14 13:55:49 +00:00
- label: sysmoCell 5000
times: '2'
2017-03-28 12:30:28 +00:00
- times: '1'
2017-09-14 13:55:49 +00:00
type: sysmo
2017-05-18 16:35:32 +00:00
ip_address:
- times: '1'
2017-03-28 12:30:28 +00:00
modem:
- times: '2'
- run hello world test
2020-06-12 15:13:26 +00:00
tst test_suite: DBG: {combining='config'}
2020-06-12 15:54:55 +00:00
tst {combining_scenarios='config'}: DBG: {definition_conf={suite={test_suite={test_timeout={timeout='1'}}}}} [test_suite↪{combining_scenarios='config'}]
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
---------------------------------------------------------------------
trial test_suite
---------------------------------------------------------------------
2017-05-14 01:37:13 +00:00
tst test_suite: reserving resources in [PATH]/selftest/suite_test/test_work/state_dir ...
2017-06-06 17:44:32 +00:00
tst test_suite: DBG: {combining='resources'}
2017-09-14 13:55:49 +00:00
tst {combining_scenarios='resources'}: DBG: {definition_conf={bts=[{'label': 'sysmoCell 5000'}, {'label': 'sysmoCell 5000'}, {'type': 'sysmo'}], ip_address=[{}], modem=[{}, {}]}} [test_suite↪{combining_scenarios='resources'}]
2018-08-21 12:58:29 +00:00
tst test_suite: DBG: {combining='modifiers'}
tst {combining_scenarios='modifiers'}: DBG: {definition_conf={}} [test_suite↪{combining_scenarios='modifiers'}]
2020-11-09 20:17:51 +00:00
tst test_suite: Verifying 3 x bts (candidates: 6)
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
tst test_suite: Verifying 1 x ip_address (candidates: 3)
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
tst test_suite: Verifying 2 x modem (candidates: 16)
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
tst test_suite: DBG: FileWatch: scheduling watch for directory [PATH]/selftest/suite_test/test_work/state_dir
2017-09-10 14:33:10 +00:00
tst test_suite: Reserving 3 x bts (candidates: 6)
2018-08-27 15:07:41 +00:00
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
2017-09-14 13:55:49 +00:00
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
2018-08-27 15:07:41 +00:00
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
2017-09-14 13:55:49 +00:00
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
2018-08-27 15:07:41 +00:00
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
2017-09-14 13:55:49 +00:00
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
2018-08-27 15:07:41 +00:00
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
2017-09-14 13:55:49 +00:00
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
2017-04-12 00:42:02 +00:00
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
2017-06-06 17:44:32 +00:00
tst test_suite: Reserving 1 x ip_address (candidates: 3)
2017-05-18 16:35:32 +00:00
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
2017-06-06 17:44:32 +00:00
tst test_suite: Reserving 2 x modem (candidates: 16)
2017-05-06 23:16:07 +00:00
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
2017-04-12 00:42:02 +00:00
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
2020-11-09 20:17:51 +00:00
tst test_suite: DBG: FileWatch: unscheduling watch <ObservedWatch: path=[PATH]/selftest/suite_test/test_work/state_dir, is_recursive=False>
2020-11-28 09:06:14 +00:00
tst test_suite: RESERVED RESOURCES for test_suite:
bts
sysmoCell 5000
sysmoCell 5000
sysmoBTS 1002
ip_address
10.42.42.1
modem
m7801
m7802
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
----------------------------------------------
trial test_suite hello_world.py
----------------------------------------------
2017-05-07 01:02:23 +00:00
tst hello_world.py:[LINENR]: hello world [test_suite↪hello_world.py:[LINENR]]
fix: line nr in test name in wrong places
test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.
Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.
Override src() in Test, not name().
Use src() in backtraces.
The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
I am 'test_suite' / 'hello_world.py:23'
becomes
I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)
Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
2020-12-04 16:25:23 +00:00
tst hello_world.py:[LINENR]: I am 'test_suite' / 'hello_world.py' [test_suite↪hello_world.py:[LINENR]]
do not redirect_stdout in test scripts
In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.
Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().
The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.
As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.
I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.
Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
2020-12-01 02:51:27 +00:00
tst hello_world.py:[LINENR]: one
two
three [test_suite↪hello_world.py:[LINENR]]
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
tst hello_world.py:[LINENR] Test passed (N.N sec) [test_suite↪hello_world.py]
---------------------------------------------------------------------
trial test_suite PASS
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
PASS: test_suite (pass: 1, skip: 7)
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
pass: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
skip: test_error.py
skip: test_fail.py
skip: test_fail_raise.py
2020-05-06 19:11:02 +00:00
skip: test_suite_params.py
2020-06-12 15:54:55 +00:00
skip: test_timeout.py
2017-03-28 12:30:28 +00:00
- a test with an error
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
---------------------------------------------------------------------
trial test_suite
---------------------------------------------------------------------
----------------------------------------------
trial test_suite test_error.py
----------------------------------------------
fix: line nr in test name in wrong places
test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.
Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.
Override src() in Test, not name().
Use src() in backtraces.
The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
I am 'test_suite' / 'hello_world.py:23'
becomes
I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)
Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
2020-12-04 16:25:23 +00:00
tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]
fix and refactor logging: drop 'with', simplify
With the recent fix of the junit report related issues, another issue arose:
the 'with log.Origin' was changed to disallow __enter__ing an object twice to
fix problems, now still code would fail because it tries to do 'with' on the
same object twice. The only reason is to ensure that logging is associated with
a given object. Instead of complicating even more, implement differently.
Refactor logging to simplify use: drop the 'with Origin' style completely, and
instead use the python stack to determine which objects are created by which,
and which object to associate a log statement with.
The new way: we rely on the convention that each class instance has a local
'self' referencing the object instance. If we need to find an origin as a new
object's parent, or to associate a log message with, we traverse each stack
frame, fetching the first local 'self' object that is a log.Origin class
instance.
How to use:
Simply call log.log() anywhere, and it finds an Origin object to log for, from
the stack. Alternatively call self.log() for any Origin() object to skip the
lookup.
Create classes as child class of log.Origin and make sure to call
super().__init__(category, name). This constructor will magically find a parent
Origin on the stack.
When an exception happens, we first escalate the exception up through call
scopes to where ever it is handled by log.log_exn(). This then finds an Origin
object in the traceback's stack frames, no need to nest in 'with' scopes.
Hence the 'with log.Origin' now "happens implicitly", we can write pure natural
python code, no more hassles with scope ordering.
Furthermore, any frame can place additional logging information in a frame by
calling log.ctx(). This is automatically inserted in the ancestry associated
with a log statement / exception.
Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
2017-06-09 23:18:27 +00:00
tst test_error.py:[LINENR]: ERR: AssertionError: test_error.py:[LINENR]: assert False [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]
2017-11-09 13:26:35 +00:00
tst test_error.py:[LINENR]: Test FAILED (N.N sec) [test_suite↪test_error.py:[LINENR]] [test.py:[LINENR]]
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
---------------------------------------------------------------------
trial test_suite FAIL
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
FAIL: test_suite (fail: 1, skip: 7)
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
skip: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
fix and refactor logging: drop 'with', simplify
With the recent fix of the junit report related issues, another issue arose:
the 'with log.Origin' was changed to disallow __enter__ing an object twice to
fix problems, now still code would fail because it tries to do 'with' on the
same object twice. The only reason is to ensure that logging is associated with
a given object. Instead of complicating even more, implement differently.
Refactor logging to simplify use: drop the 'with Origin' style completely, and
instead use the python stack to determine which objects are created by which,
and which object to associate a log statement with.
The new way: we rely on the convention that each class instance has a local
'self' referencing the object instance. If we need to find an origin as a new
object's parent, or to associate a log message with, we traverse each stack
frame, fetching the first local 'self' object that is a log.Origin class
instance.
How to use:
Simply call log.log() anywhere, and it finds an Origin object to log for, from
the stack. Alternatively call self.log() for any Origin() object to skip the
lookup.
Create classes as child class of log.Origin and make sure to call
super().__init__(category, name). This constructor will magically find a parent
Origin on the stack.
When an exception happens, we first escalate the exception up through call
scopes to where ever it is handled by log.log_exn(). This then finds an Origin
object in the traceback's stack frames, no need to nest in 'with' scopes.
Hence the 'with log.Origin' now "happens implicitly", we can write pure natural
python code, no more hassles with scope ordering.
Furthermore, any frame can place additional logging information in a frame by
calling log.ctx(). This is automatically inserted in the ancestry associated
with a log statement / exception.
Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
2017-06-09 23:18:27 +00:00
FAIL: test_error.py (N.N sec) AssertionError: test_error.py:[LINENR]: assert False
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
skip: test_fail.py
skip: test_fail_raise.py
2020-05-06 19:11:02 +00:00
skip: test_suite_params.py
2020-06-12 15:54:55 +00:00
skip: test_timeout.py
2017-05-15 16:24:35 +00:00
- a test with a failure
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
---------------------------------------------------------------------
trial test_suite
---------------------------------------------------------------------
----------------------------------------------
trial test_suite test_fail.py
----------------------------------------------
fix: line nr in test name in wrong places
test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.
Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.
Override src() in Test, not name().
Use src() in backtraces.
The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
I am 'test_suite' / 'hello_world.py:23'
becomes
I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)
Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
2020-12-04 16:25:23 +00:00
tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
fix and refactor logging: drop 'with', simplify
With the recent fix of the junit report related issues, another issue arose:
the 'with log.Origin' was changed to disallow __enter__ing an object twice to
fix problems, now still code would fail because it tries to do 'with' on the
same object twice. The only reason is to ensure that logging is associated with
a given object. Instead of complicating even more, implement differently.
Refactor logging to simplify use: drop the 'with Origin' style completely, and
instead use the python stack to determine which objects are created by which,
and which object to associate a log statement with.
The new way: we rely on the convention that each class instance has a local
'self' referencing the object instance. If we need to find an origin as a new
object's parent, or to associate a log message with, we traverse each stack
frame, fetching the first local 'self' object that is a log.Origin class
instance.
How to use:
Simply call log.log() anywhere, and it finds an Origin object to log for, from
the stack. Alternatively call self.log() for any Origin() object to skip the
lookup.
Create classes as child class of log.Origin and make sure to call
super().__init__(category, name). This constructor will magically find a parent
Origin on the stack.
When an exception happens, we first escalate the exception up through call
scopes to where ever it is handled by log.log_exn(). This then finds an Origin
object in the traceback's stack frames, no need to nest in 'with' scopes.
Hence the 'with log.Origin' now "happens implicitly", we can write pure natural
python code, no more hassles with scope ordering.
Furthermore, any frame can place additional logging information in a frame by
calling log.ctx(). This is automatically inserted in the ancestry associated
with a log statement / exception.
Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
2017-06-09 23:18:27 +00:00
tst test_fail.py:[LINENR]: ERR: EpicFail: This failure is expected [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
2017-11-09 13:26:35 +00:00
tst test_fail.py:[LINENR]: Test FAILED (N.N sec) [test_suite↪test_fail.py:[LINENR]] [test.py:[LINENR]]
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
---------------------------------------------------------------------
trial test_suite FAIL
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
FAIL: test_suite (fail: 1, skip: 7)
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
skip: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
skip: test_error.py (N.N sec)
FAIL: test_fail.py (N.N sec) EpicFail: This failure is expected
skip: test_fail_raise.py
2020-05-06 19:11:02 +00:00
skip: test_suite_params.py
2020-06-12 15:54:55 +00:00
skip: test_timeout.py
2017-05-15 16:24:35 +00:00
- a test with a raised failure
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
---------------------------------------------------------------------
trial test_suite
---------------------------------------------------------------------
----------------------------------------------
trial test_suite test_fail_raise.py
----------------------------------------------
fix and refactor logging: drop 'with', simplify
With the recent fix of the junit report related issues, another issue arose:
the 'with log.Origin' was changed to disallow __enter__ing an object twice to
fix problems, now still code would fail because it tries to do 'with' on the
same object twice. The only reason is to ensure that logging is associated with
a given object. Instead of complicating even more, implement differently.
Refactor logging to simplify use: drop the 'with Origin' style completely, and
instead use the python stack to determine which objects are created by which,
and which object to associate a log statement with.
The new way: we rely on the convention that each class instance has a local
'self' referencing the object instance. If we need to find an origin as a new
object's parent, or to associate a log message with, we traverse each stack
frame, fetching the first local 'self' object that is a log.Origin class
instance.
How to use:
Simply call log.log() anywhere, and it finds an Origin object to log for, from
the stack. Alternatively call self.log() for any Origin() object to skip the
lookup.
Create classes as child class of log.Origin and make sure to call
super().__init__(category, name). This constructor will magically find a parent
Origin on the stack.
When an exception happens, we first escalate the exception up through call
scopes to where ever it is handled by log.log_exn(). This then finds an Origin
object in the traceback's stack frames, no need to nest in 'with' scopes.
Hence the 'with log.Origin' now "happens implicitly", we can write pure natural
python code, no more hassles with scope ordering.
Furthermore, any frame can place additional logging information in a frame by
calling log.ctx(). This is automatically inserted in the ancestry associated
with a log statement / exception.
Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
2017-06-09 23:18:27 +00:00
tst test_fail_raise.py:[LINENR]: ERR: ExpectedFail: This failure is expected [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]: raise ExpectedFail('This failure is expected')]
2017-11-09 13:26:35 +00:00
tst test_fail_raise.py:[LINENR]: Test FAILED (N.N sec) [test_suite↪test_fail_raise.py:[LINENR]] [test.py:[LINENR]]
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
---------------------------------------------------------------------
trial test_suite FAIL
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
FAIL: test_suite (fail: 1, skip: 7)
refactor: fix error handling; fix log.Origin; only one trial
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-06 17:41:17 +00:00
skip: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
skip: test_error.py (N.N sec)
skip: test_fail.py (N.N sec)
fix and refactor logging: drop 'with', simplify
With the recent fix of the junit report related issues, another issue arose:
the 'with log.Origin' was changed to disallow __enter__ing an object twice to
fix problems, now still code would fail because it tries to do 'with' on the
same object twice. The only reason is to ensure that logging is associated with
a given object. Instead of complicating even more, implement differently.
Refactor logging to simplify use: drop the 'with Origin' style completely, and
instead use the python stack to determine which objects are created by which,
and which object to associate a log statement with.
The new way: we rely on the convention that each class instance has a local
'self' referencing the object instance. If we need to find an origin as a new
object's parent, or to associate a log message with, we traverse each stack
frame, fetching the first local 'self' object that is a log.Origin class
instance.
How to use:
Simply call log.log() anywhere, and it finds an Origin object to log for, from
the stack. Alternatively call self.log() for any Origin() object to skip the
lookup.
Create classes as child class of log.Origin and make sure to call
super().__init__(category, name). This constructor will magically find a parent
Origin on the stack.
When an exception happens, we first escalate the exception up through call
scopes to where ever it is handled by log.log_exn(). This then finds an Origin
object in the traceback's stack frames, no need to nest in 'with' scopes.
Hence the 'with log.Origin' now "happens implicitly", we can write pure natural
python code, no more hassles with scope ordering.
Furthermore, any frame can place additional logging information in a frame by
calling log.ctx(). This is automatically inserted in the ancestry associated
with a log statement / exception.
Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
2017-06-09 23:18:27 +00:00
FAIL: test_fail_raise.py (N.N sec) ExpectedFail: This failure is expected
2020-05-06 19:11:02 +00:00
skip: test_suite_params.py
2020-06-12 15:54:55 +00:00
skip: test_timeout.py
2017-09-10 14:33:10 +00:00
- test with half empty scenario
2020-06-12 15:13:26 +00:00
tst test_suite: DBG: {combining='config'} [suite.py:[LINENR]]
2020-06-12 15:54:55 +00:00
tst {combining_scenarios='config'}: DBG: {definition_conf={suite={test_suite={test_timeout={timeout='1'}}}}} [test_suite↪{combining_scenarios='config'}] [suite.py:[LINENR]]
2020-06-12 15:13:26 +00:00
tst {combining_scenarios='config', scenario='foo'}: DBG: {conf={}, scenario='foo'} [test_suite↪{combining_scenarios='config', scenario='foo'}] [suite.py:[LINENR]]
2017-09-10 14:33:10 +00:00
---------------------------------------------------------------------
trial test_suite
---------------------------------------------------------------------
tst test_suite: reserving resources in [PATH]/selftest/suite_test/test_work/state_dir ... [suite.py:[LINENR]]
tst test_suite: DBG: {combining='resources'} [suite.py:[LINENR]]
2017-09-14 13:55:49 +00:00
tst {combining_scenarios='resources'}: DBG: {definition_conf={bts=[{'label': 'sysmoCell 5000'}, {'label': 'sysmoCell 5000'}, {'type': 'sysmo'}], ip_address=[{}], modem=[{}, {}]}} [test_suite↪{combining_scenarios='resources'}] [suite.py:[LINENR]]
2017-09-10 14:33:10 +00:00
tst {combining_scenarios='resources', scenario='foo'}: [RESOURCE_DICT]
2018-08-21 12:58:29 +00:00
tst test_suite: DBG: {combining='modifiers'} [suite.py:[LINENR]]
tst {combining_scenarios='modifiers'}: DBG: {definition_conf={}} [test_suite↪{combining_scenarios='modifiers'}] [suite.py:[LINENR]]
tst {combining_scenarios='modifiers', scenario='foo'}: DBG: {conf={}, scenario='foo'} [test_suite↪{combining_scenarios='modifiers', scenario='foo'}] [suite.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst test_suite: Verifying 3 x bts (candidates: 6) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
[resource.py:[LINENR]]
tst test_suite: Verifying 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
[resource.py:[LINENR]]
tst test_suite: Verifying 2 x modem (candidates: 16) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
[resource.py:[LINENR]]
tst test_suite: DBG: FileWatch: scheduling watch for directory [PATH]/selftest/suite_test/test_work/state_dir [util.py:[LINENR]]
2017-09-10 14:33:10 +00:00
tst test_suite: Reserving 3 x bts (candidates: 6) [resource.py:[LINENR]]
2018-08-27 15:07:41 +00:00
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
2017-09-14 13:55:49 +00:00
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
2018-08-27 15:07:41 +00:00
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
2017-09-14 13:55:49 +00:00
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
2018-08-27 15:07:41 +00:00
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
2017-09-14 13:55:49 +00:00
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
2018-08-27 15:07:41 +00:00
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
2017-09-14 13:55:49 +00:00
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
2017-09-10 14:33:10 +00:00
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
[resource.py:[LINENR]]
tst test_suite: Reserving 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
[resource.py:[LINENR]]
tst test_suite: Reserving 2 x modem (candidates: 16) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
[resource.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst test_suite: DBG: FileWatch: unscheduling watch <ObservedWatch: path=[PATH]/selftest/suite_test/test_work/state_dir, is_recursive=False> [util.py:[LINENR]]
2020-11-28 09:06:14 +00:00
tst test_suite: RESERVED RESOURCES for test_suite:
bts
sysmoCell 5000
sysmoCell 5000
sysmoBTS 1002
ip_address
10.42.42.1
modem
m7801
m7802 [suite.py:[LINENR]]
2017-09-10 14:33:10 +00:00
----------------------------------------------
trial test_suite hello_world.py
----------------------------------------------
tst hello_world.py:[LINENR]: hello world [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
fix: line nr in test name in wrong places
test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.
Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.
Override src() in Test, not name().
Use src() in backtraces.
The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
I am 'test_suite' / 'hello_world.py:23'
becomes
I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)
Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
2020-12-04 16:25:23 +00:00
tst hello_world.py:[LINENR]: I am 'test_suite' / 'hello_world.py' [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
do not redirect_stdout in test scripts
In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.
Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().
The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.
As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.
I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.
Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
2020-12-01 02:51:27 +00:00
tst hello_world.py:[LINENR]: one
two
three [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
2017-11-09 13:26:35 +00:00
tst hello_world.py:[LINENR] Test passed (N.N sec) [test_suite↪hello_world.py] [test.py:[LINENR]]
2017-09-10 14:33:10 +00:00
---------------------------------------------------------------------
trial test_suite PASS
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
PASS: test_suite (pass: 1, skip: 7)
2017-09-10 14:33:10 +00:00
pass: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
skip: test_error.py
skip: test_fail.py
skip: test_fail_raise.py
2020-05-06 19:11:02 +00:00
skip: test_suite_params.py
2020-06-12 15:54:55 +00:00
skip: test_timeout.py
2017-09-10 14:33:10 +00:00
- test with scenario
2020-06-12 15:13:26 +00:00
tst test_suite: DBG: {combining='config'} [suite.py:[LINENR]]
2020-06-12 15:54:55 +00:00
tst {combining_scenarios='config'}: DBG: {definition_conf={suite={test_suite={test_timeout={timeout='1'}}}}} [test_suite↪{combining_scenarios='config'}] [suite.py:[LINENR]]
2020-06-12 15:13:26 +00:00
tst {combining_scenarios='config', scenario='foo'}: DBG: {conf={}, scenario='foo'} [test_suite↪{combining_scenarios='config', scenario='foo'}] [suite.py:[LINENR]]
2017-09-10 14:33:10 +00:00
---------------------------------------------------------------------
trial test_suite
---------------------------------------------------------------------
tst test_suite: reserving resources in [PATH]/selftest/suite_test/test_work/state_dir ... [suite.py:[LINENR]]
tst test_suite: DBG: {combining='resources'} [suite.py:[LINENR]]
2017-09-14 13:55:49 +00:00
tst {combining_scenarios='resources'}: DBG: {definition_conf={bts=[{'label': 'sysmoCell 5000'}, {'label': 'sysmoCell 5000'}, {'type': 'sysmo'}], ip_address=[{}], modem=[{}, {}]}} [test_suite↪{combining_scenarios='resources'}] [suite.py:[LINENR]]
2017-09-10 14:33:10 +00:00
tst {combining_scenarios='resources', scenario='foo'}: [RESOURCE_DICT]
2018-08-21 12:58:29 +00:00
tst test_suite: DBG: {combining='modifiers'} [suite.py:[LINENR]]
tst {combining_scenarios='modifiers'}: DBG: {definition_conf={}} [test_suite↪{combining_scenarios='modifiers'}] [suite.py:[LINENR]]
tst {combining_scenarios='modifiers', scenario='foo'}: DBG: {conf={}, scenario='foo'} [test_suite↪{combining_scenarios='modifiers', scenario='foo'}] [suite.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst test_suite: Verifying 3 x bts (candidates: 6) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
[resource.py:[LINENR]]
tst test_suite: Verifying 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
[resource.py:[LINENR]]
tst test_suite: Verifying 2 x modem (candidates: 16) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
[resource.py:[LINENR]]
tst test_suite: DBG: FileWatch: scheduling watch for directory [PATH]/selftest/suite_test/test_work/state_dir [util.py:[LINENR]]
2017-09-10 14:33:10 +00:00
tst test_suite: Reserving 3 x bts (candidates: 6) [resource.py:[LINENR]]
2018-08-27 15:07:41 +00:00
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
2017-09-14 13:55:49 +00:00
addr: 10.42.42.53
2017-09-10 14:33:10 +00:00
band: GSM-1800
2017-09-14 13:55:49 +00:00
ipa_unit_id: '7'
label: sysmoCell 5000
2018-08-27 15:07:41 +00:00
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
2017-09-10 14:33:10 +00:00
trx_list:
2017-09-14 13:55:49 +00:00
- max_power_red: '3'
2017-09-14 13:33:15 +00:00
nominal_power: '10'
2017-09-14 13:55:49 +00:00
- max_power_red: '0'
nominal_power: '12'
2017-09-10 14:33:10 +00:00
type: osmo-bts-trx
2018-08-27 15:07:41 +00:00
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
2017-09-10 14:33:10 +00:00
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
2018-08-27 15:07:41 +00:00
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
2017-09-10 14:33:10 +00:00
trx_list:
2017-09-14 13:55:49 +00:00
- nominal_power: '10'
- max_power_red: '1'
2017-09-14 13:33:15 +00:00
nominal_power: '12'
2017-09-10 14:33:10 +00:00
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
[resource.py:[LINENR]]
tst test_suite: Reserving 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
[resource.py:[LINENR]]
tst test_suite: Reserving 2 x modem (candidates: 16) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
[resource.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst test_suite: DBG: FileWatch: unscheduling watch <ObservedWatch: path=[PATH]/selftest/suite_test/test_work/state_dir, is_recursive=False> [util.py:[LINENR]]
2020-11-28 09:06:14 +00:00
tst test_suite: RESERVED RESOURCES for test_suite:
bts
sysmoCell 5000
sysmoCell 5000
sysmoBTS 1002
ip_address
10.42.42.1
modem
m7801
m7802 [suite.py:[LINENR]]
2017-09-10 14:33:10 +00:00
----------------------------------------------
trial test_suite hello_world.py
----------------------------------------------
2018-08-21 12:58:29 +00:00
tst hello_world.py:[LINENR]: hello world [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
fix: line nr in test name in wrong places
test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.
Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.
Override src() in Test, not name().
Use src() in backtraces.
The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
I am 'test_suite' / 'hello_world.py:23'
becomes
I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)
Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
2020-12-04 16:25:23 +00:00
tst hello_world.py:[LINENR]: I am 'test_suite' / 'hello_world.py' [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
do not redirect_stdout in test scripts
In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.
Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().
The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.
As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.
I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.
Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
2020-12-01 02:51:27 +00:00
tst hello_world.py:[LINENR]: one
two
three [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
2018-08-21 12:58:29 +00:00
tst hello_world.py:[LINENR] Test passed (N.N sec) [test_suite↪hello_world.py] [test.py:[LINENR]]
---------------------------------------------------------------------
trial test_suite PASS
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
PASS: test_suite (pass: 1, skip: 7)
2018-08-21 12:58:29 +00:00
pass: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
skip: test_error.py
skip: test_fail.py
skip: test_fail_raise.py
2020-05-06 19:11:02 +00:00
skip: test_suite_params.py
2020-06-12 15:54:55 +00:00
skip: test_timeout.py
2018-08-21 12:58:29 +00:00
- test with scenario and modifiers
2020-06-12 15:13:26 +00:00
tst test_suite: DBG: {combining='config'} [suite.py:[LINENR]]
2020-06-12 15:54:55 +00:00
tst {combining_scenarios='config'}: DBG: {definition_conf={suite={test_suite={test_timeout={timeout='1'}}}}} [test_suite↪{combining_scenarios='config'}] [suite.py:[LINENR]]
2020-06-12 15:13:26 +00:00
tst {combining_scenarios='config', scenario='foo'}: DBG: {conf={}, scenario='foo'} [test_suite↪{combining_scenarios='config', scenario='foo'}] [suite.py:[LINENR]]
2018-08-21 12:58:29 +00:00
tst test_suite: reserving resources in [PATH]/selftest/suite_test/test_work/state_dir ... [suite.py:[LINENR]]
tst test_suite: DBG: {combining='resources'} [suite.py:[LINENR]]
tst {combining_scenarios='resources'}: DBG: {definition_conf={bts=[{'label': 'sysmoCell 5000'}, {'label': 'sysmoCell 5000'}, {'type': 'sysmo'}], ip_address=[{}], modem=[{}, {}]}} [test_suite↪{combining_scenarios='resources'}] [suite.py:[LINENR]]
tst {combining_scenarios='resources', scenario='foo'}: [RESOURCE_DICT]
tst test_suite: DBG: {combining='modifiers'} [suite.py:[LINENR]]
tst {combining_scenarios='modifiers'}: DBG: {definition_conf={}} [test_suite↪{combining_scenarios='modifiers'}] [suite.py:[LINENR]]
tst {combining_scenarios='modifiers', scenario='foo'}: DBG: {conf={bts=[{'trx_list': [{'nominal_power': '20'}, {'nominal_power': '20'}]}, {'trx_list': [{'nominal_power': '20'}, {'nominal_power': '20'}]}, {'type': 'sysmo'}]}, scenario='foo'} [test_suite↪{combining_scenarios='modifiers', scenario='foo'}] [suite.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst test_suite: Verifying 3 x bts (candidates: 6) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
[resource.py:[LINENR]]
tst test_suite: Verifying 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
[resource.py:[LINENR]]
tst test_suite: Verifying 2 x modem (candidates: 16) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
[resource.py:[LINENR]]
tst test_suite: DBG: FileWatch: scheduling watch for directory [PATH]/selftest/suite_test/test_work/state_dir [util.py:[LINENR]]
2018-08-21 12:58:29 +00:00
tst test_suite: Reserving 3 x bts (candidates: 6) [resource.py:[LINENR]]
2018-08-27 15:07:41 +00:00
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
2018-08-21 12:58:29 +00:00
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
2018-08-27 15:07:41 +00:00
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
2018-08-21 12:58:29 +00:00
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
2018-08-27 15:07:41 +00:00
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
2018-08-21 12:58:29 +00:00
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
2018-08-27 15:07:41 +00:00
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
2018-08-21 12:58:29 +00:00
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
[resource.py:[LINENR]]
tst test_suite: Reserving 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
[resource.py:[LINENR]]
tst test_suite: Reserving 2 x modem (candidates: 16) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
[resource.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst test_suite: DBG: FileWatch: unscheduling watch <ObservedWatch: path=[PATH]/selftest/suite_test/test_work/state_dir, is_recursive=False> [util.py:[LINENR]]
2020-11-28 09:06:14 +00:00
tst test_suite: RESERVED RESOURCES for test_suite:
bts
sysmoCell 5000
sysmoCell 5000
sysmoBTS 1002
ip_address
10.42.42.1
modem
m7801
m7802 [suite.py:[LINENR]]
2018-08-27 15:07:41 +00:00
resources(test_suite)={'bts': [{'_hash': 'a59640b8ba6a373552b24a6f9f65cadd2347bace',
2018-08-21 12:58:29 +00:00
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'addr': '10.42.42.53',
'band': 'GSM-1800',
'ipa_unit_id': '7',
'label': 'sysmoCell 5000',
2018-08-27 15:07:41 +00:00
'osmo_trx': {'clock_reference': 'external',
'launch_trx': 'False',
'trx_ip': '10.42.42.112'},
2018-08-21 12:58:29 +00:00
'trx_list': [{'max_power_red': '3', 'nominal_power': '20'},
{'max_power_red': '0', 'nominal_power': '20'}],
'type': 'osmo-bts-trx'},
2018-08-27 15:07:41 +00:00
{'_hash': 'c2feabd082c36a1cdeccb9a5237dfff7dbadb009',
2018-08-21 12:58:29 +00:00
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'addr': '10.42.42.53',
'band': 'GSM-1800',
'ipa_unit_id': '7',
'label': 'sysmoCell 5000',
2018-08-27 15:07:41 +00:00
'osmo_trx': {'clock_reference': 'external',
'launch_trx': 'False',
'trx_ip': '10.42.42.112'},
2018-08-21 12:58:29 +00:00
'trx_list': [{'nominal_power': '20'},
{'max_power_red': '1', 'nominal_power': '20'}],
'type': 'osmo-bts-trx'},
{'_hash': '07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'addr': '10.42.42.114',
'band': 'GSM-1800',
'ipa_unit_id': '1',
'label': 'sysmoBTS 1002',
'type': 'sysmo'}],
'ip_address': [{'_hash': 'cde1debf28f07f94f92c761b4b7c6bf35785ced4',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'addr': '10.42.42.1'}],
'modem': [{'_hash': '19c69e45aa090fb511446bd00797690aa82ff52f',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'imsi': '901700000007801',
'ki': 'D620F48487B1B782DA55DF6717F08FF9',
'label': 'm7801',
'path': '/wavecom_0'},
{'_hash': 'e1a46516a1fb493b2617ab14fc1693a9a45ec254',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'imsi': '901700000007802',
'ki': '47FDB2D55CE6A10A85ABDAD034A5B7B3',
'label': 'm7802',
'path': '/wavecom_1'}]}
---------------------------------------------------------------------
trial test_suite
---------------------------------------------------------------------
----------------------------------------------
trial test_suite hello_world.py
----------------------------------------------
2017-09-10 14:33:10 +00:00
tst hello_world.py:[LINENR]: hello world [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
fix: line nr in test name in wrong places
test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.
Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.
Override src() in Test, not name().
Use src() in backtraces.
The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
I am 'test_suite' / 'hello_world.py:23'
becomes
I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)
Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
2020-12-04 16:25:23 +00:00
tst hello_world.py:[LINENR]: I am 'test_suite' / 'hello_world.py' [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
do not redirect_stdout in test scripts
In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.
Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().
The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.
As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.
I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.
Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
2020-12-01 02:51:27 +00:00
tst hello_world.py:[LINENR]: one
two
three [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]
2017-11-09 13:26:35 +00:00
tst hello_world.py:[LINENR] Test passed (N.N sec) [test_suite↪hello_world.py] [test.py:[LINENR]]
2017-09-10 14:33:10 +00:00
---------------------------------------------------------------------
trial test_suite PASS
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
PASS: test_suite (pass: 1, skip: 7)
2017-09-10 14:33:10 +00:00
pass: hello_world.py (N.N sec)
skip: mo_mt_sms.py
skip: mo_sms.py
skip: test_error.py
skip: test_fail.py
skip: test_fail_raise.py
2020-05-06 19:11:02 +00:00
skip: test_suite_params.py
2020-06-12 15:54:55 +00:00
skip: test_timeout.py
2020-05-06 19:11:02 +00:00
- test with suite-specific config
2020-06-12 15:13:26 +00:00
tst test_suite: DBG: {combining='config'} [suite.py:[LINENR]]
2020-06-12 15:54:55 +00:00
tst {combining_scenarios='config'}: DBG: {definition_conf={suite={test_suite={test_timeout={timeout='1'}}}}} [test_suite↪{combining_scenarios='config'}] [suite.py:[LINENR]]
2020-06-12 15:13:26 +00:00
tst {combining_scenarios='config', scenario='foo'}: DBG: {conf={suite={test_suite={some_suite_global_param='heyho', test_suite_params={one_bool_parameter='true', second_list_parameter=['23', '45']}}}}, scenario='foo'} [test_suite↪{combining_scenarios='config', scenario='foo'}] [suite.py:[LINENR]]
2020-05-06 19:11:02 +00:00
tst test_suite: reserving resources in [PATH]/selftest/suite_test/test_work/state_dir ... [suite.py:[LINENR]]
tst test_suite: DBG: {combining='resources'} [suite.py:[LINENR]]
tst {combining_scenarios='resources'}: DBG: {definition_conf={bts=[{'label': 'sysmoCell 5000'}, {'label': 'sysmoCell 5000'}, {'type': 'sysmo'}], ip_address=[{}], modem=[{}, {}]}} [test_suite↪{combining_scenarios='resources'}] [suite.py:[LINENR]]
tst {combining_scenarios='resources', scenario='foo'}: [RESOURCE_DICT]
tst test_suite: DBG: {combining='modifiers'} [suite.py:[LINENR]]
tst {combining_scenarios='modifiers'}: DBG: {definition_conf={}} [test_suite↪{combining_scenarios='modifiers'}] [suite.py:[LINENR]]
tst {combining_scenarios='modifiers', scenario='foo'}: DBG: {conf={}, scenario='foo'} [test_suite↪{combining_scenarios='modifiers', scenario='foo'}] [suite.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst test_suite: Verifying 3 x bts (candidates: 6) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
[resource.py:[LINENR]]
tst test_suite: Verifying 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
[resource.py:[LINENR]]
tst test_suite: Verifying 2 x modem (candidates: 16) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
[resource.py:[LINENR]]
tst test_suite: DBG: FileWatch: scheduling watch for directory [PATH]/selftest/suite_test/test_work/state_dir [util.py:[LINENR]]
2020-05-06 19:11:02 +00:00
tst test_suite: Reserving 3 x bts (candidates: 6) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: a59640b8ba6a373552b24a6f9f65cadd2347bace
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- max_power_red: '3'
nominal_power: '10'
- max_power_red: '0'
nominal_power: '12'
type: osmo-bts-trx
- _hash: c2feabd082c36a1cdeccb9a5237dfff7dbadb009
addr: 10.42.42.53
band: GSM-1800
ipa_unit_id: '7'
label: sysmoCell 5000
osmo_trx:
clock_reference: external
launch_trx: 'False'
trx_ip: 10.42.42.112
trx_list:
- nominal_power: '10'
- max_power_red: '1'
nominal_power: '12'
type: osmo-bts-trx
- _hash: 07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9
addr: 10.42.42.114
band: GSM-1800
ipa_unit_id: '1'
label: sysmoBTS 1002
type: sysmo
[resource.py:[LINENR]]
tst test_suite: Reserving 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: cde1debf28f07f94f92c761b4b7c6bf35785ced4
addr: 10.42.42.1
[resource.py:[LINENR]]
tst test_suite: Reserving 2 x modem (candidates: 16) [resource.py:[LINENR]]
tst test_suite: DBG: Picked - _hash: 19c69e45aa090fb511446bd00797690aa82ff52f
imsi: '901700000007801'
ki: D620F48487B1B782DA55DF6717F08FF9
label: m7801
path: /wavecom_0
- _hash: e1a46516a1fb493b2617ab14fc1693a9a45ec254
imsi: '901700000007802'
ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3
label: m7802
path: /wavecom_1
[resource.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst test_suite: DBG: FileWatch: unscheduling watch <ObservedWatch: path=[PATH]/selftest/suite_test/test_work/state_dir, is_recursive=False> [util.py:[LINENR]]
2020-11-28 09:06:14 +00:00
tst test_suite: RESERVED RESOURCES for test_suite:
bts
sysmoCell 5000
sysmoCell 5000
sysmoBTS 1002
ip_address
10.42.42.1
modem
m7801
m7802 [suite.py:[LINENR]]
2020-05-06 19:11:02 +00:00
resources(test_suite)={'bts': [{'_hash': 'a59640b8ba6a373552b24a6f9f65cadd2347bace',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'addr': '10.42.42.53',
'band': 'GSM-1800',
'ipa_unit_id': '7',
'label': 'sysmoCell 5000',
'osmo_trx': {'clock_reference': 'external',
'launch_trx': 'False',
'trx_ip': '10.42.42.112'},
'trx_list': [{'max_power_red': '3', 'nominal_power': '10'},
{'max_power_red': '0', 'nominal_power': '12'}],
'type': 'osmo-bts-trx'},
{'_hash': 'c2feabd082c36a1cdeccb9a5237dfff7dbadb009',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'addr': '10.42.42.53',
'band': 'GSM-1800',
'ipa_unit_id': '7',
'label': 'sysmoCell 5000',
'osmo_trx': {'clock_reference': 'external',
'launch_trx': 'False',
'trx_ip': '10.42.42.112'},
'trx_list': [{'nominal_power': '10'},
{'max_power_red': '1', 'nominal_power': '12'}],
'type': 'osmo-bts-trx'},
{'_hash': '07d9c8aaa940b674efcbbabdd69f58a6ce4e94f9',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'addr': '10.42.42.114',
'band': 'GSM-1800',
'ipa_unit_id': '1',
'label': 'sysmoBTS 1002',
'type': 'sysmo'}],
'ip_address': [{'_hash': 'cde1debf28f07f94f92c761b4b7c6bf35785ced4',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'addr': '10.42.42.1'}],
'modem': [{'_hash': '19c69e45aa090fb511446bd00797690aa82ff52f',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'imsi': '901700000007801',
'ki': 'D620F48487B1B782DA55DF6717F08FF9',
'label': 'm7801',
'path': '/wavecom_0'},
{'_hash': 'e1a46516a1fb493b2617ab14fc1693a9a45ec254',
'_reserved_by': 'test_suite-[ID_NUM]-[ID_NUM]',
'imsi': '901700000007802',
'ki': '47FDB2D55CE6A10A85ABDAD034A5B7B3',
'label': 'm7802',
'path': '/wavecom_1'}]}
---------------------------------------------------------------------
trial test_suite
---------------------------------------------------------------------
----------------------------------------------
trial test_suite test_suite_params.py
----------------------------------------------
tst test_suite_params.py:[LINENR]: starting test [test_suite↪test_suite_params.py:[LINENR]] [test_suite_params.py:[LINENR]]
do not redirect_stdout in test scripts
In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.
Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().
The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.
As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.
I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.
Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
2020-12-01 02:51:27 +00:00
tst test_suite_params.py:[LINENR]: SPECIFIC SUITE CONFIG: {'some_suite_global_param': 'heyho',
'test_suite_params': {'one_bool_parameter': 'true',
'second_list_parameter': ['23', '45']},
'test_timeout': {'timeout': '1'}} [test_suite↪test_suite_params.py:[LINENR]] [test_suite_params.py:[LINENR]]
2020-05-06 19:11:02 +00:00
tst test_suite_params.py:[LINENR]: SPECIFIC TEST CONFIG: {'one_bool_parameter': 'true', 'second_list_parameter': ['23', '45']} [test_suite↪test_suite_params.py:[LINENR]] [test_suite_params.py:[LINENR]]
tst test_suite_params.py:[LINENR] Test passed (N.N sec) [test_suite↪test_suite_params.py] [test.py:[LINENR]]
2020-06-12 15:54:55 +00:00
----------------------------------------------
trial test_suite test_timeout.py
----------------------------------------------
tst test_timeout.py:[LINENR]: starting test and waiting to receive Timeout after 1 seconds [test_suite↪test_timeout.py:[LINENR]] [test_timeout.py:[LINENR]]
tst test_timeout.py:[LINENR]: ERR: Error: test_timeout.py:[LINENR] Test Timeout triggered: 1 seconds elapsed [test_suite↪test_timeout.py:[LINENR]↪test_timeout.py] [test_suite↪test_timeout.py:[LINENR]] [testenv.py:[LINENR]: raise log_module.Error('Test Timeout triggered: %d seconds elapsed' % self._test.elapsed_time())]
tst test_timeout.py:[LINENR]: Test FAILED (N.N sec) [test_suite↪test_timeout.py:[LINENR]] [test.py:[LINENR]]
2020-05-06 19:11:02 +00:00
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
trial test_suite FAIL
2020-05-06 19:11:02 +00:00
---------------------------------------------------------------------
2020-06-12 15:54:55 +00:00
FAIL: test_suite (fail: 1, pass: 1, skip: 6)
2020-05-06 19:11:02 +00:00
skip: hello_world.py
skip: mo_mt_sms.py
skip: mo_sms.py
skip: test_error.py
skip: test_fail.py
skip: test_fail_raise.py
pass: test_suite_params.py (N.N sec)
2020-06-12 15:54:55 +00:00
FAIL: test_timeout.py (N.N sec) Error: test_timeout.py:[LINENR] Test Timeout triggered: 1 seconds elapsed [test_suite↪test_timeout.py:[LINENR]↪test_timeout.py]
2020-06-04 16:44:42 +00:00
- test with template overlay
cnf suiteC: DBG: reading suite.conf [suite.py:[LINENR]]
2020-06-12 15:13:26 +00:00
tst suiteC: DBG: {combining='config'} [suite.py:[LINENR]]
tst {combining_scenarios='config'}: DBG: {definition_conf={}} [suiteC↪{combining_scenarios='config'}] [suite.py:[LINENR]]
2020-06-04 16:44:42 +00:00
---------------------------------------------------------------------
trial suiteC
---------------------------------------------------------------------
tst suiteC: reserving resources in [PATH]/selftest/suite_test/test_work/state_dir ... [suite.py:[LINENR]]
tst suiteC: DBG: {combining='resources'} [suite.py:[LINENR]]
tst {combining_scenarios='resources'}: DBG: {definition_conf={ip_address=[{'addr': '10.42.42.2'}]}} [suiteC↪{combining_scenarios='resources'}] [suite.py:[LINENR]]
tst suiteC: DBG: {combining='modifiers'} [suite.py:[LINENR]]
tst {combining_scenarios='modifiers'}: DBG: {definition_conf={}} [suiteC↪{combining_scenarios='modifiers'}] [suite.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst suiteC: Verifying 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst suiteC: DBG: Picked - _hash: fd103b22c7cf2480d609150e06f4bbd92ac78d8c
addr: 10.42.42.2
[resource.py:[LINENR]]
tst suiteC: DBG: FileWatch: scheduling watch for directory [PATH]/selftest/suite_test/test_work/state_dir [util.py:[LINENR]]
2020-06-04 16:44:42 +00:00
tst suiteC: Reserving 1 x ip_address (candidates: 3) [resource.py:[LINENR]]
tst suiteC: DBG: Picked - _hash: fd103b22c7cf2480d609150e06f4bbd92ac78d8c
addr: 10.42.42.2
[resource.py:[LINENR]]
2020-11-09 20:17:51 +00:00
tst suiteC: DBG: FileWatch: unscheduling watch <ObservedWatch: path=[PATH]/selftest/suite_test/test_work/state_dir, is_recursive=False> [util.py:[LINENR]]
2020-11-28 09:06:14 +00:00
tst suiteC: RESERVED RESOURCES for suiteC:
ip_address
10.42.42.2 [suite.py:[LINENR]]
2020-06-04 16:44:42 +00:00
----------------------------------------------
trial suiteC test_template_overlay.py
----------------------------------------------
tst test_template_overlay.py:[LINENR]: - Testing: expect to fail on invalid templates overlay dir [suiteC↪test_template_overlay.py:[LINENR]] [test_template_overlay.py:[LINENR]]
2020-06-11 14:14:47 +00:00
tst test_template_overlay.py:[LINENR] DBG: template dir overlay set: [PATH]/selftest/suite_test/suitedirB/suiteC/nonexistent-templatedir [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py] [testenv.py:[LINENR]]
2020-12-04 17:03:22 +00:00
tst test_template_overlay.py:[LINENR]: success: setting non-existing templates dir raised RuntimeError [suiteC↪test_template_overlay.py:[LINENR]] [test_template_overlay.py:[LINENR]]
2020-06-11 14:14:47 +00:00
tst test_template_overlay.py:[LINENR] DBG: template dir overlay set: [PATH]/selftest/suite_test/suitedirB/suiteC/mytemplatedir [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py] [testenv.py:[LINENR]]
2020-06-04 16:44:42 +00:00
--- ReservedResources: DBG: requesting use of ip_address {specifics={}} [resource.py:[LINENR]]
tst suiteC: Using 1 x ip_address (candidates: 1) [resource.py:[LINENR]]
tst suiteC: DBG: Picked - _hash: fd103b22c7cf2480d609150e06f4bbd92ac78d8c
_reserved_by: suiteC-[ID_NUM]-[ID_NUM]
addr: 10.42.42.2
[resource.py:[LINENR]]
--- ReservedResources: DBG: {available=1} [resource.py:[LINENR]]
--- ReservedResources: DBG: {using={_hash='fd103b22c7cf2480d609150e06f4bbd92ac78d8c', _reserved_by='suiteC-[ID_NUM]-[ID_NUM]', addr='10.42.42.2'}} [resource.py:[LINENR]]
tst test_template_overlay.py:[LINENR]: - Testing: original template [suiteC↪test_template_overlay.py:[LINENR]] [test_template_overlay.py:[LINENR]]
run osmo-stp_10.42.42.2: DBG: {config_file='test_trial_tmp/test_run_6/suiteC/test_template_overlay.py/osmo-stp_10.42.42.2/osmo-stp.cfg'} [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
run osmo-stp_10.42.42.2: DBG: STP CONFIG:
{'stp': {'ip_address': {'_hash': 'fd103b22c7cf2480d609150e06f4bbd92ac78d8c',
'_reserved_by': 'suiteC-[ID_NUM]-[ID_NUM]',
'_used': True,
'addr': '10.42.42.2'}}} [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
cnf Templates: DBG: rendering osmo-stp.cfg.tmpl [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪Templates] [template.py:[LINENR]]
run osmo-stp_10.42.42.2: DBG: ! Configuration rendered by osmo-gsm-tester
log stderr
logging filter all 1
logging color 1
logging print category 1
2020-11-30 23:43:52 +00:00
logging print category-hex 0
logging print level 1
logging print file basename last
2020-06-04 16:44:42 +00:00
logging print extended-timestamp 1
logging level set-all debug
line vty
no login
bind 10.42.42.2
!ctrl
! bind 10.42.42.2
cs7 instance 0
xua rkm routing-key-allocation dynamic-permitted
listen m3ua 2905
accept-asp-connections dynamic-permitted
local-ip 10.42.42.2
[suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
tst test_template_overlay.py:[LINENR]: - Testing:overlay template [suiteC↪test_template_overlay.py:[LINENR]] [test_template_overlay.py:[LINENR]]
tst test_template_overlay.py:[LINENR]: - After creating the new template, still old template is used [suiteC↪test_template_overlay.py:[LINENR]] [test_template_overlay.py:[LINENR]]
run osmo-stp_10.42.42.2: DBG: {config_file='test_trial_tmp/test_run_6/suiteC/test_template_overlay.py/osmo-stp_10.42.42_2.2/osmo-stp.cfg'} [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
run osmo-stp_10.42.42.2: DBG: STP CONFIG:
{'stp': {'ip_address': {'_hash': 'fd103b22c7cf2480d609150e06f4bbd92ac78d8c',
'_reserved_by': 'suiteC-[ID_NUM]-[ID_NUM]',
'_used': True,
'addr': '10.42.42.2'}}} [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
cnf Templates: DBG: rendering osmo-stp.cfg.tmpl [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪Templates] [template.py:[LINENR]]
run osmo-stp_10.42.42.2: DBG: ! Configuration rendered by osmo-gsm-tester
log stderr
logging filter all 1
logging color 1
logging print category 1
2020-11-30 23:43:52 +00:00
logging print category-hex 0
logging print level 1
logging print file basename last
2020-06-04 16:44:42 +00:00
logging print extended-timestamp 1
logging level set-all debug
line vty
no login
bind 10.42.42.2
!ctrl
! bind 10.42.42.2
cs7 instance 0
xua rkm routing-key-allocation dynamic-permitted
listen m3ua 2905
accept-asp-connections dynamic-permitted
local-ip 10.42.42.2
[suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
tst test_template_overlay.py:[LINENR]: - New template is used after re-generating cache with set_overlay_template_dir: [suiteC↪test_template_overlay.py:[LINENR]] [test_template_overlay.py:[LINENR]]
2020-06-11 14:14:47 +00:00
tst test_template_overlay.py:[LINENR] DBG: template dir overlay set: [PATH]/selftest/suite_test/suitedirB/suiteC/mytemplatedir [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py] [testenv.py:[LINENR]]
2020-06-04 16:44:42 +00:00
run osmo-stp_10.42.42.2: DBG: {config_file='test_trial_tmp/test_run_6/suiteC/test_template_overlay.py/osmo-stp_10.42.42_3.2/osmo-stp.cfg'} [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
run osmo-stp_10.42.42.2: DBG: STP CONFIG:
{'stp': {'ip_address': {'_hash': 'fd103b22c7cf2480d609150e06f4bbd92ac78d8c',
'_reserved_by': 'suiteC-[ID_NUM]-[ID_NUM]',
'_used': True,
'addr': '10.42.42.2'}}} [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
cnf Templates: DBG: rendering osmo-stp.cfg.tmpl [suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪Templates] [template.py:[LINENR]]
run osmo-stp_10.42.42.2: DBG: ! Overlay Config file genreated by test
line vty
no login
bind 10.42.42.2
[suiteC↪test_template_overlay.py:[LINENR]↪test_template_overlay.py↪osmo-stp_10.42.42.2] [stp_osmo.py:[LINENR]]
tst test_template_overlay.py:[LINENR] Test passed (N.N sec) [suiteC↪test_template_overlay.py] [test.py:[LINENR]]
---------------------------------------------------------------------
trial suiteC PASS
---------------------------------------------------------------------
PASS: suiteC (pass: 1)
pass: test_template_overlay.py (N.N sec)
2017-03-28 12:30:28 +00:00
- graceful exit.
