A new mcc_mnc parameter is now optionally passed to connect() in order
to manually register to a specific network with a given MCC+MNC pair.
If no parameter is passed (or None), then the modem will be instructed
to attempt an automatic registration with any available network which
permits it.
We get the MCC+MNC parameter from the MSC/NITB and we pass it to the
modem object at connect time as shown in the modified tests. Two new
simple tests to check network registration is working are added in this
commit.
Ofono modems seem to be automatically registering at some point after
they are set Online=true, and we were actually using that 'feature'
before this patch. Thus, it is possible that a modem quickly becomes
registered, and we then check so before starting the scan+registration
process, which can take a few seconds.
The scanning method can take a few seconds to complete. To avoid
blocking in the dbus ofono Scan() method, this commit adds some code to
make use of glib/gdbus async methods, which are not yet supported
directly by pydbus. This way, we can continue polling while waiting for
the scan process to complete and we can register several modems in
parallel. When scan completes, a callback is run which attempts to
register. If no MCC+MNC was passed, as we just finished scanning the
modem should have enough fresh operator information to take good and
quick decisions on where to connect. If we have an MCC+MNC, then we check
the operator list received by Scan() method. If operator with desired
MCC+MNC is there, we register with it. If it's not there, we start
scanning() again asynchronously hoping the operator will show up in next
scan.
As scanning() and registration is done in the background, tests are
expected to call connect(), and then later on wait for the modem to
register by waiting/polling the method "modem.is_connected()". Tests
first check for the modem being connected and after with MSC
subscriber_attached(). The order is intentional because the later has to
poll through network and adds unneeded garbage to the pcap files bein
recorded.
Change-Id: I8d9eb47eac1044550d3885adb55105c304b0c15c
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb414406 "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
Implement the Modem.log_info() function, use that instead of logging all modem
properties.
Tweak mo_mt_sms.py print() statements.
Pass modem object to SMS generation to inlcude the modem name as SMS token.
Change-Id: I2b17fce0b3b05594fd9038b54e5b65f5127bd0a4
It would be nicer to select the network programs as scenarios, i.e.
independently from the specifics of tests that don't care whether a NITB or a
MSC+BSC is in place. See OS#2270.
For now have a separate script for BSC+MSC+HLR operation to be able to rapidly
get the binaries to work. We might even simply drop the NITB style, in which
case we don't need to make it configurable.
Change-Id: I06d2529822f3e483ce7fffc439edfa57844e01ef
I would like to use the IP addresses also for OsmoBSC processes, so it is more
than clear now that 'nitb_iface' was the wrong naming choice.
The only distinction we may need in the future is public versus loopback
interface. To add that, we may add a trait to the 'ip_address' resource
like:
ip_address:
- addr: 10.42.42.1
type: public
- addr: 127.0.0.1
type: loopback
This way we can substitute public vs loopback addresses flexibly (e.g. using
scenarios).
Change-Id: I3ad583ae7a33f7a7bb56fe78a125f73c56a0e860
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3cce3333a187a4ac6d5c6b70fe9d5c5
/suites will be the definitive GSM tests collection where everyone should contribute.
Since we're using /example on our current osmo-gsm-tester setup actually as-is,
change paths.conf to point at ../suites.
Change-Id: I7a4d0161d3dcb3a0c723b0b96db85dd032cc2159
Pau Espin
Neels Hofmeyr
Neels Hofmeyr