All config parameters are hardcoded for now and it should push
subscribers into the test (which should be a base class but not
a specific test).
Change-Id: I586b6d532c6e6395b4e6f2cf2128372237d05a7d
suite.config() hasn't been activetly used anywere. New upcoming patches
will start using it to set and overlay config parameters for objects
which are not necessarily allocated as resources.
Since these options don't apply to specific resources, there's no sense
in running replicate_times() on them.
Change-Id: I5fcdc9151d1663e738f5c8abe588a42ab51c8f6e
Before this patch, scenarios were only used to select resources with
specific attributes. This commit introduces "modifiers" in scenarios,
which allows setting or modifing config attributes of resources once
they have been reserved. This way same test can be run selecting same
resources but modifying its configuration, allowing for instance running
different number of TRX, different timeslot configuration, etc.
Modifiers are described by placing a "modifiers" dictionary in any
scenario file, similar to the current "resources" one used to select
requird resources. The "modifiers" dictionary is overlaid on top of the
"resources" one resulting from combining all the "resources" dictionary
of all scenario files.
Change-Id: If8c422c67d9a971d9ce2c72594f55cde2db7550d
After bug described in OS#3456 and fixed in last commit, let's
categorize and place variables in its correct plac to avoid similar
issus. We leave under the class keyword (class scoped variables) the
attributes which are to be used as static class attributes. All other
ones are initialized during __init__(). This way w avoid scenarios in
which while using an object from an instance attribute we end up reading
a class scoped variable which is shared among all instances.
Change-Id: I5ad4cac34a9f49eaf42966c01c9c5a4d3f3e9dc8
Some tests may want to reproduce some scenarios in which it is expected
that a BTS process is stopped, for instance if the BSC link is dropped.
Provide a keepalive parameter to start() for bts and pcu objects to
inform suite that failures are expected and that it should keep them
alive in case that ocurrs by respawning the BTS process.
Change-Id: Ia2a7539f9fad457125ac9b60a52a52999e885ba8
Since sys.path is modified idynamically to load modules from "lib" subdir of each suite, from python env point of view all those modules share a namespace. As a result, there can be name collisions.
If a name collision appears (eg test1 loads "testlib.py" and test2
afterwards also loads its own "testlib.py"), then python interpreter
thinks the testlib.py module is already loaded, so test2 ends up using
"testlib.py" from test1.
The way to solve this is to make suite local
modules to live only through the scope of the suite, and unload the
modules once the suite is finished.
Change-Id: I4efe815f85bc4ec2ca91aa9c2d3a369048f21571
Make the code more reachable for newcomers, as well as more organized
for people who work a lot with it.
SuiteRun in suite.py is already quite big, and having the Test class in
there make it unnecessarily more big, and makes it difficult to find
stuff.
At the same time, having a test.py which does actually not contain the
Test class but other stuff, makes it even more confusing.
Change-Id: I9c8d67f598466ba52a4827ff77027b9eae85929a
Since Change-Id Ia2882b7ca31a3219c676986e85045fa08a425d7a, osmo-bsc
uses osmo-mgw and utilizes libosmo-mgcp-client to talk to it.
This commit fixes latest constant failures in voice suite.
Change-Id: I1dadd781a357fce33e7bde55e4bcbdaeb4633359
As suite.conf and scenarios need to match 1-to-1 in lists, it's important to
extend the dictionaries by replicating the objects with a 'times' values
higher than 1 in order to match the objects correctly.
Since dictionanries are expanded at combine time, there's no need to
expand them during reserve() time because they are already expanded.
As a result, this commit reworks the kind of schema applied in each
place (and takes the change to start validating scenario files, which
were neglected previously).
Two unit tests are added as a show case. Unfortunately output showing
scenario dictionaries needs to be ignored while verifying because it was
encountered that different versions of python print dictionary elements
in different order.
Change-Id: I25eb639c7e3cf3b4c67a205422808bffbdd791e6
In commit 329b6f4 pcu-socket path was moved to run inside the test run
dir to avoid issues between different tests creating a socket in the
same place.
However, it seems unix sockets paths are limited to 108 bytes (with Null
char included). In some cases, the run dir for a test can be quite long,
as it contains suite name, test name, etc. and the path can be longer
that the limit defined above.
In order to fix this issue, create a tmp dir using mkdtemp to ensure the
path to be used for the pcu-socket doesn't collide between different
instances of osmo-bts-trx.
Clean up of tmp dir and pcu socket is done inside the cleanup() method
called by suite.py.
method pcu_socket_path() is added to help with new implementation, and
it will be used as well as a public API later soon to be used by OsmoPcu
classes.
Related: OS#2507
Change-Id: I0c53a0a3ccc5eb2823265fe14c0f7b8f4adb1038
In commit 6100b629e5 I assumed esme
already had a cleanup function, which doesn't seem to be the case. This
means if a test calls esme.connect() and fails afterwards without
calling esme.disconnect(), the connection will be kept but as the test
will finish and the NITB is closed, then the connection will be kept
forever in an erroneous state. With the cleanup function we ensure we
will close the connection after the test.
Change-Id: Ia3a23bb8bbccfe7ac5c544521b5566164609f0a5
Let's accept that during cleanup time some stuff may not be in a good
state, specially in case of a failure, but let's keep harder to clean
up everything as much as possible.
This should fix an issue in which after a test failure using an esme object,
an smpp connection is kept in a buggy state and never removed/cleaned up, and
makes all tests after it fail too because the conn is triggered at
poll() time:
File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/suite.py", line 260, in run_tests
self.objects_cleanup()
File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/suite.py", line 199, in objects_cleanup
obj.cleanup()
File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/ofono_client.py", line 345, in cleanup
self.dbus.cleanup()
File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/ofono_client.py", line 170, in cleanup
self.set_powered(False)
File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/ofono_client.py", line 304, in set_powered
self.set_bool('Powered', powered)
File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/ofono_client.py", line 295, in set_bool
event_loop.poll()
File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/event_loop.py", line 39, in poll
func()
File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/esme.py", line 78, in poll
self.client.poll()
File "/usr/local/lib/python3.4/dist-packages/smpplib/client.py", line 321, in poll
self.read_once(ignore_error_codes)
File "/usr/local/lib/python3.4/dist-packages/smpplib/client.py", line 279, in read_once
p = self.read_pdu()
File "/usr/local/lib/python3.4/dist-packages/smpplib/client.py", line 206, in read_pdu
raise exceptions.ConnectionError()
smpplib.exceptions.ConnectionError
Change-Id: Ie7ef9284490f12f5cfd76c35b33b57eefab20eb6
As defined in [1], the different related actors are implemented in this
commit: ESME and SMSC.
SMSC: In Osmocom, the SMSC is currently implemented inside the NITB or
the MSC. A new Smsc abstract class is created to shared code between the
NITB and the MSC, and also makes it easier for later when the SMSC is
splitted. ESMEs can be dynamically added to its configuration in a
similar way to how the BTSs are added.
ESME: A new class Esme is created which can be used by tests to control
an ESME to interact with the SMSC. The ESME functionalities are
implemented using python-smpplib. Required version of this library is at
least 43cc6f819ec76b2c0a9d36d1d439308634716227, which contains support
for python 3 and some required features to poll the socket.
This commit already contains a few tests which checks different
features and tests the API. Extending tested features or scenarios can be
later done quite easily.
The tests are not enabled by default right now, because there are several
of them in a suite and the ip_address resources are not freed after every
tests which ends up in the suite failing due to missing reserved
resources. All the tests run alone work though. When the issue is fixed
they can then be added to the default list of tests to be run.
[1] http://opensmpp.org/specs/SMPP_v3_4_Issue1_2.pdf
Change-Id: I14ca3cb009d6d646a449ca99b0200da12085c0da
Fix bug in put_all(). That method was unused before this commit.
Clean the process list after the processes are stopped,
otherwise the suite eventually fails with a 'process terminated
prematurely' error. Before it was not necessary because it was not
reused (a new suite run was created).
Change-Id: Iee12866045145544076c6c18786e1a54f18fc293
Processes created have the scope of the test, so we should store
everything in a per-suite_run/per-test directory, otherwise everything
is stored in the same trial run_dir directory and it's really messy.
Change-Id: I06be2dd21710e14c1337d13b1fe6c2f68f037957
With the recent fix of the junit report related issues, another issue arose:
the 'with log.Origin' was changed to disallow __enter__ing an object twice to
fix problems, now still code would fail because it tries to do 'with' on the
same object twice. The only reason is to ensure that logging is associated with
a given object. Instead of complicating even more, implement differently.
Refactor logging to simplify use: drop the 'with Origin' style completely, and
instead use the python stack to determine which objects are created by which,
and which object to associate a log statement with.
The new way: we rely on the convention that each class instance has a local
'self' referencing the object instance. If we need to find an origin as a new
object's parent, or to associate a log message with, we traverse each stack
frame, fetching the first local 'self' object that is a log.Origin class
instance.
How to use:
Simply call log.log() anywhere, and it finds an Origin object to log for, from
the stack. Alternatively call self.log() for any Origin() object to skip the
lookup.
Create classes as child class of log.Origin and make sure to call
super().__init__(category, name). This constructor will magically find a parent
Origin on the stack.
When an exception happens, we first escalate the exception up through call
scopes to where ever it is handled by log.log_exn(). This then finds an Origin
object in the traceback's stack frames, no need to nest in 'with' scopes.
Hence the 'with log.Origin' now "happens implicitly", we can write pure natural
python code, no more hassles with scope ordering.
Furthermore, any frame can place additional logging information in a frame by
calling log.ctx(). This is automatically inserted in the ancestry associated
with a log statement / exception.
Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
Even if aborted due to signal, write a JUnit report XML, and make sure to
indicate the runs as erratic.
Change-Id: I7a334ef3463896c543c0fe592d3903c15e67d4c4
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb414406 "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
For all those API functions that directly use reserved_resources.get(), add a
'specifics' argument to be able to pick specific resources. For example, this
allows to pick a suite.bts(specifics={'type': 'osmo-bts-sysmo'})
I needed this to test error reporting for over-using resources, but will most
probably make sense in the future.
Change-Id: If6f175f4bb53dec5306fb3c6479202a7bf1c7116
The "Affero" nature makes sense for the Osmocom network components like
BSC, SGSN, etc. as they are typically operated to provide a network
service.
For testing, this doesn't make so much sense as it is difficult to
imagine people creating a business out of offering to run test cases on
an end-to-end Osmocom GSM network. So let's drop the 'Affero' here.
All code is so far developed by sysmocom staff, so as Managing Director
of sysmocom I can effect such a license change unilaterally.
Change-Id: I8959c2d605854ffdc21cb29c0fe0e715685c4c05
After a suite was done, the modem object would linger. If two suites were run
consecutively, the first suite's modem objects would still log incoming SMS.
Add an object cleanup mechanism in the SuiteRun class. Start by adding a
cleanup() to the Modem object and subscribing created modems there.
Move the modem_obj() function into SuiteRun, there is no use of it being
separate, and it makes for better logging.
Change-Id: I0048d33e661d683a263c98128cd5c38b8d897dab
Add missing code to free resources, not upon program exit, but when a suite is
done.
This allows running more than one suite in a row.
Also add a check to not attempt to free if there is nothing to be freed, to
avoid a regression test failure triggered when a suite exits without reserving
anything.
Change-Id: Ic017a1cf07052f5e48812c8553fba6f972d280f0
Related: OS#2301
I would like to use the IP addresses also for OsmoBSC processes, so it is more
than clear now that 'nitb_iface' was the wrong naming choice.
The only distinction we may need in the future is public versus loopback
interface. To add that, we may add a trait to the 'ip_address' resource
like:
ip_address:
- addr: 10.42.42.1
type: public
- addr: 127.0.0.1
type: loopback
This way we can substitute public vs loopback addresses flexibly (e.g. using
scenarios).
Change-Id: I3ad583ae7a33f7a7bb56fe78a125f73c56a0e860
Make sure we free the reserved resources and kill launched subprocesses
before stopping. Before this patch it was not the case for instance if we
received a SIGTREM signal from kill.
Change-Id: I039e4d1908a04bf606b101ddc6a186ba67e6178e
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3cce3333a187a4ac6d5c6b70fe9d5c5
On the log output, clearly show which state dir is used. Since all runs should
use the same state dir, this is important and/or reassuring to see.
Change-Id: Ie92f1879a35d8bb399ee916b7ef72f9ee7d47409
Holger Hans Peter Freyther
Pau Espin
Neels Hofmeyr
Harald Welte