First user will be the upcoming handover_2G/handover.py test in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.
Change-Id: Id799b3bb81eb9c04d13c26ff611e40363920300e
Allow enriching the junit output with arbitrary subtasks within a test.
The current aim is, for handover tests, to not just show that a test
failed, but to show exactly which steps worked and which didn't, e.g.:
handover.py/01_bts0_started PASSED
handover.py/02.1_ms0_attach PASSED
handover.py/02.2_ms1_attach PASSED
handover.py/02.3_subscribed_in_msc PASSED
handover.py/03_call_established PASSED
handover.py/04.1_bts1_started FAILED
In this case it is immediately obvious from looking at the jenkins
results analyzer that bts1 is the cause of the test failure, and it is
visible which parts of the test are flaky, over time.
First user Will be the upcoming handover_2G suite, in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.
Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59
In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.
Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().
The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.
As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.
I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.
Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.
Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.
Override src() in Test, not name().
Use src() in backtraces.
The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
I am 'test_suite' / 'hello_world.py:23'
becomes
I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)
Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
Remove ARFCNs as a concept from resource pool, assign a fixed ARFCN to
each BTS and TRX in the resource pools.
Using ARFCNs on specific bands as resources was an idea that is hard to
implement, because specific BTS dictate selection of bands which
influences which ARFCNs can be picked. That means reserving ARFCN
resources is only possible after reserving specific BTS resources, but
the tester is currently not capable of such two-stage resolution.
Writing handover tests, I got the problem that both BTS in a scenario
attempt to use the same ARFCN.
The by far easiest solution is to assign one fixed ARFCN to each BTS and
TRX. If ever needed, a scenario modifier can still configure different
ARFCNs.
(Due to uncertainty about OC2G operation stability, I prefer to leave
OC2G on ARFCN 50, as it happened to end up being configured before this
patch.)
Change-Id: I0a6c60544226f4261f9106013478d6a27fc39f38
On non-debug log level, show something like this at the beginning of
each suite:
03:45:49.439720 tst handover:sysmo+secondbts-trx-b200: RESERVED RESOURCES for handover:
bts
sysmoBTS 1002
Ettus B200
ip_address
10.42.42.2
10.42.42.3
10.42.42.4
10.42.42.5
10.42.42.6
10.42.42.7
modem
sierra_1st
sierra_2nd
Change-Id: Ic23556eafee654c93d13c5ef405028da09bd51d7
Before this patch, almost everything was in place to support concurrent
osmo-gsm-tester instances sharing a common state dir. However, during
resource reservation, if the reservation couldn't be done due to too
many resources being in use, osmo-gsm-tester would fail and skip the
test suite.
With this patch, OGT will wait until some reserved resources are
released and then try requesting the reservation again.
Change-Id: I938602ee890712fda82fd3f812d8edb1bcd05e08
tests can now use 'tenv.test().set_kpis(some_dict)' to set any kind of
data as KPIs, which will be presented in the junit report.
The representation of KPIs in the xml file doesn't follow the junit
format, mainly because it has no support for per-test properties.
Change-Id: I00e976f65a202e82d440bf33708f06c8ce2643e2
Timeout value can be specified by test in suite.conf:
config:
suite:
<suite_name>:
<test_name>:
timeout: 2 # 2 seconds timeout
Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706
This feature is not really implemented and maybe never was. In any case,
it makes sense to have that working per-test so we can specify different
values per test in case it's needed.
Change-Id: I3c1b95c10e974da87ec9abd25578d8bcc0bc55a3
This way tests which require a very specific config file can override
specific template files used by object classes.
Change-Id: I65d1b1e826d2d430ee83810d998b98d0ccaa07cd
tgz files in trials can be categorized in subdirectories, allowing to
select different bianry files at runtime based on the target run node
which is going to run them. This way for instance one can have a binary
linked against libs for eg. CentOS under run_label "centos/" or an ARM
target under "arm", and then use "run_label: arm" on the resource using
it.
Change-Id: Iaf2e97da3aff693395f44f0e93b184d4846cf6da
This allows inheriting suites or scenarios from eg. sysmocom/ dir, while
still allowing to apply new suites and scenarios on top.
Change-Id: Icecdae32d400a6b6da2ebf167c1c795f7a74ae96
It has been notified that current configuration system is difficult to
understand and to use, so it has been envisioned to refactor it a bit.
The idea is that the user passes a -c path/to/main.conf file, which in
turn contains whatever osmo-gsm-tester main settings supports (basically
what old paths.conf used to be, plus some files harcoded to the same -c
directory are now configurable through the main configuration file).
Change-Id: Ieca65b71b543c44cfcec8e83efd0fe053c432e55
Environment variable OSMO_GSM_TESTER_CONF is marked as deprecated and an
error is logged each time it's used. Same feature is available through
"-c" command line parameters, so having the envvar only makes things
more complex for no good reason. It cannot yet be completely dropped
since some environemnt still make use of it. Give some time to users to
adapt their setups.
Other environment variables setting some config apths can be dropped
since they are not being really used in any setup.
Change-Id: I7eb69f870d0dcb5906d45ae067d6bed1aabf5862
On older versions apparently upon child termination due to SIGINT
subprocess.poll() returned 1. On new python versions (such as 3.8.2),
-2 is returned, according to documentation:
A negative value -N indicates that the child was terminated by signal N (Unix only).
Let's catch the SIGINT in the child process and exit with a known 42
value to fix different behavior.
Change-Id: I7949ff2b435e91e890061e6840b0f411f8b0a817
If using python 3.8.2, the trailing comma at the end of parameters is
not longer there (probably got fixed, since it's not needed). That
change breaks expected output.
Change-Id: I6da3024f946c0e761099058e812c0eacf3d6071f
This way we benefit from:
* knowing which attributes are used/required by each object class and
subclass
* Having validation function definitions near the class going to use them
Change-Id: I8fd6773c51d19405a585977af4ed72cad2b21db1
If class Test calls get_run_dir(), it will generate the run dir from
Suite.get_run_dir() + append, and Suite.get_run_dir() in turn will do
the same with Trial.get_run_dir(). In suite_test.py, however, we are
passing a plain log.Origin object (parent of Trial), which doesn't
implement this function. We need to add it since tests will start using
get_run_dir() to write their own log files.
Generated tmp files are deleted on success and left there on error, in
order to be able to debug stuff easily.
Change-Id: Ic8326659a9ced0351e1e7e57ef21ce36f777729a
This avoids extra unneeded logging about killing with signal when
actually no signal is being sent.
Change-Id: I5b5dd78fe3301d8eef2ab93da3b37029268ae198
It's always been the default anyway, so no need to explicitly state it.
Furthermore, recently (osmobsc 167cb828665b91ed7d28007abd4a1862ebbb78c9)
force-combined-si was enabled by default for nanoBTS due to issues with
some MS not registering when it's not enabled, so let's osmo-bsc decide
were should it be enabled.
Related: OS#3063
Change-Id: I97b10406a4a0048c66a47aa7d32a113f0c0366d1
VTY cmd "logging level all" is deprecated in favour of "logging level
set-all", but the required behaviour in this case is "logging level
set-all".
Change-Id: Ice54fca6be86f05a02ce4ae783e6e42e6958b95d
* This commit is a preparation for future commits to add support for
different osmo-trx devices and backends like osmo-trx-lms.
* Drop deprecated osmo-trx-* cmd line params and use VTY cfg to set them.
* As number of osmo-trx related osmo-gsm-tester attributes grow, group
them togther in an "osmo_trx" dictionary.
Change-Id: I77d29413c9e3b600b796627ba366f80c3281b7e1
Before this patch, scenarios were only used to select resources with
specific attributes. This commit introduces "modifiers" in scenarios,
which allows setting or modifing config attributes of resources once
they have been reserved. This way same test can be run selecting same
resources but modifying its configuration, allowing for instance running
different number of TRX, different timeslot configuration, etc.
Modifiers are described by placing a "modifiers" dictionary in any
scenario file, similar to the current "resources" one used to select
requird resources. The "modifiers" dictionary is overlaid on top of the
"resources" one resulting from combining all the "resources" dictionary
of all scenario files.
Change-Id: If8c422c67d9a971d9ce2c72594f55cde2db7550d
We are using the CDF to decide which percentage of the jobs should
be running at a given point. The x-axis is time and the y-axis the
percentage of how many jobs should be running.
There are three functions to do this. The first one is a constant
which would result in everything being started right now, one to
start them linearly and the last (formula from Qt/3rdparty) to first
accelerate and decelerate slowly.
Change-Id: I9e3064f4c3c4c7af5d3491f850090516e541f4d3
Since libosmocore doesn't support XOR at all, it seems weird to use it in the
test, even though it's just a selftest without libosmocore involved...
Change-Id: I51edec255e7ef277907817b3187c2f492465467f
Neels Hofmeyr
Pau Espin