Commit Graph

162 Commits

Author SHA1 Message Date
Pau Espin 374e8a0b26 suite.py: Fix typo in variable name used during exception
Fixes a possible double exception introduced in 0ffb4144

Change-Id: I44a30f51c23e6a57ccae3483c806c0ed25d2ecf9
2017-05-22 17:52:09 +02:00
Pau Espin 469316fae0 Handle termination signals to exit gracefully and prevent resource leak
Make sure we free the reserved resources and kill launched subprocesses
before stopping. Before this patch it was not the case for instance if we
received a SIGTREM signal from kill.

Change-Id: I039e4d1908a04bf606b101ddc6a186ba67e6178e
2017-05-19 14:40:18 +02:00
Pau Espin 0ffb414406 Add JUnit XML reports; refactor test reporting
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.

Change-Id: Iedf6d912b3cce3333a187a4ac6d5c6b70fe9d5c5
2017-05-19 14:22:39 +02:00
Pau Espin 023fd2c748 pcap_recorder: use self.filters instead of gen_filter
Change-Id: I53779d521fbeed7c37bcaeb12be2531e4dbc005c
2017-05-15 15:01:15 +02:00
Pau Espin 6ab98983bd Enable GSMTAP on BTS and record it on the main unit NITB
Change-Id: I057a83bc20c5e664ac1778812452f189166919ba
2017-05-15 14:57:42 +02:00
Pau Espin 8cd347b2c5 nitb: Filter ssh packets in pcap
Change-Id: I62a6ae7bd3a84baceb684c26727d2269c86ed023
2017-05-15 14:49:22 +02:00
Neels Hofmeyr 95fd67374c pcap: pass complete filter string to PcapRecorder
Composing the filter in gen_filter() is more complex than we need. We pass the
address and potentially further filter elements separately, and then
gen_filter() has to guess how to combine these. Instead, have just a filter
string to use right from the start, so that the caller has full control (and
full responsibility).

Remove the addr argument, which was only used for filtering.

This is my conclusion of looking at the patches with change Ids
I62a6ae7bd3a84baceb684c26727d2269c86ed023 (PS 1) and
Icbb0f8d2058fa7ebb7f0f731645f9266cacdb120

I62... PS 2 will add the SSH filtering.

The name 'filter' is a python built-in, which is why I chose the argument
name 'filters' instead. It works with 'filter' as well, but let's try to
avoid naming conflicts like that.

Change-Id: Iff7ddf51d3bf0189ce07b488a3dcdcfce6907aba
2017-05-15 14:49:22 +02:00
Neels Hofmeyr bcd4332b6b jenkins-run: clean up also upon error
Change-Id: I949849d85d93cb47c839cd738788cf6f9b22fc12
2017-05-15 14:25:24 +02:00
Neels Hofmeyr e44a0cb0d9 trial: always use abspath
No actual failure known, but using the abspath will prevent further problems
when running a binary in its own run dir and passing trial paths to it.

Adjust test expectations.

Change-Id: Ia2ba876a42eef9122a92645edd1f28ddb56e6be3
2017-05-15 14:25:24 +02:00
Neels Hofmeyr e60df6961c fix name of local var, to not shadow 'suite' module
This caused "UnboundLocalError: local variable 'suite' referenced before
assignment".

Change-Id: Ic9aa383f5cdc9818c5a21b101fb6ddea4132d9fb
2017-05-15 12:23:57 +00:00
Neels Hofmeyr 2123541574 fix: exception outside of trial run should be logged and cause nonzero rc
From an earlier stage of the code, there was still an exception catcher that
makes no sense. Remove it.

Change-Id: I8085318c91b06a3e8f7d3f8cfdd15a99650666e2
2017-05-15 12:23:57 +00:00
Neels Hofmeyr b06485fcad cosmetic: contrib: add 'build-' to script names
Distinguish the scripts that build binaries from the others by adding "build-"
to their names.

Change-Id: I0c3b0c06210f5cb6a936c5e90ebc9167b57eeae2
2017-05-15 12:23:57 +00:00
Neels Hofmeyr c59e079573 contrib: rethink jenkins jobs with build slave approach
Previously, the idea was to run a test remotely via SSH. This has moved to
operating an actual jenkins build slave on the main unit, which simplifies
things.

Also, previously (in the old osmo-gsm-tester), there was a daemon running,
accepting new trials by a directory appearing. Now, we're directly launching a
test run, which simplifies things.

Remove all SSH stuff from the "run" script.

Compose the checksums of the built binary-tars in the actual build jobs. In the
"run" job, only combine those checksums to one file. (Otherwise we would
compose the sums after transmitting over the network to the build slave.)

Instead of storing trials in a common location on the system, just keep one
trial in the workspace per run. After the run, tar up the trial dir to enable
easy archiving of build results.

The common location on the system might still be a good idea if we're one day
going to move to concurrent runs with scheduling of hardware resources, but so
far we're having simple one-shot runs.

Change-Id: I129f5189ce2bebd761bc752b158fb8778b31010e
2017-05-15 12:23:57 +00:00
Neels Hofmeyr 0765098946 checkout dependencies from script
The jenkins jobs used to have multiple source repositories configured, but it
is really cumbersome to add all the SCM links to a jenkins job.

Instead, clone the git sources from the script, as we also do in various other
contrib/ scripts.

This partly duplicates the osmo-ci scripts, but since this code is much more
intended to run on other peoples' jenkins instances, rather keep the osmo-ci
dependency out of here.

Change-Id: Ie05cb9cd2bd884f72cbdbcc96d9c4389ffca7357
2017-05-15 12:23:57 +00:00
Neels Hofmeyr 143ab81aac pcap recorder: include vim indenting rules as comment
Change-Id: Idbc4521053610d55c9c32eff950ed8fa68dda171
2017-05-15 12:23:36 +00:00
Neels Hofmeyr 39b0b89caf log the location of the detailed log file
The idea is that if someone reads the short log and scratches her head, she can
notice that there is a more detailed log available.

Change-Id: Id5f443f1af0dc774d4d4c588a238b786a83e44d3
2017-05-15 12:22:35 +00:00
Neels Hofmeyr 2296883fc3 log: tweak osmo-gsm-tester.py stdout: all_origins=False, src=False
Change-Id: I24a600f67f7a9f3d1235abfceb86f311ffa97746
2017-05-15 12:22:35 +00:00
Neels Hofmeyr 7e2e8f1d65 log state dir
On the log output, clearly show which state dir is used. Since all runs should
use the same state dir, this is important and/or reassuring to see.

Change-Id: Ie92f1879a35d8bb399ee916b7ef72f9ee7d47409
2017-05-15 12:22:35 +00:00
Neels Hofmeyr 619447404f debug suite: add test that deliberately fails
Change-Id: I42ab624f8dd8e91ee62b8eccfa451e01fd3694d0
2017-05-15 12:22:10 +00:00
Neels Hofmeyr 1fe012e6a8 cosmetic: rather have an underscore in pcap_recorder.py
Change-Id: Ib4886cb3fe0cb9c66d170097b98b4d1eb67cab81
2017-05-15 12:43:35 +02:00
Neels Hofmeyr cf0304ff26 cosmetic: cmdline help: add line breaks, fix typo
Fix typo 'patch' -> 'match'

Argparse has a peculiar way of treating the cmdline help strings. Include
explicit newlines in cmdline help, and place a comment explaining the details.

Change-Id: I34a2333c55719986b729b07386bc47314635ee04
2017-05-12 14:34:41 +02:00
Neels Hofmeyr aca2624273 paths: have one common parent dir /var/tmp/osmo-gsm-tester
In the example config and the jenkins scripts, use paths below common parent
dir /var/tmp/osmo-gsm-tester.

1. example: put the state dir in /var/tmp/osmo-gsm-tester/state, instead of in
the config dir like /etc/osmo-gsm-tester.

2. contrib scripts: place trials in /var/tmp/osmo-gsm-tester/trials, and to
move into place atomically, use /var/tmp/osmo-gsm-tester/.prep-trials as
temporary location.

The OsmoGSMTester manual is currently also being updated to setup these paths,
with /var/tmp/osmo-gsm-tester owned by a common group and having group-sticky
as well has group-writable access rules.

Change-Id: I2961e9d1d9b14859b886058b54ffcb36f4d88bc1
2017-05-11 17:39:02 +02:00
Neels Hofmeyr cceb0f56ab readme: add some ssh details
Change-Id: Idb5d254768b52300d99b5a1f11f92e8536aec371
2017-05-11 10:34:03 +00:00
Neels Hofmeyr 47de6b0a24 ofono/dbus: detach from signals as appropriate
In the dbus_connect() code path, return the subscription token.

In the Modem class, use this token to disconnect signal callbacks when an
interface is removed.

Generalize the signal connect/disconnect handling: have one tuple defining all
signals and their handler functions, add generalized loop to attach them. Store
all subscription tokens in a dict of lists, any number of signal callbacks per
general interface name. When an interface is announced to be removed from
ofono, detach all signals for that interface implicitly.

So far this only handles the MessageManager interface's IncomingMessage signal,
but others will likely follow soon.

Fixes: OS#2242
Change-Id: I0939ef414bc599ee8742df48da04d8d9569d00ba
2017-05-11 12:33:00 +02:00
Neels Hofmeyr 803d87c2e8 util: simplify listdict
listdict came into the code base from some other python code of mine, for no
apparent reason: it is actually not used here at all. However, an upcoming
patch will use a dict of lists.

Also, the listdict implementation is convoluted/complex (to allow accessing
keys as direct object members, which we don't need). Simplify the
implementation to be used by I0939ef414bc599ee8742df48da04d8d9569d00ba.

Change-Id: I09adfd128a19c6c5ba36aae1d4cab83dbd07e0fb
2017-05-11 12:33:00 +02:00
Neels Hofmeyr a6278b7fd0 cosmetic: lock test: add quotes
Mostly to clarify an empty lock in the end.

Change-Id: If80960b331d30125e97878d52edba9b4a63e87c7
2017-05-11 12:33:00 +02:00
Neels Hofmeyr 2321a19aa5 lock test: make more robust against timing
Remove the stdout printing from the lock_test_help.py, so that there is no
problem with a failing test because one output ending up in stdout before the
other. So only output the current lock from lock_test.sh.

Change-Id: I7fe80a9de88ddee1d66ca5c692948fc9673e4230
2017-05-11 10:31:46 +00:00
Neels Hofmeyr 4f33dcc4df run dir: add a second log file containing a non-debug log
Change-Id: Ic51b8f87abc52cc4af3672ee70045ba08334f62e
2017-05-10 13:08:52 +00:00
Neels Hofmeyr feb56c0a04 readme: format so that dependencies can be copy-pasted to terminal
Change-Id: I455cb75d6cdd583d31ab0188476bee813a2ab789
2017-05-10 13:08:45 +00:00
Pau Espin 13143bc4df Log network activity using tcpdump for nitb interface
Change-Id: I4c5d0e2d9857160f905e743517e744f1a06368af
2017-05-10 13:48:31 +02:00
Pau Espin ecf107983b suite: Terminate processes in LIFO order
Usually the first process we started is the one we want to stay until
the end.

Change-Id: I08ea01a42af68191a659fdf8173e3fec9b1e1cfd
2017-05-08 16:57:53 +02:00
Pau Espin e39c6f1a52 Prepend LD_LIBRARY_PATH instead of overwritting it
My current distribution ships a newer libcrypto and libssl which are not
ABI compatible with the ones generated by Jenkins. I had to copy those
libraries locally and use LD_LIBRARY_PATH to be able to run binaries
compiled coming from the jenkins slave. Without this patch I am not
able to run it because it is overwriting the previous variable.

Change-Id: Id9b16d13d343616cbf87b9da8a99e3fae48da6bd
2017-05-08 16:34:30 +02:00
Neels Hofmeyr a88b0c7424 propagate Timeout class to test scope, use in debug suite
Change-Id: Idad34fe799bc8b8e03d773898473773656b005bd
2017-05-08 13:06:58 +02:00
Neels Hofmeyr 1ffc3fed6b on timeout, raise dedicated Timeout exception
Change-Id: I085a52194dde0f5e6dd845ffde1197f7dc83b306
2017-05-08 13:06:58 +02:00
Neels Hofmeyr d1bedb1160 selftest: suite_test: ignore line numbers
Change-Id: Ideab363753559ba0b49352dc9da03a27d7b0dadf
2017-05-08 13:06:07 +02:00
Neels Hofmeyr 67cea22236 add debug suite
Change-Id: I818d130d01ed26e38b5f33341de3727e9f17ca52
2017-05-08 11:04:26 +00:00
Neels Hofmeyr cccbe5987d improve logging for resource allocation
Log on level 'log', more clearly show whether it's for reservation or actual
use, show the origin that is asking for them.

Change-Id: I3b78c7bdcaec90943900343c878099160f8d2f64
2017-05-08 12:50:42 +02:00
Neels Hofmeyr 56b3d7e685 defaults.conf: use only TCH/F instead of dynamic channels
Let's opt for the oldest/simplest case by default.

Change-Id: I89d634cc51e13bdf6ec157ffb64baa80a469f4c8
2017-05-08 10:43:59 +00:00
Neels Hofmeyr f31c545f48 log.py: add a log target option to omit the list of deeper origins
Change-Id: I8a0a508ee4aeffe940be906de0e431740653a08c
2017-05-08 10:12:25 +00:00
Neels Hofmeyr fd7b9d040d run dir: write detailed log file
For each run on a trial, create a log target that logs to a file in the run
dir. Set all levels to DBG in that log target.

Related: OS#2206
Change-Id: Ie7279aeaf32950f85d4145abdc917024003d1d99
2017-05-08 10:12:25 +00:00
Neels Hofmeyr 532126a725 log: make 32 the default origin_width.
Change-Id: I1159395251332f3b1af3b3a322e7191559105faa
2017-05-08 10:12:25 +00:00
Neels Hofmeyr 8f4f17437d log.py: add FileLogTarget
Will be used in a subsequent commit.

Change-Id: Id3dfdeea236eb8ade5e6c80e64d5c3ce4de96b81
2017-05-08 10:12:25 +00:00
Neels Hofmeyr 1a2177cd1b log.py: LogTarget: return self for chaining modifiers
For example this allows
  tgt = LogTarget(foo).set_all_levels(bar).style_change(baz)

Change-Id: If168cc06f644bde6183f3bc51e394c7705386b3e
2017-05-08 10:12:25 +00:00
Neels Hofmeyr f816688279 log.py: tweak LogTarget list, require explicit LogTarget
Rename log_sink to log_write_func to make it more clear.

Remember the list of log targets as static member LogTarget.all_targets.

Make each LogTarget instance register with the list of targets implicitly.

No longer create a default log target, rather create one explicitly in
osmo-gsm-tester.py.

Change-Id: I5844485eaed536cb34b44bfe23dc635fe1982dcd
2017-05-08 10:12:25 +00:00
Neels Hofmeyr f9de78fc58 improve doc and usability of osmo-gsm-tester.py -t option
Allow exact matches by prepending a '='.
Properly document the -t option.

Change-Id: I61bed7e05f0c1f08cf3de72f62d9ba6609d44aee
2017-05-08 10:12:10 +00:00
Neels Hofmeyr f49c7daa81 sms: log info as dbg
Change-Id: Ia9ae0fa63a96a541e7d66cf0d8a9032b135760e6
2017-05-08 10:12:00 +00:00
Neels Hofmeyr 2e41defdd4 sms: don't log info
Change-Id: Ie9c4757de1eedcbcc85b7b99b94d164cc6ae3c59
2017-05-08 10:12:00 +00:00
Neels Hofmeyr 9b90770716 Resource.find: allow returning empty instead of raising
Add flag raise_if_missing, and if False, instead of raising an exception,
return an empty list for that kind of resource. This makes sense for a caller
that requests a single resource.

When finding a single resource to use in ReservedResources.get(), use this to
raise a more adequate exception message if none was found.

Change-Id: Ia296ea68a787bede037a6cea38563b570fb0766e
2017-05-08 10:11:52 +00:00
Neels Hofmeyr 2fade33e49 doc: Resources.find: add API doc + inline comments
Change-Id: Ib6376794e28f8faff82517af99dfb38ab4689ac8
2017-05-08 10:11:51 +00:00
Neels Hofmeyr acf0c937e7 fix prompt()
The prompt() is useful for supervisor (user) interaction during tests.

However it had numerous problems:
- closed stdin, so second prompt() didn't work
- no editing
- no utf-8 multichar
- unflexible poll interval (poll often to stay responsive to input)
and unrelated:
- stdin was hijacked by subprocess.Popen

Firstly pass stdin=PIPE to all subprocesses to leave the tester's stdin
untouched.

Secondly use python input() to read the user entry (instead of mucking about
with the stdin fd), and import readline for history and editing features.

The old approach was put in place to allow polling DBus and processes
regularly. Instead, allow this by running input() in a separate thread while
polling regularly and slowly in the main thread.

The prompt code is now simpler, cleaner and works better.
Will be used in the upcoming 'debug' suite.

Change-Id: I580aca52cd038b59418055259d0d09e9aab49124
2017-05-08 10:10:03 +00:00