Commit Graph

1175 Commits

Author SHA1 Message Date
Pau Espin ab78df2902 doc: manual: Write initial Test API section
Change-Id: I86bc9a8a0b5ff50e72c21a4bd8a857830bd84c4c
2020-06-04 20:03:07 +02:00
Pau Espin 166dc10769 testenv: Support test overlaying a directory to look for templates
This way tests which require a very specific config file can override
specific template files used by object classes.

Change-Id: I65d1b1e826d2d430ee83810d998b98d0ccaa07cd
2020-06-04 20:03:07 +02:00
Pau Espin 4ddcdaf62c selftest: template: Test existing overlay dir features
Change-Id: Iff1a4dafcbf2193e95b1fb6e5a6712f1bcdd6309
2020-06-04 17:39:17 +02:00
Pau Espin 315ba57651 testenv: Get rid of 'resources' global variable
Change-Id: Id7576d21d0f9de29dd9bc4211eaf9b9ec634c990
2020-06-04 17:09:16 +02:00
Pau Espin 7e30d84034 process: Improve launch_sync() exception due to process failing
After this commit, in some situations ssh related errors are printed
directly in the exception to quickly find cause of the issue.

Example:
    FAIL: ping.py (5.0 sec) Error: rm-remote-dir(pid=25913): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.110 port 22: No route to host) [trial↪4g:srsue-rftype@zmq+srsenb-rftype@zmq+mod-enb-nprb@6↪ping.py:9↪ping.py↪srsepc_10.42.42.118↪host-jenkins@10.42.42.110↪rm-remote-dir(pid=25913)]

Change-Id: Ia16c7dec96f70d761600ad6a50d9df8382d9c2c8
2020-06-04 16:46:43 +02:00
Pau Espin cf6bec7542 log: Improve log ctx information in log.Error exception
Before, it would show somethig like:
"""
osmo_gsm_tester.core.log.Error: Exited in error 255
"""

Now:
"""
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=24820): Exited in error 255 [trial↪4g:srsue-rftype@zmq+srsenb-rftype@zmq+mod-enb-nprb@6↪ping.py:9↪ping.py↪srsepc_10.42.42.118↪host-jenkins@10.42.42.110↪rm-remote-dir(pid=24820)]
"""

Change-Id: I8873f67a2f3df21c4dd552c92510535bf95e2c9d
2020-06-04 16:24:33 +02:00
Pau Espin 19fe3a1caf process: Fix wrong use of log.ctx(self)
That's not needed and will produce some parent loop detection in
log.find_on_stack() if logging is called under that stack frame.

Change-Id: I4ab7e8977fa9bad5c8956b7c1df1513b27bb5aa2
2020-06-04 16:11:39 +02:00
Andre Puschmann eb7ced1367 ms_srs: sum bitrates for all component carrier
when calculating the dl and ul bitrate, it is required to take the
sum of all active component carriers since they are reported seperately
inside the CSV file

Change-Id: If30ec3f22ce5871f911433e6a6997e9c6e7ca210
2020-05-29 09:55:59 +00:00
Andre Puschmann 346ffd9192 enb: remove freq config for zmq operation
specifiying the rx/tx_freq for the eNB is optional. But when
it's done it needs to be correct. There seems to be an issue
when mapping the freq[0-3] between srsENB and Amarisoft eNB.

So it's safe to just drop the config here altogether.

Change-Id: I70f5776173c2df5894031f68df2f6c3a900b79f8
2020-05-29 09:55:40 +00:00
Pau Espin eddff7c03c ms_{srs,amarisoft}: Support selecting tgz based on run_node's run label
Change-Id: I2c4cab02fa34c7acc7e57c3fec5a42c26d2a9208
2020-05-28 13:13:05 +00:00
Pau Espin b452ed6db1 enb_{srs,amarisoft}: Support selecting tgz based on run_node's run label
Change-Id: Ie1c8ccd2ffc9680bd72159872f3de0d59f1b9b0b
2020-05-28 13:13:05 +00:00
Pau Espin 5f826be20a ms_{srs,amarisoft}: Reuse run_node to hold run info
Change-Id: Ifdea48fed0bfc152131c53b87c34b4514c632917
2020-05-28 13:13:05 +00:00
Pau Espin 1abff4e260 enb: Reuse run_node to hold run info
Change-Id: If396940f154813d42ff3d3364c44b8979179127f
2020-05-28 13:13:05 +00:00
Pau Espin e921995ce7 epc_{amarisoft,srs}: Support selecting tgz based on run_node's run label
Change-Id: I90b8dab8bdc9980a7d2486d890cff183c07f1809
2020-05-28 13:13:05 +00:00
Pau Espin 6e0b6fb074 Support identifying different tgz files based on run_label attribute
tgz files in trials can be categorized in subdirectories, allowing to
select different bianry files at runtime based on the target run node
which is going to run them. This way for instance one can have a binary
linked against libs for eg. CentOS under run_label "centos/" or an ARM
target under "arm", and then use "run_label: arm" on the resource using
it.

Change-Id: Iaf2e97da3aff693395f44f0e93b184d4846cf6da
2020-05-28 13:13:05 +00:00
Pau Espin 2e495749d5 Fix processes run with RemoteProcessSafeExit not receiving stdin
Since the process is run in the background through the wrapper bash
script, stdin was disabled there. By explicitly redirecting the bash
process stdin we make sure it is always able to read from it.

Change-Id: I6cb7979aae0a7457919f353cbeb4c3b78cdd4919
2020-05-27 12:15:08 +02:00
Andre Puschmann 26890f3c31 ms_srs: wait 6s after stopping srsUE
we've only waited when running with ZMQ but in fact we need
to always wait because the UE may need some time to
gracefully exit, i.e. send the detach request and shut down correctly.

this is checked in cleanup() and verity_metrics(), so also before copying back
the CSV metrics. But we only wait once.

Change-Id: I47e46b8ccce41c9a62d2d6866260d22c927e710d
2020-05-26 13:55:39 +00:00
Andre Puschmann 71b430c22f enb: fix UL max rate calculation
don't double the maximum rate for UL traffic in MIMO since
we are only using MIMO in the downlink

Change-Id: I018e8ba8cb822a4bd57b8d0e420dd496172ac229
2020-05-26 13:55:39 +00:00
Andre Puschmann 1a96ecdeb6 ms_srs: allow + operator when specifying CSV columns
this allows to sum dl_brate and ul_brate and perform
the known operations, like max, average, etc. on both
UL and DL bitrate combined

For examples, with this CSV ..

time;cc;pci;earfcn;rsrp;pl;cfo;dl_mcs;dl_snr;dl_turbo;dl_brate;dl_bler;ul_ta;ul_mcs;ul_buff;ul_brate;ul_bler;rf_o;rf_u;rf_l;is_attached
1999;0;1;3400;-67;59;913;27;29;0.50;10432.000000;0;2.6;20;0.0;10848.000000;0;0.0;0.0;0.0;1.0
2998;0;1;3400;-67;59;912;28;29;0.50;2136.000000;0;2.6;20;0.0;1680.000000;0;0.0;0.0;0.0;1.0
3997;0;1;3400;-69;61;909;28;29;0.50;2136.000000;0;2.6;20;0.0;1680.000000;0;0.0;0.0;0.0;1.0

.. one could calculate the average of DL and UL bit rate with:

ue.verify_metric(1e6, operation='avg', metric='dl_brate+ul_brate', criterion='gt', window=test_duration)

and veriy that it's average is greater than 1 Mbit/s

Change-Id: I5c7c80bb107fa6b93b215176e6ebbb5dc8594860
2020-05-26 13:55:39 +00:00
Andre Puschmann 2cfa4a3d3c ms_srs: add max rolling average as metric operation
this allows to calculate the rolling average over a specific
window in time and take the maximum of that

this is useful to get average value for 30s UDP traffic for
example from the UE metrics

Change-Id: I34bbfe08dbc1f27b86c9805f54649d44d697fa18
2020-05-26 13:55:39 +00:00
Pau Espin 9dbdb623a0 process: Log stdout tail when process exits with error
This is useful since remote processes we run under ssh end up merging
both remote stdout and sterr into local stdout.

Change-Id: Ibbfb099a667f21641075faa1858e0b9acd706fd2
2020-05-25 16:45:36 +02:00
Pau Espin 6f31de823e enb_srs: Use RemoteProcessSafeExit instead of RemoteProcess
It is known that sometimes srsENB hangs until it is killed -9, specially
when using ZMQ backend. Let's use RemoteProcessSafeExit in order to make
sure it is killed in an acceptable time (srs binaries use some
preventive sigalarm 6 seconds auto-kill procedure, hence we use 7
seconds) before next test is started and potentially try to re-use the
same ENB and fails due to previous one still running.

Change-Id: I905bd753c7822feccf1c1bb59752698f1d1b85f0
2020-05-25 16:12:09 +02:00
Pau Espin 0d64f16544 Replace RemoteProcessFixIgnoreSIGHUP with RemoteProcessSafeExit
The API was doing far more stuff than its name indicated. Even more
important stuff, like making sure the process is killed at the end with
-9 after ssh connection is dropped.

Change-Id: If043ecab509b34b0922824d73db916196274ec64
2020-05-25 15:52:59 +02:00
Pau Espin d84a8387be 4g: Pass max bitrate to iperf3
Otherwise bitrate 0 (unlimited) does weird stuff, like iperf3 hanging
forever in DL.

Change-Id: I631243cce132923d9024f8161cf96161c51cdae5
2020-05-25 14:23:39 +02:00
Pau Espin f4ab97fb4a iperf3: Support runing test with bidirectional data
Change-Id: Ia6f12c65fd3ea26d412997741cb8685a9646af43
2020-05-25 14:11:07 +02:00
Pau Espin 66ef9457cb config: suites_dir and scenarios_dir are now a list of paths
This allows inheriting suites or scenarios from eg. sysmocom/ dir, while
still allowing to apply new suites and scenarios on top.

Change-Id: Icecdae32d400a6b6da2ebf167c1c795f7a74ae96
2020-05-25 13:33:50 +02:00
Pau Espin 3a0dea682e iperf3: Support using configuring UDP protocol
Change-Id: I0b3545847b01ae3d029881f243fea276f0fa39de
2020-05-22 15:20:27 +02:00
Pau Espin d79e719804 schema: Allow objects registering their own schema types
Change-Id: I998c8674a55531909bfeac420064c3f238cea126
2020-05-21 15:41:14 +02:00
Pau Espin 27b609f4d3 ms_srs.py: Fix typos in comments
Change-Id: Iaa09ed696eb58e63c366f26e94c66f27b5f733bf
2020-05-21 13:21:56 +02:00
Pau Espin e972c9c524 config: Allow setting trial directory in main.conf
Change-Id: Ia4141001d084f690897dbdff5eae6c69ff2e521c
2020-05-12 16:27:33 +02:00
Pau Espin 656952760c Move update_version.sh to contrib/
Change-Id: I26218cd512a739fc26ba04be05f01260152d785f
2020-05-12 14:07:36 +02:00
Pau Espin 1337fb8e48 Move check_dependencies.py to contrib/
Change-Id: I8def867b1042cbb98bb36ff4a473b4cdb92ce122
2020-05-12 14:07:36 +02:00
Pau Espin fceb8e14de doc/manuals: Swap order of schemas and config files
First explain the different config files and directories, later describe
the schemas used in each of them.

Change-Id: Iaf31808a655a5c77a1dfaa155e86d42585130820
2020-05-12 14:07:36 +02:00
Pau Espin 6c6c0e8599 Cmdline arg -c sets main configuration file (old paths.conf) instead of dir containing it
It has been notified that current configuration system is difficult to
understand and to use, so it has been envisioned to refactor it a bit.
The idea is that the user passes a -c path/to/main.conf file, which in
turn contains whatever osmo-gsm-tester main settings supports (basically
what old paths.conf used to be, plus some files harcoded to the same -c
directory are now configurable through the main configuration file).

Change-Id: Ieca65b71b543c44cfcec8e83efd0fe053c432e55
2020-05-12 14:07:29 +02:00
Pau Espin 0433c9b671 selftest: Introduce scenario_test
This covers some unit tests for scenario module (Scenario class).

Change-Id: I4c80047bb03ae8254c192057007fa7df84478605
2020-05-11 16:53:16 +02:00
Pau Espin 4e6b5077d0 Split Scenario class to its own file
Change-Id: Ia029de7ecda4c8dc3d0b4c412e4c9c0a65cf0185
2020-05-11 15:12:07 +02:00
Pau Espin b1d8d30526 esme: fix smpplib imports
smpplib modules are used in several places, so it's not enough to import
them in the constructor since they are only imported for the function
scope.

Change-Id: I6511e28c7ef6aa86e85a7e03aa10a67d87896588
2020-05-11 11:55:48 +02:00
Pau Espin 620d2ee5ba ms_srs: fix numpy import
numpy is used in a few places in the class code so we need to jhave it
available in a bigger scope, while still only loading when needed.

Change-Id: Iea66e623e1c980a62d691e20dacb00df99fdd78f
2020-05-11 11:52:14 +02:00
Pau Espin aebaa3cf31 suites/gprs: Catch unexpected iperf3 result content and print it
Change-Id: I10f7f682da023f5e05e461b73f55cc50c5f36f79
2020-05-11 11:29:46 +02:00
Pau Espin 2a2d846430 obj: Fix objects placing their content in suite rundir instead of test rundir
Change-Id: I421d35473575086b93d5c8db98b909cc6c83d896
2020-05-11 10:56:52 +02:00
Pau Espin 0bd048c620 osmo-gsm-tester.py: Update help description and remove old nope cmdline arg
Change-Id: I879a5c30c185905614c2c5a7d54eb33817c94c31
2020-05-07 19:38:37 +02:00
Pau Espin 06c82aee5c Deprecate envvar OSMO_GSM_TESTER_CONF and remove other ones
Environment variable OSMO_GSM_TESTER_CONF is marked as deprecated and an
error is logged each time it's used. Same feature is available through
"-c" command line parameters, so having the envvar only makes things
more complex for no good reason. It cannot yet be completely dropped
since some environemnt still make use of it. Give some time to users to
adapt their setups.
Other environment variables setting some config apths can be dropped
since they are not being really used in any setup.

Change-Id: I7eb69f870d0dcb5906d45ae067d6bed1aabf5862
2020-05-07 19:09:23 +02:00
Pau Espin 18ff1e6e25 ms_driver: Fix typos after recent refactorings
Change-Id: Id9709743648d60792ec64c97ab1bc6811c4e3e1e
2020-05-07 17:10:51 +02:00
Pau Espin fa02b9c677 suites: 4g: Parametrize handover.py test
Change-Id: I7de5acdaa6b905e52655786758bd51490add1d2b
2020-05-07 15:06:01 +00:00
Pau Espin 306373027a Allow suites to dynamically register schemas so tests can receive parameters
Change-Id: Idbe99a35993d193cd97059feb980e61ff14c67ad
2020-05-07 15:06:01 +00:00
Pau Espin 94e7ef0f6c Makefile: Implement clean target
Change-Id: Ia21b0f184bfd454b3835774949f581a27860a80a
2020-05-07 13:27:55 +00:00
Pau Espin 3135a1ae7e Makefile: Add manual target to build user manual
Change-Id: I61ff89db5666248de47cb9752a2aecdd6fd06f02
2020-05-07 13:27:55 +00:00
Andre Puschmann d27b3ba2e2 srs_{enb,ue}: don't set sc12 as OTW format
Change-Id: I7fc20d998b082f5c37b2451f1900161f997954b1
2020-05-06 20:21:18 +00:00
Andre Puschmann 549826d424 enb: add rf_port to cell list param
we need to make the rf_port a param in the cell list because for CA with
two cells on different EARFNCs, the RF port is different

for two cells without CA on the same EARFCN the RF port is the same

so we need to be able to configure it

Change-Id: I32e83cc34811588757104c5ce1cae7277402addd
2020-05-06 20:21:18 +00:00
Pau Espin e0b899079c schema: Allow keys to start with a number
YAML allows it, and it will allow suites tarting with a number on its
name (like the '4g' one) to register its own schema on next commits.

Change-Id: I64e5a9d6604085d3b17eba30498a5e7a66242cc8
2020-05-06 17:14:46 +00:00