Commit Graph

128 Commits

Author SHA1 Message Date
Pau Espin c6f8d55337 vty.py: Fix endl removed when pkt buffer content ends exactly on a newline char
As a result, osmo_interact_vty.py was printing incorrect output:
"""
-        <param name='shutdown' doc='Remove the APN from administrative shut-down' />
-      </params>
+        <param name='shutdown' doc='Remove the APN from administrative shut-down' />      </params
"""

Change-Id: Ib1dbf39db1b27331ea4c39051e550a87780d9f76
2019-08-23 15:54:27 +02:00
Neels Hofmeyr ff0607cbf7 tweak README
Most importantly, mention 'python3 setup.py'.

Tweak indenting and some wording.

Change-Id: Id8c277de280b54d04edcafa77ed93017d6da473d
2019-07-08 15:19:19 +02:00
Daniel Willmann ee9f2e211b osmo_trap2cgi.py: Don't recurse in ctrl_client()
Use a loop instead. Without it the script will eventually crash with a
RecursionError.

   File "/usr/bin/osmo_trap2cgi.py", line 211, in conn_client
     await ctrl_client(proxy, reader, writer)
   File "/usr/bin/osmo_trap2cgi.py", line 202, in ctrl_client
     proxy.dispatch(wr, data)
[...]
   File "/usr/bin/osmo_trap2cgi.py", line 202, in ctrl_client
     proxy.dispatch(wr, data)
   File "/usr/bin/osmo_trap2cgi.py", line 201, in ctrl_client
[...]
RecursionError: maximum recursion depth exceeded in comparison

Change-Id: Ic909e371771f3056cb87e18793fd4225ffb90a2c
Related: SYS#4399
2019-06-26 14:44:46 +02:00
Daniel Willmann f52d4a63d9 osmo-trap2cgi: Remove invalid specifier for /etc
On prod-bscnat the specifier %E (for /etc) does not exist so let's not
use it.

Change-Id: Ia3d8838475c2cc8f9f4ba3a0bc5483c6e1645f8f
Related: SYS#4399
2019-06-19 17:38:05 +02:00
Max 8bf2565b50 deb: fix wrong changelog date
Issue was discovered with lintian - likely a copy-paste error.

Change-Id: Ie73b1f29bd13e68e9b5e05896f1f048bc8108d17
2019-03-26 11:59:19 +01:00
Daniel Willmann a7eda7f113 osmo-trap2cgi: Fix script installation and dependencies
* Install systemd file with debian
* Depend on python3-aiohttp in python3-osmopy-utils as it is required
  for osmo-trap2cgi

Related: SYS#4399
Sponsored-by: On-Waves ehf
Change-Id: I6add5f37b63dd5fcfdefe0459b539e0050d03897
2019-02-04 15:45:23 +01:00
Max 684388f4e1 Add initial version of asyncio trap2cgi script
It uses the same config file format to simplify migration from ctrl2cgi.py

Change-Id: I7428cbfbc9f1b80ce42a70be555a38a3497d1cf9
Related: SYS#4399
2019-01-08 15:19:05 +01:00
Max 19b77a426c ctrl2cgi: update shared helper
* split hash generation into simpler functions
* move them into helper file
* split log init into separate function

This facilitates code sharing with new version of ctrl2cgi proxy in
folow-up patches.

Change-Id: Ic5c349349d2c246f934d6701435b1a2d9de77de1
Related: SYS#4399
2019-01-07 17:04:10 +01:00
Max 7230f68311 Mark soap.py as deprecated
It might be removed in one of the future releases as we move away from
Twisted. Let's mark it as deprecated to make sure it won't be a
surprise.

Change-Id: Iefd69e7712d456e3bc42c3277bbfa103a2448bb6
2019-01-07 17:04:10 +01:00
Max 17fe718687 ctrl2cgi: update default parameters in config
Change-Id: I8e013fefaafb3f2417701676a74d499e44b30191
2019-01-07 17:04:10 +01:00
Max 5530d7cc27 ctrl2cgi: explicitly ignore unrelated TRAPs
Previously we've tried to determine the type of TRAP and handle as many
as possible via generic method dispatcher similar to parent class. Let's
make code simpler by explicitly ignoring all but 'location-state'
TRAPs. This should also reduce debug log output because we'll only log
the TRAP variable for ignored messages instead of entire content.

Change-Id: I42b715cb82eb309950ff387649726504b6c7cf61
Related: SYS#4399
2018-12-23 12:45:37 +00:00
Max 66441e73a7 ctrl2cgi: update comments
* remove obsolete comments
* add missing function comments

Change-Id: I5ace594adbccf7ce7eda2c19aa639a4f46285c25
2018-12-21 18:44:01 +01:00
Max c4502377fd ctrl2cgi: account wait and http times separately
Save and propagate separate timestamp for HTTP request duration to
distinguish between time spent waiting on connection limit availability
from actual http request time.

Change-Id: I14d1db5206864ac9ab7c64f96f67139113ab3fcf
Related: SYS#4399
2018-12-21 13:36:36 +01:00
Oliver Smith 95952909fa scripts/*.py: mark as executable
osmo-mgw's configure.ac won't find osmotestvty.py, unless it is
executable. This allows passing its ./configure with placing the scripts
dir of osmo-python-tests in PATH, instead of installing it.

Change-Id: If6cefd32d0dc544093db08efc114db86b9bb18b0
2018-12-13 13:12:57 +01:00
Max e5cc253829 twisted: remove low-level logging in dispatchers
The same messages are better handled in derived classes using
message-specific functions instead of generic dispatchers.

Change-Id: I4a054eec9ef7ab39e15ac0cd64834a251d4fb2d7
2018-12-06 10:18:53 +00:00
Max ee3a70ebce Trap handlers: adjust http error log
Log repr() of error because it's more conscience which makes it easier to
read the logs: in case of "e" there's additional
"Failure instance: Traceback (failure with no frames)" prefix which is
pretty useless because we won't have proper tracebacks for deferred anyway.

Change-Id: I39870b83cacee9a589636974d04e356ef0d32345
2018-12-06 11:03:24 +01:00
Max 0028ce1fb5 ctrl2cgi: make http request timeout configurable
Change-Id: I7839121da831aeab0be7e599049bf79471f845fd
Related: SYS#4399
2018-12-05 18:17:33 +01:00
Max 4e95b8f34a ctrl2cgi: log request time
Log http request timestamp and duration on debug loglevel. This is
especially helpful while troubleshooting issues with multiple concurrent
requests under significant load while network issues are present.

Change-Id: I11c8ac67a2730a9c6912694e5b83bbdf08fe357d
Related: SYS#4399
2018-12-05 18:01:10 +01:00
Max 40a375228c Trap handlers: log request/reply as info level
Log the basic location-state trap triggering request and command reply
as info level - that way we can receive useful information without
turning on full debug output.

Change-Id: If8eae9f36292930571954becf6613fada43d301c
2018-12-05 17:58:00 +01:00
Max 7f0f786c27 Trap handlers: propagate expected BSC id to command processor
When receiving commands for particular BSC, log BSC id known at the time
when request was made, not the one which is part of the reply.

Change-Id: I6acdfddb9a1132f978f2b55c769559b0c29eb3e8
2018-12-05 17:55:03 +01:00
Max 7a04bbca59 ctrl2cgi: add config file example
Change-Id: I1e5c37a75cf21184d5bd3ef01b452d45c8f6d7ac
2018-12-05 16:10:17 +01:00
Max 8f8d9144ec ctrl2cgi: properly limit number of requests
Manual acquire()/release() of semaphore does not limit number of
concurrent requests when combined with explicit yield. Fix this by using
semaphore.run() and removing inilineCallbacks decorator.

Change-Id: I47b8b9f5b726ca0905bb7c023d63b325c7f7d85f
Related: SYS#4399
2018-12-05 16:03:57 +01:00
Max fa80a22563 ctrl2cgi: fix broken config override
Previously command-line arguments without defaults took precedence over
config file variables while values from config file which had
command-line counterparts with default value were silently ignored.

Let's fix this by making config file option mandatory values and
removing overlap between command-line and config file parameters.

Change-Id: I471b5a6497eadce6456e835233fdaba88a593324
Related: SYS#4399
2018-12-05 16:00:29 +01:00
Max f0f8a352cc Trap handlers: always log to stdout
Since the scripts are intended to be used as systemd services, there's
no need in separate logging via syslog: systemd will take care of
properly collecting and storing script output. Hence we can drop extra
options and function parameters.

Change-Id: Ifcad1877d45d43b3a2e617775a1c9b256e190591
Related: SYS#4399
2018-12-05 15:10:18 +01:00
Max 25a8297fe8 Improve code style
* reorder imports to make pylint3 happy
* drop unused imports
* use proper spacing for list constants
* don't use reserved names for internal variables

The check was run as follows:
pylint3 -d C0103,C0301,C0410,C0326,R0913,R0901 ...
to disable useless warnings.

Change-Id: I5b90ee790f73dc509081401776911f25e43f1801
2018-11-28 12:52:48 +00:00
Max f7255fa7cf Move command processing into shared function
Change-Id: I4e40607a9aa5e03a7b3f5b68e4261828209a5813
2018-11-28 12:52:48 +00:00
Max 2cc29b5ef7 Drop unused Trap() class
Change-Id: Ic2066a66f99a059fc65cf3092170e248288c04d4
2018-11-28 12:52:48 +00:00
Max ec3944e7ae ctrl2cgi: fix deferred callbacks
Previously handle_reply() was marked as deferred callback unlike soap.py
where it's synchronous function. This seems to be causing issues where
some of the callbacks are not yield properly. Let's move to the
known-to-work semantics of soap.py where async functions are limited to
Trap() class.

Change-Id: Ib2c28dd7f79cbd28d475de93750703659ddd18f1
Related: SYS#4399
2018-11-28 12:52:48 +00:00
Max 2a1d8930f6 osmo_ctrl.py: properly ignore out-of-order traps
Sometimes when we set variable we might receive arbitrary number of TRAP
messages before we get SET_REPLY. Those could be either separate
messages or combined together with SET_REPLY depending on tcp buffering
at server side. Let's handle this gracefully by skipping over all TRAP
messages.

An example command which often triggers this behavior:
./osmo_ctrl.py -s -d 127.0.0.1 -p 4249 bts.0.location (date +%s)",fix2d,1,2,3"

Change-Id: Ia6de02c2f13a56f0381c97a9ab02c6c7a31cc32f
Related: SYS#4399
2018-11-28 12:37:23 +01:00
Max 1c668f2c47 ctrl: add function to skip TRAP messages
This allows to easy skip TRAP messages when we do not want to process
them (for example when waiting for REPLY to a single command). Update
documentation and version accordingly.

Change-Id: I51ce207c19a1ca96c3e2af7d5efd64f79b02fbb4
2018-11-28 12:37:23 +01:00
Max 217837cf55 ctrl: cosmetic cleanup
* remove unused function
* move internal function call outside of main

Change-Id: I3c1bf59775f08f6252d731de653ee9cc212b31da
2018-11-28 12:29:23 +01:00
Max a07e55335d Move Trap class back to separate files
After further testing it turns out that Trap() have too many
implementation details which makes it cumbersome to be shared. Instead,
it's easier to make it into wrapper over shared functions.

Change-Id: I8a3c62bcdf4286f8127c5b6d8dee6d740aca23b9
2018-11-27 18:25:23 +01:00
Max e18d0661b5 Update trap helper
* add missing import
* update docs
* add helper functions

Change-Id: Ie6dc8808efc76ad96b400913e5caa405bce7d970
Related: SYS#4399
2018-11-27 17:41:02 +01:00
Max ac8de71ac7 Re-apply changes to trap_helper.py
Due to changes getting out of sync, changes from
I44035323b70f04eb8f5dc12123cb708f53eba188 and
I21ff593be420a374a00073953c9254a013c43164 were overwritten by
I7b59f2dbded9074d15f2d2f40bf5a92ed02601e2 moving code to different
file. Fix this by re-applying those changes in new location.

Change-Id: I811f307ded63e7e1544243921ee07dceae81e295
2018-11-27 13:14:20 +01:00
Max 5baba8c078 Move common Trap-related code into separate file
The ctrl2cgi.py is heavily based upon soap.py - let's move all the
shared code into separate file to make further modifications easier.

Change-Id: I7b59f2dbded9074d15f2d2f40bf5a92ed02601e2
Related: SYS#4399
2018-11-27 11:51:37 +00:00
Max 98b993f010 osmo_ipa: update split_combined docs
Change-Id: I1cc6dc314fa0f147ab64ad29d9b892c3637a51c7
2018-11-26 19:30:43 +01:00
Max f0aa41e6a1 osmo_ipa: add extended checks
Make sure we properly handle None objects.

Change-Id: If5ddc04b8a5dc26e56e5f5bbec7f28cf0af4a97b
2018-11-26 19:01:30 +01:00
Max 04e7c64902 ctrl: introduce constant for TRAP id
Use it for TRAP message parsing and generation.

Change-Id: I0381eb60d10e7278b966464add79772a9f7c8118
2018-11-26 16:03:49 +00:00
Max 735d5b4165 ctrl2cgi: remove unused function
Change-Id: I1bff680b301b06e3b178d59bdcb86c4773ad9296
2018-11-26 15:04:35 +00:00
Max 9fa695d714 ctrl2cgi: explicitly ignore arguments to lambda function
Make sure that we don't fail regardless of number of arguments given.

Change-Id: I44035323b70f04eb8f5dc12123cb708f53eba188
2018-11-26 14:41:58 +01:00
Max f2199446b4 ctrl2cgi: log traps via debug facility
Change-Id: I21ff593be420a374a00073953c9254a013c43164
Related: SYS#4399
2018-11-26 13:31:00 +00:00
Max c16fa3c750 ctrl2cgi.py: log address on errors
Change-Id: I80cdcda134766c679c2b80b848fb9db853321cfd
Related: SYS#4399
2018-11-22 18:45:21 +01:00
Harald Welte dbb57eb57d make osmpy IPA code aware of RSPRO
RSPRO (Remote Sim PROtocol) is used by osmo-remsim and basd on the IPA
multiplex.  Let's add knowledge about this stream identifier to osmopy.

Change-Id: I5c0e674390c5672ee1083b1226c206557f6a980b
2018-10-13 13:15:51 +02:00
Pau Espin 1920276f75 osmo-ctrl2cgi.service: Fix exec path not absolute
systemd errors due to executabl path not being absolute:
systemd[1]: [/lib/systemd/system/osmo-ctrl2cgi.service:7] Executable path is not absolute, ignoring: python3 /usr/bin/ctrl2cgi.py
systemd[1]: osmo-ctrl2cgi.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

Change-Id: I8e3d695951fb36d53c112b6dce10a275b757ca5a
2018-09-25 11:30:19 +02:00
Neels Hofmeyr e0325b432b osmo_interact/common: ignore trailing empty lines in received results
If the VTY/CTRL output contains trailing newlines, do not cause those to
trigger mismatch errors, rather drop those.

Rationale: in transcript scripts, the amount of empty lines between expected
results is ignored, to allow cosmetically arranging into sections. So trailing
newlines are dropped, and there is no way to preserve those.

Change-Id: I1930868451c5738b17e318a9807590d948210b9a
2018-09-10 17:54:24 +02:00
Neels Hofmeyr b0621451e3 osmo_interact/vty.py: fix parsing of vty logging
Because the VTY logging target currently prints '\n\r' instead of '\r\n', the
python splitlines() separates those as interleaving empty lines. Ignore '\r' to
allow verifying VTY logging output as well.

Change-Id: I8de9cebaa9aad275f65c3672985b7cbca343b5a6
2018-09-10 15:50:08 +02:00
Pau Espin 4c8a0bdc39 ctrl2cgi: Add systemd service file and install with debian
Change-Id: I062ae5b9ed09f52b27c7d85b31daba0d14751846
2018-08-29 19:13:53 +02:00
Pau Espin f1f8285b79 debian: Add python module deps for python3-osmopy-utils
Change-Id: I97adcb156205389a75d13a53ecab5638c4672436
2018-07-12 11:12:29 +02:00
Pau Espin 10fbb40b24 Introduce script ctrl2cgi.py
This scripts implements a similar proxy to the one already available in
soapy.py, but aiming at converting CTRl into a regular HTTP POST Request
to be handled by a CGI script in the HTTP server.

Furthermore, it includes several improvements over the soapy.py script,
such as using a token to authenticate users and capability to read some
parameters from a cfg file.

Change-Id: I7a83ecb379835e295c8cb4c27c39883783746544
2018-07-11 22:47:54 +02:00
Pau Espin 8739f9c4fc Move twisted_ipa.py to osmopy lib
twisted_ipa is a module expected to be used by other python scripts, so
move it to osmopy dir to be installed as a lib (in python path) so it
can be later used.

twisted_ipa has some features to use it standalone for testing purposes.
This feature can still be used from new installation path.

Change-Id: I33a0c4263297abd9e111bf43e6e9967c06474318
2018-07-11 14:05:28 +02:00