Commit Graph

149 Commits (master)

Author SHA1 Message Date
Oliver Smith 92014a44fd tests/test_py3: fix deprecated loop arg
The loop arg has been removed in python 3.10. The same event loop will
be used without passing the argument, so it can be removed.

Fix for:
  TypeError: gather() got an unexpected keyword argument 'loop'

Change-Id: I776bc463246e7a1a9adbb7c06012353d65694507
2023-09-13 16:10:58 +00:00
Oliver Smith f5837cacdb contrib/jenkins: do install --user
Don't install to /usr/local.

Fix for:
  error: can't create or remove files in install directory

Change-Id: Iebed8e3832047e8f19e5a88fcddd1a8242b9f096
2023-09-13 16:10:58 +00:00
Pau Espin 24aa7bf861 Bump version: 0.2.1 → 0.3.0
Change-Id: I56fbc66dd7bbf50a966063a00f8a8e6c583d7d56
2023-09-13 14:48:03 +02:00
Vadim Yanitskiy 706e3d9ba5 osmoutil: return proc's return code from end_proc()
Change-Id: I1e11fcb6c5a587c27fc00920b6e157862d972fd9
Related: OS#5665
2023-06-01 19:59:18 +07:00
Oliver Smith f5e2a84962 systemd: depend on
Related: SYS#6400
Change-Id: Ifbac18a38e086e42c67499b7818cdf08629d0000
2023-05-26 14:10:48 +02:00
Vadim Yanitskiy 36cfd17c82 do not install scripts/{soap,ctrl2cgi}.py
These scripts have been deprecated and are not maintained anymore.

Change-Id: Iffe8f752793127077f25df13167ccc5190e54ff6
2023-05-24 09:03:08 +00:00
Vadim Yanitskiy b40b7bf230 osmoutil: print return code in end_proc()
Change-Id: If11616be6e9afc85814fa0b494e231cc7530459c
Related: OS#5665
2023-05-23 20:26:41 +07:00
Oliver Smith 75ad67f1bb debian: set compat level to 10
Related: OS#5958
Change-Id: Ib45ad7c44cdefa4c5acab2da09f24796253a133d
2023-04-25 16:48:32 +02:00
Vadim Yanitskiy 6dea9111b5 InteractVty: re.escape() the VTY prompt in connect()
The VTY prompt may contain regular expression metacharacters in it,
so the self.re_prompt will not match as expected.  One example is
the modem app in osmocom-bb.git using promot 'OsmocomBB(modem)'.

Change-Id: I47b1f1c1765acb1b935dcbcc54bbb5eb539db4f0
2023-03-25 20:01:48 +07:00
Oliver Smith 95edea9fa7 Bump version: 0.2.1
Tag a new release and to update debian/changelog, with the intention to bring
this package to the latest feed (and not only nightly).

Related: OS#5604
Change-Id: I4cb57c82f4ef28e59495bbe406a042209d210add
2022-07-06 11:26:00 +02:00
Harald Welte 22e3186410 update git URLs (git -> https; gitea)
Change-Id: I9adbf43ae589461c34814157ff41e403dc54091c
2022-06-18 12:31:58 +02:00
Oliver Smith 116cf25fe2 treewide: remove FSF address
Remove the paragraph about writing to the Free Software Foundation's
mailing address. The FSF has changed addresses in the past, and may do
so again. In 2021 this is not useful, let's rather have a bit less
boilerplate at the start of source files.

Change-Id: Ifd7b7e548c9c6db140fa8cbc8a3c8dfaf4603d18
2021-12-14 12:19:36 +01:00
Pau Espin be9f0bba94 osmo_ipa: Fix parsing ERROR msgs in CTRL iface
The UserManual documentation specifies errors to use the "ERROR"
keyword, not the "ERR" one. Furthermore, libosmocore (reference
implementation) sends "ERROR" too.

Change-Id: I82cbe8ad0ab51fa073ac3b2d0b959a1610f683b1
2021-05-18 12:29:22 +02:00
Vadim Yanitskiy 2768e5e923 Use proper python3 shebang in all modules
Change-Id: I1295be505f4ec30e809297a474b689e6c0cdf505
2021-04-04 20:59:35 +02:00
Pau Espin b0c18591b0 README: Drop python2 references
Python2 support was dropped recently, remove references informing how to
install python2 modules (operation would fail).

Change-Id: Iffe403377afe9534dbd61d65bfd52e479e125e68
2021-01-12 18:04:16 +01:00
Harald Welte be7fcf5f28 remove_tmpdir(): Support recursion / sub-directories
The existing code assumed that tmpdir only contained regular files,
not an entire hierarchy of files with sub-directories or the like.

In case sub-directories exist, the current code fails as follows:

line 176, in test_all_apps
  File "/usr/local/lib/python3.5/dist-packages/osmopython-0.2.0-py3.5.egg/EGG-INFO/scripts/", line 142, in remove_tmpdir
    os.unlink(os.path.join(tmpdir, f))
IsADirectoryError: [Errno 21] Is a directory: 'writtenconfig/ericsson'

Change-Id: I896b99032d94ba0cdd340a8eed7c7b625661ad69
2020-07-15 08:53:10 +02:00
Harald Welte 4a7a208ce0 update version to 0.2.0 to mark python2 deprecation
Change-Id: Ie0e0bef3167648ad54e191378dfe170e1493c27e
2019-12-17 12:43:26 +01:00
Oliver Smith 934caf1a81 Drop python2 support (again)
Re-apply reverted commit Iabda95073faa2191fd117e9637e0858c589e9d9e
("Drop python2 support"), but with additional changes to make the
scripts actually work with python3 and to make it build without python2.

I have verified, that the contrib/ scripts of all Osmocom
repositories (with their python3 patches on top) are working with this
patch and that all Osmocom repositories with the python3 patches build
in OBS (tested in own namespace).

All related patches for changing from python2 to 3 in other repositories
must be merged shortly after this one, as soon as the build slaves were
(automatically) updated to have the new osmo-python-tests installed:

New fixes:
* osmopy/ verify: fix compare
  Comparing maps in python3 does not work the same as in python2. Convert
  them to lists first, so the compare works as intended again.
  Fix error:
	  File "/home/user/code/osmo-dev/src/osmo-python-tests/scripts/", line 57, in test_history
	    assert(self.vty.w_verify(test_str, [t1]))
* osmopy/ use enc/dec with send/recv
  Fix error:
		self.socket.send("%s\r" % request)
	TypeError: a bytes-like object is required, not 'str'
* scripts/ use encode() before writing to file
  Fix error:
	File "/home/user/code/osmo-dev/src/osmo-python-tests/scripts/", line 91, in copy_config
	File "/usr/lib/python3.5/", line 622, in func_wrapper
		return func(*args, **kwargs)
	TypeError: a bytes-like object is required, not 'str'
* debian/control: add --buildsystem=pybuild. Otherwise "--with python3"
  is ignored and the build fails if python2 is not installed, with:
  	Can't exec "pyversions": No such file or directory at /usr/[...]/ line 120.

Related: OS#2819
Change-Id: I3ffc3519bf6c22536a49dad7a966188ddad351a7
2019-12-11 12:28:40 +01:00
Oliver Smith b0ba927e33 Revert "Drop python2 support"
This reverts commit fb1dc7c405.

I was under the impression, that all previous scripts were already
working with python 3. But as it turns out, this isn't true. Reverting,
so I can properly post follow-up patches, that fix the issues before we
apply this "drop python2" patch again.

Related: OS#2819
Change-Id: Ic1559d1a9f7839fa86a841d62a04b22e1caed466
2019-12-06 16:06:38 +01:00
Oliver Smith fb1dc7c405 Drop python2 support
Remove all compatibility code for python2.

All scripts are already python 3 compatible since
I80e5850a8978d78cda793e2192ef4bd3fd54a121 and

dpkg-buildpackage shows that it is still invoking with python
in addition to python3, on debian stretch. But after spending quite some
time on trying to convince it to not care about python2 without success
(trying different variables, overrides, --without python2 flags etc.),
I'm leaving it as is. The resulting package is the python3 package, which
is what we want.

Related: OS#2819
Change-Id: Iabda95073faa2191fd117e9637e0858c589e9d9e
2019-12-05 13:41:38 +01:00
Neels Hofmeyr fb4b813d4d fix py3 encoding bug
That code in is hit when invoking via It has a
unicode string already, its attempt to decode hits a python exception (no
'decode' method).

Must be a long standing bug that no-one saw because we're only ever using
the variant.

Change-Id: I1b4a629f12863c498a8681b555f57b4e255cebfb
2019-10-31 05:32:31 +01:00
Pau Espin c6f8d55337 Fix endl removed when pkt buffer content ends exactly on a newline char
As a result, was printing incorrect output:
-        <param name='shutdown' doc='Remove the APN from administrative shut-down' />
-      </params>
+        <param name='shutdown' doc='Remove the APN from administrative shut-down' />      </params

Change-Id: Ib1dbf39db1b27331ea4c39051e550a87780d9f76
2019-08-23 15:54:27 +02:00
Neels Hofmeyr ff0607cbf7 tweak README
Most importantly, mention 'python3'.

Tweak indenting and some wording.

Change-Id: Id8c277de280b54d04edcafa77ed93017d6da473d
2019-07-08 15:19:19 +02:00
Daniel Willmann ee9f2e211b Don't recurse in ctrl_client()
Use a loop instead. Without it the script will eventually crash with a

   File "/usr/bin/", line 211, in conn_client
     await ctrl_client(proxy, reader, writer)
   File "/usr/bin/", line 202, in ctrl_client
     proxy.dispatch(wr, data)
   File "/usr/bin/", line 202, in ctrl_client
     proxy.dispatch(wr, data)
   File "/usr/bin/", line 201, in ctrl_client
RecursionError: maximum recursion depth exceeded in comparison

Change-Id: Ic909e371771f3056cb87e18793fd4225ffb90a2c
Related: SYS#4399
2019-06-26 14:44:46 +02:00
Daniel Willmann f52d4a63d9 osmo-trap2cgi: Remove invalid specifier for /etc
On prod-bscnat the specifier %E (for /etc) does not exist so let's not
use it.

Change-Id: Ia3d8838475c2cc8f9f4ba3a0bc5483c6e1645f8f
Related: SYS#4399
2019-06-19 17:38:05 +02:00
Max 8bf2565b50 deb: fix wrong changelog date
Issue was discovered with lintian - likely a copy-paste error.

Change-Id: Ie73b1f29bd13e68e9b5e05896f1f048bc8108d17
2019-03-26 11:59:19 +01:00
Daniel Willmann a7eda7f113 osmo-trap2cgi: Fix script installation and dependencies
* Install systemd file with debian
* Depend on python3-aiohttp in python3-osmopy-utils as it is required
  for osmo-trap2cgi

Related: SYS#4399
Sponsored-by: On-Waves ehf
Change-Id: I6add5f37b63dd5fcfdefe0459b539e0050d03897
2019-02-04 15:45:23 +01:00
Max 684388f4e1 Add initial version of asyncio trap2cgi script
It uses the same config file format to simplify migration from

Change-Id: I7428cbfbc9f1b80ce42a70be555a38a3497d1cf9
Related: SYS#4399
2019-01-08 15:19:05 +01:00
Max 19b77a426c ctrl2cgi: update shared helper
* split hash generation into simpler functions
* move them into helper file
* split log init into separate function

This facilitates code sharing with new version of ctrl2cgi proxy in
folow-up patches.

Change-Id: Ic5c349349d2c246f934d6701435b1a2d9de77de1
Related: SYS#4399
2019-01-07 17:04:10 +01:00
Max 7230f68311 Mark as deprecated
It might be removed in one of the future releases as we move away from
Twisted. Let's mark it as deprecated to make sure it won't be a

Change-Id: Iefd69e7712d456e3bc42c3277bbfa103a2448bb6
2019-01-07 17:04:10 +01:00
Max 17fe718687 ctrl2cgi: update default parameters in config
Change-Id: I8e013fefaafb3f2417701676a74d499e44b30191
2019-01-07 17:04:10 +01:00
Max 5530d7cc27 ctrl2cgi: explicitly ignore unrelated TRAPs
Previously we've tried to determine the type of TRAP and handle as many
as possible via generic method dispatcher similar to parent class. Let's
make code simpler by explicitly ignoring all but 'location-state'
TRAPs. This should also reduce debug log output because we'll only log
the TRAP variable for ignored messages instead of entire content.

Change-Id: I42b715cb82eb309950ff387649726504b6c7cf61
Related: SYS#4399
2018-12-23 12:45:37 +00:00
Max 66441e73a7 ctrl2cgi: update comments
* remove obsolete comments
* add missing function comments

Change-Id: I5ace594adbccf7ce7eda2c19aa639a4f46285c25
2018-12-21 18:44:01 +01:00
Max c4502377fd ctrl2cgi: account wait and http times separately
Save and propagate separate timestamp for HTTP request duration to
distinguish between time spent waiting on connection limit availability
from actual http request time.

Change-Id: I14d1db5206864ac9ab7c64f96f67139113ab3fcf
Related: SYS#4399
2018-12-21 13:36:36 +01:00
Oliver Smith 95952909fa scripts/*.py: mark as executable
osmo-mgw's won't find, unless it is
executable. This allows passing its ./configure with placing the scripts
dir of osmo-python-tests in PATH, instead of installing it.

Change-Id: If6cefd32d0dc544093db08efc114db86b9bb18b0
2018-12-13 13:12:57 +01:00
Max e5cc253829 twisted: remove low-level logging in dispatchers
The same messages are better handled in derived classes using
message-specific functions instead of generic dispatchers.

Change-Id: I4a054eec9ef7ab39e15ac0cd64834a251d4fb2d7
2018-12-06 10:18:53 +00:00
Max ee3a70ebce Trap handlers: adjust http error log
Log repr() of error because it's more conscience which makes it easier to
read the logs: in case of "e" there's additional
"Failure instance: Traceback (failure with no frames)" prefix which is
pretty useless because we won't have proper tracebacks for deferred anyway.

Change-Id: I39870b83cacee9a589636974d04e356ef0d32345
2018-12-06 11:03:24 +01:00
Max 0028ce1fb5 ctrl2cgi: make http request timeout configurable
Change-Id: I7839121da831aeab0be7e599049bf79471f845fd
Related: SYS#4399
2018-12-05 18:17:33 +01:00
Max 4e95b8f34a ctrl2cgi: log request time
Log http request timestamp and duration on debug loglevel. This is
especially helpful while troubleshooting issues with multiple concurrent
requests under significant load while network issues are present.

Change-Id: I11c8ac67a2730a9c6912694e5b83bbdf08fe357d
Related: SYS#4399
2018-12-05 18:01:10 +01:00
Max 40a375228c Trap handlers: log request/reply as info level
Log the basic location-state trap triggering request and command reply
as info level - that way we can receive useful information without
turning on full debug output.

Change-Id: If8eae9f36292930571954becf6613fada43d301c
2018-12-05 17:58:00 +01:00
Max 7f0f786c27 Trap handlers: propagate expected BSC id to command processor
When receiving commands for particular BSC, log BSC id known at the time
when request was made, not the one which is part of the reply.

Change-Id: I6acdfddb9a1132f978f2b55c769559b0c29eb3e8
2018-12-05 17:55:03 +01:00
Max 7a04bbca59 ctrl2cgi: add config file example
Change-Id: I1e5c37a75cf21184d5bd3ef01b452d45c8f6d7ac
2018-12-05 16:10:17 +01:00
Max 8f8d9144ec ctrl2cgi: properly limit number of requests
Manual acquire()/release() of semaphore does not limit number of
concurrent requests when combined with explicit yield. Fix this by using and removing inilineCallbacks decorator.

Change-Id: I47b8b9f5b726ca0905bb7c023d63b325c7f7d85f
Related: SYS#4399
2018-12-05 16:03:57 +01:00
Max fa80a22563 ctrl2cgi: fix broken config override
Previously command-line arguments without defaults took precedence over
config file variables while values from config file which had
command-line counterparts with default value were silently ignored.

Let's fix this by making config file option mandatory values and
removing overlap between command-line and config file parameters.

Change-Id: I471b5a6497eadce6456e835233fdaba88a593324
Related: SYS#4399
2018-12-05 16:00:29 +01:00
Max f0f8a352cc Trap handlers: always log to stdout
Since the scripts are intended to be used as systemd services, there's
no need in separate logging via syslog: systemd will take care of
properly collecting and storing script output. Hence we can drop extra
options and function parameters.

Change-Id: Ifcad1877d45d43b3a2e617775a1c9b256e190591
Related: SYS#4399
2018-12-05 15:10:18 +01:00
Max 25a8297fe8 Improve code style
* reorder imports to make pylint3 happy
* drop unused imports
* use proper spacing for list constants
* don't use reserved names for internal variables

The check was run as follows:
pylint3 -d C0103,C0301,C0410,C0326,R0913,R0901 ...
to disable useless warnings.

Change-Id: I5b90ee790f73dc509081401776911f25e43f1801
2018-11-28 12:52:48 +00:00
Max f7255fa7cf Move command processing into shared function
Change-Id: I4e40607a9aa5e03a7b3f5b68e4261828209a5813
2018-11-28 12:52:48 +00:00
Max 2cc29b5ef7 Drop unused Trap() class
Change-Id: Ic2066a66f99a059fc65cf3092170e248288c04d4
2018-11-28 12:52:48 +00:00
Max ec3944e7ae ctrl2cgi: fix deferred callbacks
Previously handle_reply() was marked as deferred callback unlike
where it's synchronous function. This seems to be causing issues where
some of the callbacks are not yield properly. Let's move to the
known-to-work semantics of where async functions are limited to
Trap() class.

Change-Id: Ib2c28dd7f79cbd28d475de93750703659ddd18f1
Related: SYS#4399
2018-11-28 12:52:48 +00:00
Max 2a1d8930f6 properly ignore out-of-order traps
Sometimes when we set variable we might receive arbitrary number of TRAP
messages before we get SET_REPLY. Those could be either separate
messages or combined together with SET_REPLY depending on tcp buffering
at server side. Let's handle this gracefully by skipping over all TRAP

An example command which often triggers this behavior:
./ -s -d -p 4249 bts.0.location (date +%s)",fix2d,1,2,3"

Change-Id: Ia6de02c2f13a56f0381c97a9ab02c6c7a31cc32f
Related: SYS#4399
2018-11-28 12:37:23 +01:00