Commit Graph

140 Commits

Author SHA1 Message Date
Max 735d5b4165 ctrl2cgi: remove unused function
Change-Id: I1bff680b301b06e3b178d59bdcb86c4773ad9296
2018-11-26 15:04:35 +00:00
Max 9fa695d714 ctrl2cgi: explicitly ignore arguments to lambda function
Make sure that we don't fail regardless of number of arguments given.

Change-Id: I44035323b70f04eb8f5dc12123cb708f53eba188
2018-11-26 14:41:58 +01:00
Max f2199446b4 ctrl2cgi: log traps via debug facility
Change-Id: I21ff593be420a374a00073953c9254a013c43164
Related: SYS#4399
2018-11-26 13:31:00 +00:00
Max c16fa3c750 ctrl2cgi.py: log address on errors
Change-Id: I80cdcda134766c679c2b80b848fb9db853321cfd
Related: SYS#4399
2018-11-22 18:45:21 +01:00
Harald Welte dbb57eb57d make osmpy IPA code aware of RSPRO
RSPRO (Remote Sim PROtocol) is used by osmo-remsim and basd on the IPA
multiplex.  Let's add knowledge about this stream identifier to osmopy.

Change-Id: I5c0e674390c5672ee1083b1226c206557f6a980b
2018-10-13 13:15:51 +02:00
Pau Espin 1920276f75 osmo-ctrl2cgi.service: Fix exec path not absolute
systemd errors due to executabl path not being absolute:
systemd[1]: [/lib/systemd/system/osmo-ctrl2cgi.service:7] Executable path is not absolute, ignoring: python3 /usr/bin/ctrl2cgi.py
systemd[1]: osmo-ctrl2cgi.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

Change-Id: I8e3d695951fb36d53c112b6dce10a275b757ca5a
2018-09-25 11:30:19 +02:00
Neels Hofmeyr e0325b432b osmo_interact/common: ignore trailing empty lines in received results
If the VTY/CTRL output contains trailing newlines, do not cause those to
trigger mismatch errors, rather drop those.

Rationale: in transcript scripts, the amount of empty lines between expected
results is ignored, to allow cosmetically arranging into sections. So trailing
newlines are dropped, and there is no way to preserve those.

Change-Id: I1930868451c5738b17e318a9807590d948210b9a
2018-09-10 17:54:24 +02:00
Neels Hofmeyr b0621451e3 osmo_interact/vty.py: fix parsing of vty logging
Because the VTY logging target currently prints '\n\r' instead of '\r\n', the
python splitlines() separates those as interleaving empty lines. Ignore '\r' to
allow verifying VTY logging output as well.

Change-Id: I8de9cebaa9aad275f65c3672985b7cbca343b5a6
2018-09-10 15:50:08 +02:00
Pau Espin 4c8a0bdc39 ctrl2cgi: Add systemd service file and install with debian
Change-Id: I062ae5b9ed09f52b27c7d85b31daba0d14751846
2018-08-29 19:13:53 +02:00
Pau Espin f1f8285b79 debian: Add python module deps for python3-osmopy-utils
Change-Id: I97adcb156205389a75d13a53ecab5638c4672436
2018-07-12 11:12:29 +02:00
Pau Espin 10fbb40b24 Introduce script ctrl2cgi.py
This scripts implements a similar proxy to the one already available in
soapy.py, but aiming at converting CTRl into a regular HTTP POST Request
to be handled by a CGI script in the HTTP server.

Furthermore, it includes several improvements over the soapy.py script,
such as using a token to authenticate users and capability to read some
parameters from a cfg file.

Change-Id: I7a83ecb379835e295c8cb4c27c39883783746544
2018-07-11 22:47:54 +02:00
Pau Espin 8739f9c4fc Move twisted_ipa.py to osmopy lib
twisted_ipa is a module expected to be used by other python scripts, so
move it to osmopy dir to be installed as a lib (in python path) so it
can be later used.

twisted_ipa has some features to use it standalone for testing purposes.
This feature can still be used from new installation path.

Change-Id: I33a0c4263297abd9e111bf43e6e9967c06474318
2018-07-11 14:05:28 +02:00
Harald Welte 8cd233a826 Add debian packaging rules
This allows us to package osmopy as debian package.

The package currently includes both the library as well as the
executables/scripts.  It might make sense to separate that into
a pure library package and an "utils" package

Related: SYS#3028
Change-Id: I332fb82d0e8dd09c5cf53caafa6a63297a8e3881
2018-06-26 15:58:45 +02:00
Max 901f5eb259 Fix install in the presence of old modules
If older incompatible version of osmopython is already available, it
might be chosen fori mport instead of current version. Fix this by
explicitly prepending the proper version to path.

Change-Id: Icbe2af1e3815406213be29e0c0360432dc9fd6fb
Related: OS#2821
2018-01-15 14:24:15 +01:00
Max c41b751dec Print version before tests
This helps with debugging of import-related issues - we know the version
under test before the test has a chance to hang.

Change-Id: If13cba60a19e9c15885355f85def4d134fa37993
Related: OS#2821
2018-01-15 14:10:27 +01:00
Max cd5a6e419e Fix absolute import issue in py3
As of 577f2a95e4f01c58a0a4f4ccb3b70d9c048b626e in osmo-ci, the
contrib/jenkins.sh isused forinstallation. This causes the issue with
python3 because test coded use absolute import by default.

Fix this by adding relative path and import from ../osmopy to make
sure test code uses the current module and not the one which might be
already installed in the system.

Change-Id: I8ac3c0d45fb2e1d18646048703ac405be1c7e539
2018-01-14 18:33:26 +01:00
Max 34d2ca5bc8 Add rate counter dumper
This simple tool dump all the rate counters available via ctrl interface
to cvs file (or stdout).

Change-Id: I12b369434e4ee9b38f92872f297e1f3d4f0698c2
Fixes: OS#2550
2018-01-12 16:43:37 +01:00
Neels Hofmeyr 58bd53c87b cosmetic: setup.py: move osmo_interact_* next to osmo_verify_*
Those scripts are semantically closely related, so place them next to each
other.

Change-Id: Id7380f9d536c83ee1d612457af423dde9b96f5ad
2018-01-09 12:56:20 +01:00
Neels Hofmeyr 56d08ae735 cosmetic: setup.py: tweak indenting, add vim comment
We usually do only space indenting in our py code, replace tabs with spaces.

Change to one script path per line for easier patch reading in the future.

Add a vim expandtab comment.

Change-Id: I8a501715127c2583a630b1efb0d5316574eea173
2018-01-09 12:54:43 +01:00
Neels Hofmeyr f949f8a5a0 remove osmodumpdoc.py and osmotestconfig.py from py3 install
Install these scripts as py2 only.

These scripts were added to py3 in error by commit "Improve python3
compatibility", I80e5850a8978d78cda793e2192ef4bd3fd54a121, which only addresses
print() compatibility but not encoding.

Recent patch Ief11b75acc973dc7aeaac9c2298a21dc5390efe4 removes osmotestvty.py
from py3 installation. osmodumpdoc.py and osmotestconfig.py are still py2 for
the same reasons and are also likely to fail when run as python3.

Related: OS#2821
Change-Id: Ibf8fc488ae2c022f75f0ca63483d80ed701e8c40
2018-01-09 12:43:44 +01:00
Max 99f1c4714c Remove osmotestvty.py from python 3 installation
This script is actually not python 3 compatible yet despite passing
current tests. Let's not install it for python 3 until we can figure out
a way to catch this incompatibility with test code.

Change-Id: Ief11b75acc973dc7aeaac9c2298a21dc5390efe4
Related: OS#2821
2018-01-08 12:15:55 +01:00
Max 2f266e0cec Improve python3 compatibility
Use proper print() function to make scripts compatible with both python
2 and 3. This paves the way to deprecating python 2 support altogether.

Change-Id: I80e5850a8978d78cda793e2192ef4bd3fd54a121
2018-01-05 10:15:47 +00:00
Max 566f2a7590 Update ctrl command parsing for python3
* make parse() return command id in addition to variable name and value
* introduce parse_kv() wrapper which ignores that id and use it instead
  of old parse()
* make parse() compatible with python3 where we got bytes, not string
  from the socket so we have to decode it properly before using split()
* expand test_py3.py with simply asyn server which verifies that
  osmo_ctrl.py works properly

Change-Id: I599f9f5a18109929f59386ab4416b8bfd75c74d1
2017-12-28 19:34:48 +01:00
Max 8a02e36575 Use python3 for osmo_ctrl.py
It's a standalone script illustrating the use of ctrl protocol from
python. Since it's not used as a library and nothing depends on it, we
can safely switch to python3.

Change-Id: I2461dd9af67771beed5306116e8a1b0ee2285aa8
2017-12-21 14:37:23 +01:00
Max 6588960d1d Add scripts to compilation tests in jenkins
Related: OS#2684
Change-Id: If098c20b1ad17c3868073b9fe07fd3795217c526
2017-12-21 14:35:16 +01:00
Max 379fb9783b Ignore compilation byproducts
Change-Id: Ic04a1b9411b851547ef7ce7a9b562cb290081ca9
2017-12-20 16:05:49 +00:00
Neels Hofmeyr 4c283e4f4a README: mention twisted_ipa.py in scripts section, not as lib
Change-Id: Ie536d61c5af89a530533b22d8ecab1c804a38289
2017-12-20 16:03:47 +00:00
Neels Hofmeyr e5731e03b6 README: mention osmo_interact_* and osmo_verify_transcript_*
Move the libs/scripts that they (possibly) replace below the new additions and
mention that they are superseded.

Change-Id: Ic50e15d6cb6e94b6e171d954d75b94e09b22d548
2017-12-20 16:03:35 +00:00
Neels Hofmeyr be76f4d8a8 add cmdline doc to osmo_interact_* and osmo_verify_*
The tools are so far badly under-documented. Alleviate that by comprehensive
description and examples shown by -h option output. Hint at that from the
README.

Change-Id: I94dcad257558b917cb54fc877122594cd164f496
2017-12-20 16:00:32 +00:00
Neels Hofmeyr 150a6eac9b scripts: add symlink to osmopy
To allow running the scripts from within the scripts/ dir without having to
install osmopy on the system first, place a symlink to the osmopy module dir.
This allows testing modifications without having to run setup.py all the time.

Change-Id: I880c8afa8d02ef112a80cb54f649ec71902bbd4c
2017-12-20 11:11:55 +00:00
Neels Hofmeyr 56aa4785c0 fix osmo_interact_* and osmo_verify_transcript_* after dir split
After I30cdf0f85b2a60a235960911c9827f4129da40db,
* the osmo_interact_{vty,ctrl}.py can no longer import osmo_interact_common,
  since it was moved to scripts/ in error.
* the osmo_verify_{vty,ctrl} scripts can no longer import osmo_interact_{vty,ctrl},
  since it is also in scripts/. Notably, the osmo_interact_{vty,ctrl}.py also
  served as scripts while being modules at the same time, which is not good.

Fix these issues by adding a new osmopy/osmo_interact/ submodule with
osmopy/osmo_interact/common.py, /vty.py and /ctrl.py as modules, and add in
scripts thin wrappers that invoke the modules' main().

Change-Id: I40a37b212274cb70ebb1e1d9d1b3743eb2d64d05
2017-12-19 15:11:31 +00:00
Max 1a5364696e Explicitly check syntax in jenkins tests
By default setup.py ignores compilation errorsand proceed with
installation anyway. Make sure that it fails if any errorsis encountered
bymanually call compilation step.

Change-Id: I2235e3fff21b7c09d3fba00ecdccca7f4515e071
Related: OS#2770
2017-12-18 13:12:48 +01:00
Max 6ccd0785d8 Fix python3 compatibility
Use print() function with proper import.

Change-Id: Ib10dfbec18202245031a634fc3f19677fc952e60
2017-12-18 13:12:48 +01:00
Max 120075a6a5 Move scripts to appropriate directory
Having the scripts in the same directory with library code means that
it'll be installed unconditionally regardless of version check in
setup.py which makes it impossible to write separate tests for python 2
and 3. Fix this by moving the scripts into separate directory and
adjusting init and setup files accordingly.

This is necessary for implementing proper CI tests in follow-up patches.

Change-Id: I30cdf0f85b2a60a235960911c9827f4129da40db
2017-12-18 13:12:08 +01:00
Max c35bb473e4 Expand CI tests
* make sure jenkins.sh fails on any errors similar to other jenkins jobs
* always explicitly use python2 instead of generic python
* add basic module import tests for python 2 and 3
* add comments

Change-Id: I0f4639537d227c513859d4552533ce1e41df9deb
2017-12-18 10:11:39 +01:00
Max 7f7c4b4410 Use setuptools for packaging
According to https://docs.python.org/3/library/distutils.html the
setuptools are used in place of distutils anyway. Using it directly
allows us to make packaging more flexible: specify dependencies,
automatically find package name etc.

Change-Id: I39ee53f352001e47c6df055cbec52d638480253d
2017-12-15 20:40:41 +01:00
Neels Hofmeyr 738acb7b02 add contrib/jenkins.sh from current jenkins.osmocom.org job
i.e. from https://jenkins.osmocom.org/jenkins/job/update-osmocom-python-on-slaves

Change-Id: Iefa0261303982fc66d0845348b9c4a0ff6b94ab5
2017-12-15 04:35:08 +01:00
Pau Espin 1aa0fb4910 osmo_ctrl.py: Set shebang to use python v2
Nowadays bin/python usually points to python3, and this script is written
in python2, which means if run directly from terminal it will fail with
some print syntax errors.

Change-Id: I24a568fc2c3e69e7c9f7b9729bd023fe1f0691c7
2017-12-11 14:02:44 +01:00
Neels Hofmeyr 48b951a7a6 osmo_interact_common: add '... !regex' wildcard
Sometimes it is desirable to match any lines, just make sure that a given regex
does *not* exist in the command reply.

For example, during handover config tests, to verify that no handover config
exists, one would have to match an entire 'show running-config' output, which
is subject to change whenever the VTY changes.

Add a wildcard '... !' which allows passing a regex that must not be
encountered, e.g.:

  OsmoBSC(config-net)# show running-config
  ...
  network
  ... !handover
   bts 0
  ...
    handover 1
  ...

This checks that there is no handover config on the network level while bts 0
does have handover config.

Change-Id: I7c1ebb2e7f059047903a53de26a0ec1ce7fa9b98
2017-11-29 18:21:18 +01:00
Max 751491cdf9 Add osmo_ipa.py to setup's install
When running jenkins job, osmo_ipa is not available despite latest
osmo-python-tests being installed on the build slave. Let's try to fix
it by explicitly adding it to setup.py

Change-Id: Ic3e160c220ca6d2b55df0dfcdcdcb56e4e240bb3
2017-11-27 15:21:17 +01:00
Max e732c2ca3b Add *.py from OsmoBSC
Those are generic enough and can be used as advanced example of uzing
CTRL interface from python on top of osmo_ipa.

Change-Id: Ic4e1155d3bd546feaabab34a46e354c69058056e
2017-11-27 14:07:07 +00:00
Max d05da3e68d Add git-review config
Change-Id: I2359e960eda1f9c941fab22b50ebe56242149c78
2017-11-23 14:46:07 +01:00
Neels Hofmeyr 066a95dec5 cosmetic: cmdline arg doc tweaks, comment
Change-Id: I60dc059879dfe15cd3415ce1ed068ded28ff084e
2017-10-18 03:57:27 +02:00
Neels Hofmeyr 6562c085c4 osmo_verify_*: clarify naming to osmo_interact_*
Have common implementations in one place, and expose distinct command line
argument signatures to obtain the separate tasks as separate scripts.

osmo_interact_common.py implements the parts common to all VTY and CTRL
interactions.

osmo_interact_vty.py and osmo_interact_ctrl.py implement commands I/O but only
expose command line args to directly pipe commands and responses.

osmo_verify_transcript_vty.py and osmo_verify_transcript_ctrl.py act as before,
now implemented by importing osmo_interact_{vty,ctrl}.py, only exposing the
verifification command line arguments.

Change-Id: Ie0cbd5db85ebebc893df638a07f5568632563dc9
2017-10-18 03:54:45 +02:00
Neels Hofmeyr 08d645b01d osmo_verify_transcript: allow running/piping commands directly
To retrieve the VTY reference XML from 'show online-help', which we need for
the osmo-gsm-manuals, it is useful to directly print the output of a single VTY
command.

I have also always wanted to be able to just type CTRL commands and get their
results.

Add --command and --output arguments to run specific commands and pipe to
stdout instead of verifying a transcript file, in a way that works generically
for both VTY and CTRL interface.

The scripts' names are now a bit of a misnomer, to be tweaked separately.

Change-Id: I2c31a87fa255092bcd06da239651cc50174c8faa
2017-10-18 03:43:21 +02:00
Neels Hofmeyr 32cbce1a16 osmo_verify_transcript_vty: automatically obtain the prompt string
If -n is omitted, retrieve the proper prompt name from the initial VTY opening
prompt.

Change-Id: Id5d953156023fcbfc8867c1c3634d787d0b3ac57
2017-10-18 03:42:13 +02:00
Neels Hofmeyr 7b5203f6aa osmo_verify_transcript: add error handling for missing port
Change-Id: I1627c36cfb627e7f3a9b4d9f087bf10875ae0690
2017-10-18 02:09:08 +02:00
Neels Hofmeyr 4e8d6eec5c setup.py: don't install py3 scripts with python2
If one calls 'python2 setup.py install', python replaces the script shebangs
with python2 ones, regardless of the code being mint python3, and vice versa.
In setup.py, name only the scripts that match the current python version.

Change-Id: I84a5c8ec108a99f527b0d2fb5bf4e312df5f7403
2017-10-17 04:06:33 +02:00
Neels Hofmeyr 726b58dcfb add osmo_verify_transcript_{vty,ctrl}.py for easier vty and ctrl testing
While adding VTY and CTRL tests to new programs like OsmoHLR, I wanted to have
a simple way to translate a VTY interaction transcript to a VTY python test. It
is fairly trivial to simply read in a transcript, extract both the commands to
send as well as the expected results, and to verify these without having to
write one line of application-specific code. From there it was just a little
step to allow the same for CTRL interaction.

With osmo_verify_transcript_vty.py and osmo_verify_transcript_ctrl.py, it is
possible to have a simple text file of a telnet VTY or CTRL interface
interaction and run it against a given application. With the --update option,
the scripts run the given command and rewrite the transcript file to whatever
the application currently produces as response. Backed by version control, it
is super easy to tweak commands, --update the test results and verify that only
the desired bits changed. A '...' wildcard can skip any number of lines in the
expected result and is usually preserved during --update.

This python3 implementation is independent from the previous obscvty
implementations.

Take the opportunity to clarify/fix a few aspects: for example, it is now
possible to verify the hints that the interactive VTY displays when the user
enters '?' in various places, and to evaluate the prompt character '>'/'#'.

Unitl now, code is duplicated/scattered across various vty_test_runner.py
scripts in different git repositories. Now, a VTY or CTRL transcript is enough
to put a complete test in place.

The simplest invocation is directly from the Makefile, feeding an application
commandline, the proper port number to contact it and e.g. a VTY prompt name.
This new code is also usable as python modules, to be able to build more
complex tests that require specialized intermediate actions, possibly
coordinating launch of applications or data manipulation.

The first repository to employ this is osmo-hlr.git. See change-ids
I42b3b70a0439a8f2e4964d7cc31e593c1f0d7537 for VTY and
Iff93abe370b8f3ecf42082d1d0eaa1fbeca5b122 for CTRL.

Change-Id: Id47331009910e651372b9c9c76e12f2e8964cc2c
2017-10-17 02:11:58 +02:00
Neels Hofmeyr 92f310500c add osmo_ipa.py, from originally openbsc.git/contrib/ipa.py
At least now that openbsc.git has been split, the IPA and Ctrl API offered by
this file are used across multiple git repositories for python tests. Instead
of having a separate copy of ipa.py in each, have one here.

Rename to osmo_ipa.py to avoid any confusion between old and new files that
might be lying around in the python path.

Change-Id: Iaef955f04b97e8f266339b37b46d2523d1247cb9
2017-10-15 02:57:10 +02:00