Commit Graph

29 Commits

Author SHA1 Message Date
Oliver Smith 033c6ab1be */jenkins.sh: set -e after setting clean up trap
Abort the script and trigger the clean up script, whenever any of the
commands below to prepare the testsuite are failing. This saves time
with figuring out why suddenly all or most tests are failing, and avoids
running the entire testsuite on jenkins if it's obviously not going to
work.

Related: OS#3208
Change-Id: Ie68da2affda8c96b3a515a857a921a05f1bf8ef7
2021-02-26 09:32:34 +01:00
Oliver Smith 12b961942a jenkins-common.sh: add common clean up trap
Add set_clean_up_trap() in jenkins-common.sh and run it at the beginning
of the jenkins.sh files. Move the common clean up code from the end of
every jenkins.sh file into clean_up_common(), which gets called by the
trap. Add a custom clean_up() function to those jenkins.sh files that
need additional clean up.

Replace explicit container stop commands (for containers attached to the
docker network) with one call to network_clean() in clean_up_common(). It
kills all containers attached to the docker network.

The motivation for this change is the upcoming optional build of initrd
and kernel during ttcn3-ggsn-test/jenkins.sh. After building these, a
short smoke test will be performed to make sure we can boot the kernel
and initrd, before continuing to run the entire testsuite against it. If
building or the smoke test fails, we must do a proper clean up of the
network and fix permissions.

Related: OS#3208
Change-Id: I807c6221d22fec31db23e2516e477a3eb43347fb
2021-02-26 09:32:06 +01:00
Oliver Smith 6f6724178b ttcn3: use REGISTRY arg with debian-stretch-titan
Allow jenkins to fetch the image from our private docker registry.
Outside of jenkins, the image is built locally just like before.

Related: OS#5017
Change-Id: I46cc176ea09d8badc359b627d7ce2f459211258c
2021-02-11 16:27:23 +01:00
Oliver Smith 2b38b8e779 ttcn3: move shared run code to script
While I'm at it with tidying up the Dockerfiles, create a
ttcn3-docker-run.sh with shared run code.

Related: OS#5017
Change-Id: Id90769707158f0488eca2313c57b99ea7a4a27c8
2021-02-11 15:45:40 +01:00
Oliver Smith 892d910444 ttcn3: move prepare code to shared script
Move the git fetch/checkout code and make call to build the testsuite,
to debian-stretch-titan/ttcn3-docker-prepare.sh. In the next patch, I
will extend the script to update deps right before building too (e.g.
because OSMO_TTCN3_BRANCH changed).

Related: OS#5017
Change-Id: I4b5bedf058dc527e821f9b7204c632820e671af9
2021-02-11 15:45:40 +01:00
Oliver Smith 61fed0ad39 ttcn3: move initial clone to debian-stretch-titan
Clone the osmo-ttcn3-hacks and all dependency repositories less often by
moving related commands to the shared debian-stretch-titan image.

Remove the 'git checkout -f -B master origin/master' line, because the
master branch is checked out by default.

While at it, move the shared "git config" commands too, and move them
before cloning the repositories, so they don't run again whenever the
deps change (logic to invalidate the cache if deps change will be added
in the next patch).

Related: OS#5017
Change-Id: I2bb142dce061eba4b6a828c4e435510e309989fd
2021-02-11 15:45:40 +01:00
Oliver Smith db3f98113c ttcn3: remove /root/projects/git symlink
Remove leftover from old TTCN-3 build scripts, before refactoring ttcn3
Dockerfiles. This line has already been removed in 357ec806 from 2017 for
ggsn-test.

In osmo-ttcn3-hacks.git, this is only referenced in the obsolete
bin/install.script (looks like we could remove that, together with the
rest of the bin dir?).

Related: OS#5017
Change-Id: Id23e7fae58ba246916a38aa0a10035d4f67f7588
2021-02-11 15:45:40 +01:00
Pau Espin 1f45fb08ff jenkins-common.sh: Add function to print docker network related parameters
Change-Id: Ie00561de7a4494065156a124565e2190151e6019
2020-08-27 12:49:43 +02:00
Harald Welte 306a51dd30 Enable IPv6 in docker networking
We previously only allocated IPv4 sub-nets to the network segments
created with 'docker network'.  Let's fix that by assigning both
IPv4 and IPv6 address ranges to each docker network.

Related: OS#4700
Change-Id: I8802208fddcce1ffa57e5626575d23d02b320d99
2020-08-13 15:01:58 +00:00
Neels Hofmeyr 04c20edefd collect core dumps: '--ulimit core=-1' and WORKDIR=/data everywhere
Change-Id: Ief73b53ed9da6f5af82975bc36d851277d5d3185
2020-06-21 20:58:22 +00:00
Oliver Smith 61f2be7bd4 Fix git checkout for branches and commits
Replace 'git checkout -f -B $BRANCH origin/$BRANCH && \' in all
Dockerfiles that accept branch variables ($OSMO_TTCN3_BRANCH,
$OSMO_MGW_BRANCH, ...) with the following:
	git checkout $BRANCH && \
	(git symbolic-ref -q HEAD && git reset --hard origin/$BRANCH || exit 1); \

This allows using branch names and commit hashes in the $BRANCH
variables. Using commits is needed for the bisect script added in [1].

The second line ("(git symbolic...") checks if we are in detached HEAD
state after the checkout, and if we are not, pulls in all new commits
from origin/$BRANCH. Note that it ends in ';' instead of '&&', because
the command in the next line should be executed even if
"git symbolic-ref" does not exit with 0 (detached HEAD state).

Here is an example, to illustrate that the new command does the right
thing. Clone a repository and be 50 commits behind origin/master:
$ git clone "https://git.osmocom.org/osmo-mgw"
$ cd osmo-mgw
$ git reset --hard origin/master~50

With BRANCH="master":
$ git checkout master && \
  (git symbolic-ref -q HEAD && git reset --hard origin/master || exit 1); \
  echo "done"
Already on 'master'
...
done
$ git status
Your branch is up-to-date with 'origin/master'.

With BRANCH="85978d":
$ git checkout 85978d && \
  (git symbolic-ref -q HEAD && git reset --hard origin/85978d || exit 1); \
  echo "done"
Note: checking out '85978d'.
...
done
$ git status
HEAD detached at 85978dad

Related previous changes:
* [2] made it work for commit hashes, but broke using branch names other
      than master, and pulling in new commits from master
* [3] made branches other than master work again, but did not fix
      pulling in new commits from master
* [4] reverted [3] and the git checkout related part from [2]

[1] Change-Id: I11f7e61a9b30d58a0fdfcaf77dde447806bf661f
[2] Change-Id: If3bc5fae07cf63c4fef68306be4c4328660bc869
[3] Change-Id: I2ff745c8d19b777d876170d5717c082ceb68a1f3
[4] Change-Id: Ie6da0f9ea96f11407e38545a6b3cf22ef9cadc25

Related: OS#4015
Change-Id: I4004980baf0b7d6096702b6f3067ccbdb369a28c
2019-05-22 11:18:56 +02:00
Harald Welte 06ac6e455e Revert "Simplify git checkout, allow branches and commits"
Unfortuately this commit will check out *local* master (i.e. the
previous checkout) rather than the origin/master as it's supposed to
be. This means that ever since merging this patch, our ttcn3 tests
were running some "undefined" stale versions and not current master.

This reverts commit 26565bb729.

Change-Id: Ie6da0f9ea96f11407e38545a6b3cf22ef9cadc25
2019-05-21 17:53:11 +02:00
Oliver Smith 26565bb729 Simplify git checkout, allow branches and commits
Replace the following statements:

a) "git checkout -f -B $BRANCH origin/$BRANCH"
b) "git checkout -f -B $BRANCH $BRANCH"

with:

c) "git checkout -f $BRANCH"

Regarding a), we don't need to specify 'origin/' for each branch, since
we are cloning the repositories in the same Dockerfile, and therefore we
know for sure that there is only one remote and branch names won't be
ambiguous. Removing the 'origin/' allows to put commit hashes into the
branch variables (like done in the new bisect script [1]).

Version b) does not work with branch names:
$ git checkout -f -B osmith/check-imei-before-lu osmith/check-imei-before-lu
fatal: Cannot update paths and switch to branch 'osmith/check-imei-before-lu' at the same time.
Did you intend to checkout 'osmith/check-imei-before-lu' which can not be resolved as commit?

New version c) works with both commits and branches, and it is shorter.

[1] Change-Id: I11f7e61a9b30d58a0fdfcaf77dde447806bf661f

Change-Id: I2ff745c8d19b777d876170d5717c082ceb68a1f3
2019-05-07 10:24:52 +02:00
Harald Welte cffe1022b5 print branch name and git commit hash during docker container build
This is quite useful when looking at jenkins logs to know which exact
version was built at the time.

Change-Id: Id52c382b454e2beecf46820752aeff15b2c1a0ae
2019-03-29 16:40:40 +00:00
Oliver Smith 7ed7a2bdd2 Fix "'laforge/debian-jessie-build' not found"
Instead of hardcoding laforge's username in all FROM statements in the
Dockerfiles, make use of the USER variable (as passed through by the
"make/Makefile" with "docker build --build-arg USER=..."). Thanks to
fixeria for proposing this fix!

This requires running docker-ce, old versions of docker (such as the
one in the official repositories of the latest Fedora) don't support
variables in the FROM line. But docker-ce can be installed after
adding docker's 3rd party repositories.

Closes: OS#3457

Change-Id: Ic5f11c8a4e247f632cb6aea6d147e94c53e0130f
2018-09-06 15:17:06 +02:00
Pau Espin 1694453aee ttcn3-nitb-sysinfo: Remove input logfiles after merging them
All other ttcn3 projects do it.

Change-Id: Iee0e96d3832d88f2ee5958158b372243441750b3
2018-07-06 16:10:16 +02:00
Pau Espin 618505f98f ttcn3-*: Merge logfiles no matter resolution of tests
Previous to this commit, log messages were not being merged if tests
were resolved as failed (test-suite.sh ending with exit code != 0),
which can happen if at least one test failed.

Change-Id: If293fc2d3182ef2a7b997faa8b41129a9dd89c45
2018-07-06 14:50:47 +02:00
Neels Hofmeyr 18392ee604 jenkins.sh: stay in the jenkins workspace, not in /tmp
On jenkins, place all logs and manage docker volumes in the workspace instead
of a /tmp/* dir. Use $WORKSPACE/logs as docker volume base to begin with, thus
there needs to be no copy from /tmp to $WORKSPACE/logs.

On non-jenkins runs, place all in a /tmp/* dir still, but also skip copying of
the logs: just have a /tmp/logs symlink to the last tmpdir.

Change-Id: I8cf6014725ae8ba602be5f3ec31dfb8e49ff993e
2018-05-02 09:36:57 +00:00
Neels Hofmeyr 2f61bf81e9 jenkins.sh: have one common function to collect logs
To remove code dup and prepare a change to where logs are written, add
collect_logs() to jenkins_common.sh and call that from each jenkins.sh after
the tests are done.

The 'rm -rf' is already done before a test starts. No need to do that again
after each test.

Change-Id: I5d8472ec36b07c828685b1bd7718e31392d168a3
2018-04-24 13:46:05 +00:00
Harald Welte 2f5541b134 ttcn*test: use log_merge.sh to generate per-testcase merged logs 2018-03-18 13:33:39 +00:00
Harald Welte 35d2cf3c0f ttcn3-nitb-sysinfo: Fix docker network usage
In d770629280 we broke the
ttcn3-nitb-sysinfo builds when switching to jenkins-common.sh

This commit fixes it.
2018-02-14 21:08:53 +00:00
Harald Welte 10305f2f8f ttcn3: Add OSMO_TTCN3_BRANCH build-time argument to build non-master branches 2018-02-14 21:57:39 +01:00
Harald Welte d770629280 jenkins: Add shared jenkins-common.sh
the common script contains some copy+pasted parts between the different
jobs.
2018-02-06 20:20:49 +00:00
Harald Welte bcc29baf39 jenkins: prefix all container names with $BUILD_TAG
... to make them unique and to allow for multiple tests run in parallel
without any clashes.
2018-02-06 19:57:24 +00:00
Harald Welte facbb6575e move every 'test network' to its own IP address space
Ideally we would want to launch a group of containers with their own
private network segment and use the same static IP addresses in those
isolated networks.

The stupidity of docker is requiring unique IPv4 addresses even on
isolated (!) networks.  This means we have to manually give each of our
test setups a different subnet, and then we can at least run one
instance that test in parallel to at most one instance of each other
test.

If this weird reestriction about unique IPv4 addresses didn't exist,
we could start any number of test runs in parallel.
2018-02-06 19:08:45 +01:00
Harald Welte 58e97d1f0b ttcn3 tests: split 'make deps' in separate RUN command
This avoids re-downlaoding all dependency repositories each time
osmo-ttcn3-hacks has received a change
2017-12-17 12:20:05 +00:00
Harald Welte 357ec806e1 Move to new TTCN-3 build system/Makefiles
A lof of what the Dockerfiles so far did in terms of cloning library
repositories and calling shell scripts has now been implemented as
part of the Makefiles inside osmo-ttcn3-hacks.git, so we can drop it
here and simply use those.
2017-12-13 10:19:28 +01:00
Harald Welte 5c0fd74c5f sysinfo: add missing .release file 2017-08-20 23:41:55 +02:00
Harald Welte 9c937dd432 add ttcn3 sysinfo container 2017-08-20 23:38:09 +02:00