As we run more testsuites in parallel, the load was higher and so this
timeout was reached.
Related: OS#5802
Change-Id: I823010106a99a4906e32198ed42b36f9fe662c4e
Put the current SUBNET into NET_NAME, not the previous one. Without this
patch it works too, but this makes more sense.
Related: OS#5802
Change-Id: I6c90505fd6c0ba62244ff67709b0bf31bf44b449
Fix the bug that running a test twice on the same host results in the
second test stopping the containers and network of the first test:
* Have the subnet name in the network name, so there's no collision of
network names when cleaning up the network.
* Don't clean up the network when creating a new network. Rely on the
clean up trap to clean up the network instead.
* Don't set NET_NAME when sourcing jenkins-common.sh anymore, as
network_create will set it. There's also network_bridge_create, which
only gets used by osmo-ran (which we don't run in jenkins). Add a
check to this function to ensure NET_NAME is set before calling it,
which osmo-ran is already doing.
Related: OS#5802
Change-Id: Ifcd384272c56d585e220e2588f2186dc110902ed
Try multiple subnet numbers until successfully creating a network. This
way we can run the same ttcn3 testsuite multiple times in parallel
without conflicts (e.g. once against latest, once against nightly). Also
we don't need to make sure each new testsuite has a unique subnet
number anymore.
I've considered also adjusting network_bridge_create, but that gets used
exclusively by osmo-ran/jenkins.sh, a script which we don't actually run
in jenkins. It seems that in this script it makes more sense to not get
a random subnet number.
Related: OS#5802
Change-Id: I57152b08ef0f38e17e7019a8df032189b03f56cf
Use 'set +x' before echo, so messages are not printed twice and the
output becomes more readable.
Example:
+ echo Creating network ttcn3-bts-test
Creating network ttcn3-bts-test
Related: OS#5802
Change-Id: I213cb80bd1d9401281ae1fb7394b2319453550dd
Use debian:bullseye instead of debian-bullseye-build. Building the
latter requires a lot of time, it's definitively not needed for just
running chmod. Use debian:bullseye, since we use it as base for most
images and it's therefore very likely that it's already downloaded
(and if not, it would get downloaded here).
Change-Id: I88f513a0a8ee04628842a7aa1b663c390d2032cd
Add "--rm" to each "docker run" command, so they don't continuously fill
up disk space.
Fix this even in the pipework script. We don't use the code path there,
but by always having --rm after "docker run" (same line or next line),
a new lint script in osmo-ci I8ab9c291504475d670bdefc50c4524c5bdd4c880
can help us avoid this in the future.
In ttcn3-ggsn-test/jenkins.sh, move one existing --rm in a later line
upwards so the linter can find it.
Related: SYS#5827, OS#5099
Related: https://github.com/jpetazzo/pipework/pull/248
Change-Id: I48b01c43fedf379b8a565eaab0369806d7831bd8
CentOS Linux 8 is EOL, attempting to install packages in it results in
an error. CentOS Linux is a rebuild of RHEL (stable versions). The
CentOS projects recommends to use CentOS Stream instead, which is a
build of the "public development branch for RHEL".
After the early EOL was announced on 2020-12-08, alternative projects
AlmaLinux and Rocky Linux have been established as binary compatible
forks of RHEL 8 (stable versions).
Both Alma and Rocky seem to be solid projects, see related Wikipedia
articles and their sources. Pick Alma and adjust the whole tree to use
the almalinux:8 docker image instead of centos:centos8.
Fix for:
Error: Failed to download metadata for repo 'appstream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Related: SYS#5818
Related: https://www.centos.org/cl-vs-cs/
Related: https://www.centos.org/centos-linux-eol/
Related: https://bugs.centos.org/view.php?id=18394
Related: https://en.wikipedia.org/wiki/AlmaLinux
Related: https://en.wikipedia.org/wiki/Rocky_Linux
Change-Id: I30e1a773b901b1d2187214445116c7f2aecc4e36
After the default UPSTREAM_DISTRO was changed from debian:stretch to
debian:bullseye, the "debian9-repo-install-test" container has gotten
built with bullseye instead of stretch. This is the reason for failures
of the jenkins job Osmocom-repo-install-debian9 we have seen over the
past days.
With this patch applied, it runs through again:
https://jenkins.osmocom.org/jenkins/job/Osmocom-repo-install-debian9/339/
Change-Id: I98a19184ba936114c03cd5cc4f54a3173cbd9cfe
Add missing container for -latest tests. Currently this causes the
-latest tests to fail.
Related: OS#4969
Change-Id: I1230e87784bc21b5a6424db0bd8734181ead9bfd
So far we were using ancient Debian 9 (stretch) for our IUT
containers. Let's upgrade that to Debian 11 (bullseye).
Related: OS#4969
Change-Id: Ic6bece9cb695e6eccfcb1e83fdbf1048724a3cf9
So far we were executing all our TTCN-3 tests from a container
image with Debian stretch (9) plus a custom more recent eclipse-titan
package from the osmocom feed.
Let's update the container base OS from stretch (9) to bullseye (11)
while using the same packaged eclipse-titan version (8.0.0) for running
the tests. So this should be a low-risk change, as titan runtime
remains identical.
I've executed all test suites locally and couldn't see any regressions.
Related: OS#4969
Change-Id: Ib3bdfa3bec8f8ef42c55ca61cdee8fbca923874f
Support running multiple TTCN-3 testsuites in parallel when the
jenkins.sh scripts get called from outside of jenkins too.
Closes: OS#5358
Change-Id: Iaaf5719f758553687f2ab57975be275e7ac07e36
Pull debian-stretch-titan from registry.osmocom.org if the repository is
enabled. Otherwise the image gets only pulled the first time, and does
not get updated once it becomes outdated.
Fixes: OS#5336
Change-Id: I1cf998c21e4ee1f723c3b783703e339328377f3e
Do not build debian-stretch-titan, when registry.osmocom.org is enabled.
The image that would be built at this point is not useful, since other
images like ttcn3-msc-test will use the debian-stretch-titan image from
the registry instead of the one that was just built locally.
Related: OS#5336
Change-Id: I7127e3ebac3a6a985c3ba50ba8c7cb8c5de978d9
Write a line like 'Misc_Helpers.mp_osmo_repo := "nightly"' into the
TTCN-3 config file (e.g. BSC_Tests.cfg), before starting the testsuite.
This allows executing different code paths in the tests based on the
repository.
Related: OS#5327
Change-Id: Ic06532f7a67e59458652c5cf4c8f6fee8113e703
Allow to optionally skip listing the installed packages. This is useful
to speed up tests when running locally.
Related: osmo-dev I3d9976d07f088e2d106d23259889a4bffe1abbcb
Change-Id: Iaafb81e9ee94850ae3b6318574ad5370d74a2838
Since recently, we also have a new IMAGE_SUFFIX '2021q1-centos8'.
We cannot rely on image_suffix_is_latest() anymore, because this
prefix does not qualify as 'latest'. If one needs to enable a
feature that is only available in master, then here is a way:
if ! image_suffix_is_master; then
# Patch the configuration files
fi
Change-Id: I65df66cd2e14934bed8114618d6ffad7831d1025
Adjust docker_depends and docker_dir_from_image_name for 2021q1. Extend
README.md with examples.
Related: SYS#5370
Change-Id: Icf1e3f1928658ff3d82190c4176252cf3590a1f5
Display an overview of installed Osmocom packages at the start of each
ttcn3-*/jenkins.sh script (and others making use of
docker_images_require).
Closes: OS#5091
Change-Id: I62670017d790fd67c6a0d8d464386c0f8e94c1a8
Add a new centos8-obs-latest container with shared code (just like
debian-stretch-obs-latest) and adjust the package install blocks in
osmo-*-latest.
Related: OS#5049
Change-Id: I274b31d4903df07e99fd434af25bdb72991f6923
Move common code from osmo-*-latest to debian-stretch-obs-latest:
* add the OBS repository key
* add the OBS repository with "latest" feed
* install telnet
* invalidate the cache once the repository is updated
This is in preparation to make osmo-*-latest work with centos8,
preventing the following issues we would have without this refactoring:
* avoid need to read the repository index of both debian-stretch and
centos8, and to invalidate the cache if either of them changes
(instead of just the relevant one). This is currently a problem in
osmo-*-master/Dockerfile, Id69f3eaf37f823bac0947366f27a25edd277a4cb
will fix it there.
* no need to add boilerplate code to install the centos8 repository to
each osmo-*-latest/Dockerfile
While at it, also remove deprecated MAINTAINER lines.
Related: OS#5049
Related: https://docs.docker.com/engine/reference/builder/#maintainer-deprecated
Change-Id: I443d52eba416ea298cc5421e280ad2f4ca59283b
Disable KVM if /dev/kvm is missing, as it is apparently the case on our
jenkins nodes. This makes the tests run a bit slower, but not much. Add
kernel_test_wait_for_vm to sleep until the VM is booted up (~5s instead
of ~1s without KVM), so the first test in the testsuite does not fail
when using KVM.
The variable is useful to test the non-KVM code path even if KVM is
available on the host.
Related: OS#3208
Change-Id: I1f337af1e2de6db05b22636bc31a535404235559
Make it possible to run ttcn3-ggsn-test against osmo-ggsn with the GTP
kernel module. This feature is enabled by environment variables
explained in ttcn3-ggsn-test/jenkins.sh, and can either use the
pre-built kernel from Debian, or build a kernel from source.
The kernel runs inside QEMU inside the usual osmo-ggsn-master docker
image.
Thanks to Pau for providing the kernel config and kernel-test scripts
that this patch is based on! I ended up modifying them quite a bit to
fit the docker use case, and to build the initrd (so we can use the
stock debian kernel).
Related: OS#3208
Change-Id: I54e752048eed2a170a1cf4e0fd410de255b9de5d
Don't call fix_perms from collect_logs anymore, it looks like it was
only added there to avoid adding it at the end of each jenkins.sh. Now
that we have clean_up_common, put it there.
Related: OS#3208
Change-Id: I6e7e5b35198f420e130b1282eb948b96ccd6c8e0
Add set_clean_up_trap() in jenkins-common.sh and run it at the beginning
of the jenkins.sh files. Move the common clean up code from the end of
every jenkins.sh file into clean_up_common(), which gets called by the
trap. Add a custom clean_up() function to those jenkins.sh files that
need additional clean up.
Replace explicit container stop commands (for containers attached to the
docker network) with one call to network_clean() in clean_up_common(). It
kills all containers attached to the docker network.
The motivation for this change is the upcoming optional build of initrd
and kernel during ttcn3-ggsn-test/jenkins.sh. After building these, a
short smoke test will be performed to make sure we can boot the kernel
and initrd, before continuing to run the entire testsuite against it. If
building or the smoke test fails, we must do a proper clean up of the
network and fix permissions.
Related: OS#3208
Change-Id: I807c6221d22fec31db23e2516e477a3eb43347fb
Similar to what is already done with DISTRO, which points to given image
of ours based on name. This time we do the same with upstream images,
such as debian:stretch or centos:centos8.
This way, for instance calling docker_images_require
"osmo-bsc-latest-centos8" would try to build the
osmo-bsc-latest/Dockerfile file starting from a centos8 image.
Change-Id: I33cb21aa024396974559fd98f9f3c64e2c351eda
We previously only allocated IPv4 sub-nets to the network segments
created with 'docker network'. Let's fix that by assigning both
IPv4 and IPv6 address ranges to each docker network.
Related: OS#4700
Change-Id: I8802208fddcce1ffa57e5626575d23d02b320d99
When my disk runs full, jenkins-common.sh fails to create a VOL_BASEDIR right
at the start, and hence no testsuite gets its external volumes mounted
properly. However, the test suite still starts up with missing cfg files etc,
and fails in non-obvious ways, until I find out the disk is full some minutes
later. Instead, verify that the base dir really exists or bail out right there.
Change-Id: I6540797c8e0ee3e7b09d4a80592d5e270e7d9adc
Make it possible to call the ttcn3-*/jenkins.sh scripts with:
IMAGE_SUFFIX="master-centos8"
The existing docker_images_require lines of these jenkins.sh scripts
will then call docker_images_require with arguments like
"osmo-mgw-master-centos8". For example, from ttcn3-mgw-test/jenkins.sh:
docker_images_require \
"osmo-mgw-$IMAGE_SUFFIX" \
"ttcn3-mgw-test"
Let docker_images_require build the image from
osmo-mgw-master/Dockerfile (-centos8 is cut off from the dirname) and
with DISTRO=centos8 as argument.
Collisions with the debian-stretch images are avoided by setting IMAGE
to the full image name (e.g. osmo-mgw-master-centos8).
Related: OS#4564
Change-Id: I598a262fe1a7ed4dd89e13c53e4ded103c6e3b91
Add a function to automatically resolve the most common dependencies,
osmo-* need debian-stretch-build and ttcn3-* need debian-stretch-titan.
Do this in a shell function in jenkins-common.sh, so we can extend it in
the next patch with distribution specific dependencies (osmo-*-centos8
needs centos8-build).
Related: OS#4564
Change-Id: I98760c6dc3206529c0e267338c9f6f8e1d5be33a
We can't create the net if it exists, and it can't be removed until all
attached containers are dead, so ensure this is the case upon net
creation. This fixes test failures due to stale nets and half-killed
test runs.
Change-Id: Id6d13b233ebfd808d8dfe83b6d1d1ba20c3392c8
When fixing the permissions at the end of a test run, it really doesn't
matter if we're running a current debian-stretch-build image, or if we
are using an ancient one. All we need is a 'chmod' binary.
Change-Id: Ib008ee72f40aeefb307c0ba74a57b70daae9d07b
Related: OS#4060
Prevent building docker images on top of outdated Debian images, where
the package download feeds have been disabled. Use 'docker build --pull'
instead of 'docker build', whenever the "FROM" line in the Dockerfile
does not contain '$USER' (meaning this is an upstream image).
Checking the FROM line is necessary, as downloading '$USER' images will
fail (we never upload those).
Related: OS#3869
Change-Id: I1076bbb7d77bdc99f5d60d641c09ce965fb9dfc6
Use the debian-jessie-build container to ensure all files under
${VOL_BASE_DIR} are readable.
Call this directly from collect_logs so we handle all ttcn3 jobs.
Fixes: OS#3800
Change-Id: I124c554eee69ec5586547d3c96b8ec571e0653a7
docker_images_require() must fail when a new image can not be built,
even if an old image already exists.
We have this situation in a lot of the -latest tests right now, which
makes it harder to find the actual error.
Note that we can't simply use #!/bin/sh -e in all the jenkins.sh
scripts that source jenkins_common.sh, because they must be able to
clean up the docker containers on failure. So they can't abort
somewhere in the middle of the scripts. We could rewrite them to do the
clean up in a trap, but this should be done in a separate commit.
Related: OS#3767
Change-Id: I7039089457b62b8798a79c5efd62bd91622986d3
This new function can be used on top of each *-test/jenkins.sh to
automatically build the required Docker images before starting a
testsuite.
The top-level Makefile should also be able to build all Docker images
on which a job depends. But in that Makefile the list of dependencies
can not be specified dynamically, as it will be necessary for OS#3268:
ttcn3-mgw-test may depend on osmo-mgw-latest or on osmo-mgw-master.
This will be used by Idbb708ab16cb71bab5069127945b63388222369e.
Related: OS#3268
Change-Id: I0afdf3400282634fca9b31d57eaeedd6b5c28aa1
On jenkins, place all logs and manage docker volumes in the workspace instead
of a /tmp/* dir. Use $WORKSPACE/logs as docker volume base to begin with, thus
there needs to be no copy from /tmp to $WORKSPACE/logs.
On non-jenkins runs, place all in a /tmp/* dir still, but also skip copying of
the logs: just have a /tmp/logs symlink to the last tmpdir.
Change-Id: I8cf6014725ae8ba602be5f3ec31dfb8e49ff993e
To remove code dup and prepare a change to where logs are written, add
collect_logs() to jenkins_common.sh and call that from each jenkins.sh after
the tests are done.
The 'rm -rf' is already done before a test starts. No need to do that again
after each test.
Change-Id: I5d8472ec36b07c828685b1bd7718e31392d168a3