Add the following projects to our coverity test job:
osmo-pcap
osmo-e1d
osmo-e1-recorder
osmo-upf
osmo-isdntap
osmo-uecups
osmo-remsim
Change-Id: Idee49aa4c15fcfb1a469db9e4978523af5608d70
Building debian-buster-jenkins on the raspberry pis may take > 1h,
depending on what changed. It has a lot of things we don't use, so build
and use a minimal image with the rpis instead.
Closes: OS#5863
Depends: docker-playground I4fb4b48b233acaef815c4c27ec6f17f12cfe836b
Change-Id: I73e62fbbf93824c0d37963039de4e00f26a43cbe
This script was supposed to cache already built Osmocom libraries. But
it was only used with openbsc, didn't work as expected anymore (looking
at openbsc-master, it just builds all deps from source without caching)
and will be replaced with logic that allows using ccache in a future
patch.
Using ccache has the big advantage that we can cache all build objects
where the source files did not change, and not just do it on a
dependency level. Oftentimes only few source files change in our
patches, so we can really cache everything else that is not affected
within any git repository and not just a dependency that didn't change.
ccache also does automatic cache size management to ensure a cache stays
within a given size limit.
Related: OS#5848
Depends: openbsc Ib3272feec76b30412ca60dec204255b64e33831b
Change-Id: Id94d6126b476077d57839e4a884621b8c034f0c6
Post a link to gerrit when starting the build that contains the link to
the pipeline, without sending mail notifications.
This is useful when a gerrit verification build takes rather long, and
it's not clear if a build for gerrit verification was actually started
or not. Also I find this useful when debugging the CI scripts.
Change-Id: I75c5b8874f606739ff557ff0711bb9449a2b4259
Rename the jenkins job from gerrit-pipeline-result to
gerrit-verifications-comment, as the next patch will not only use this
job to post a comment at the end of the gerrit verification, but also at
the beginning when the pipeline starts.
Give the pipeline_summary* scripts more generic names as well.
Change-Id: I1b947522aa5f2bb21f5e438db9df3420c998f1bc
We started using git submodules with osmo-trx. Adjust the obs scripts
to actually make the git submodules part of the source packages.
Note that this didn't fail in jenkins before with the rpm build
verification, as jenkins updates git submodules on its own.
Fix for:
[ 165s] Makefile.am:32: error: required directory ./osmocom-bb/src/host/trxcon does not exist
Change-Id: I51b423f3885d6ead5c21a83bdf8ef6051dc34fe3
Apparently the output of repoquery isn't always sorted, it started
failing on jenkins after moving the test inside qemu. Add an explicit
sort.
Fix for:
+ comm -23 osmocom_packages_all.txt blacklist.txt
comm: file 1 is not in sorted order
Related: OS#5365
Change-Id: Icb00df102555e06b66b1c2597488b625e3c77f1c
While I'm developing this, Jenkins is currently failing here. Make it
easier to debug this by printing the file contents.
Related: OS#5365
Change-Id: Ifbf4ca7f49c1f4441f84695aea0936515e01ffd4
The DISTRO variable is either debian10 or debian11, fix the broken
check. This condition is there in the first place, because we don't
build the usrp1 backend for centos8.
Change-Id: I987f27db257961faf06824df2dcc8f9db1fedccf
Related: OS#5365
Add services from new projects and enable previously disabled services,
now that this test runs in qemu and services have more permissions like
setting realtime priorities.
Related: OS#5365
Change-Id: Iec7db433cac4c77226e0f1ae2ba502de0d1a8a2b
Change repo-install-test to run inside of qemu instead of docker. This
job needs to run systemd to verify that the systemd services start up
properly. Running systemd inside docker was never officially supported,
it worked with cgroups1 but does not work anymore with cgroups2.
An alternative approach was running inside podman instead of docker
(running systemd inside of podman is officially supported). However we
would have needed various workarounds with podman and wouldn't be able
to test all Osmocom systemd services in the end, due to lack of
permissions (see review of I394918fc61de36acce65ffb33defcb8fc21801c4).
By running with a separate kernel inside qemu we can run all Osmocom
services.
Related: OS#5365
Change-Id: Ie7f1bccb05779cb3614016c0b965b810bbb1471b
Remove the fallback clean up code, as it also may lead to images getting
removed right before we need to use them. Besides that, it should be
dead code by now since docuum should be running on all our jenkins nodes
to clean up old images based on last use date.
Change-Id: I9ca0c2ba245bdd75d9fb8eaf341055e8c2ab1b55
Don't delete images while they are being used, to fix these errors we
see from time to time in the middle of "docker build" on jenkins:
unknown parent image ID sha256:1b072e35048cd8b680eddabdc641ac678edb1184d222d5e7b3fbe0b3c333129a
This happens because "docker build" creates so-called dangling images
for each step processed of a Dockerfile. The "docker system prune" call
deletes these dangling images (among other things).
Remove the "docker system prune" call. We already have the docuum daemon
to deal with unused images (dangling and not dangling), it removes them
based on last use date so that the used space is always below a
configured limit. As it deletes images that haven't been used the
longest when it reaches the limit, it will not result in the problem
explained above.
Besides images, "docker system prune" also removes unused containers
(instances of images created with 'docker run' without --rm) and
networks. Add "docker container prune" and "docker network prune"
commands to remove them from now on.
Also remove the redundant container removal logic (previous it was
redundant with "docker system prune", now redundant with "docker
container prune").
Related: https://docs.docker.com/config/pruning/
Change-Id: Ia1b466eea43dd135373949e8e3e6b005c169ea0c
After building packages with the OBS scripts, show their contents. This
allows easy checking if config files were correctly packaged etc. By
adding it here, it will also show up at the end of the related jenkins
jobs for gerrit verifications.
Related: OS#5817
Change-Id: Ie30b07f35f7e41990fa352523427d86458291d4d
Build the docker image right before using it, instead of building it in
another jenkins job update-osmo-ci-on-slaves via
osmo-ci-docker-rebuild.sh.
The logic in osmo-ci-docker-rebuild.sh was broken. I didn't realize at
the time that this image is only used for the virtual osmo-gsm-tester,
not the physical ones. But only the machines running the physical
osmo-gsm-tester have the /var/tmp/osmo-gsm-tester/state path. The
virtual osmo-gsm-tester isn't running on these machines but on generic
jenkins nodes.
Building the image right before using it makes sense for this job, as it
is the only user of the image. If it was already built from the same
Dockerfile, a cached version is used.
Fix for:
Unable to find image 'osmocom-build/osmo-gsm-tester:latest' locally
Fixes: 9139e76b ("osmo-ci-docker-rebuild: don't always build osmo-gsm-tester")
Change-Id: Icad9459de1d3a3a4e65ecacf7f903433bb504cc9
Don't attempt to build openbsc 1.4.0 and 1.4.1 in the
osmocom-build-old-tags-against-master job, as it fails to build with a
modern gcc with a potential null pointer dereference warning that gets
treated as error. This was fixed in 1.4.2 with openbsc patch
I93d816a20ba208e9fd32a1fc172a78ccd326e1ba.
A more modern gcc gets used, as this job gets changed to not only run on
jenkins nodes with debian 9 installed, but also more modern versions.
This job does not run in docker, and since it still passes it doesn't
seem worth changing that now.
Related: OS#5793
Change-Id: Ibf4b66195fbd70a68d63929b156a9aac54a699bc
With recent changes in open5gs, building the source packages we
generated fails:
[ 45s] ../lib/metrics/meson.build:52:4: ERROR: Automatic wrap-based subproject downloading is disabled
[ 45s] dh_auto_configure: error: cd obj-x86_64-linux-gnu && LC_ALL=C.UTF-8 meson .. --wrap-mode=nodownload --buildtype=plain --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=lib/x86_64-linux-gnu --libexecdir=lib/x86_64-linux-gnu returned exit code 1
Fix this by downloading all subprojects. This is now possible as I added
a meson.build file to the prometheus-client-c repository that open5gs
uses: https://github.com/open5gs/prometheus-client-c/pull/2
Change-Id: If3910b520382b177a77b216c93771ea88414723d
Fix an error when the same Change-Id is used on multiple branches or in
theory projects. This is actually allowed by gerrit, and we use this
e.g. when backporting patches from master.
Use the project, change number (e.g. 30147) and patchset number (e.g. 2)
instead of the Change-Id.
Fix for:
+ ssh -p 29418 -l jenkins gerrit.osmocom.org gerrit review 4835a62cd88f0d69db76fb3bfd2df02176a91a6d --json
fatal: "4835a62cd88f0d69db76fb3bfd2df02176a91a6d" matches multiple patch sets
Related: https://gerrit-review.googlesource.com/Documentation/cmd-review.html
Change-Id: I2d627f8f3b400fa57a50a228d47df2194f60fd08
The number of builders connected to the OBS server seems to fluctuate a
bit (used to be 14, then 17 now 16). Make sure that we have at least 10
builders connected. Fix that the test didn't exit with 1 on error.
Related: https://obs.osmocom.org/monitor
Change-Id: Iedd506601a5450550e21bf701309b4ea79a3d897
Add the "%ext_man .gz" macro from OBS, so building rpms with man pages
does not fail with errors like:
File not found: /home/user/rpmbuild/BUILDROOT/osmo-ggsn-1.9.0.3.0d3b-0.x86_64/usr/share/man/man8/osmo-ggsn.8
Fixes: OS#5737
Change-Id: Ib6950fb46e9f94aabae98021b215f69838557045
Add a list of users that do not only get a notification mail when the
gerrit verification failed, but also on success.
Change-Id: I603b8a911c8f17aa726d9e3d5d644ad3262b42dd
Add a job that checks if the amount of connected builders to the OBS
server matches what we expect. This should prevent what we had today,
that it only was noticed by chance several days after all builders
failed to connect and the packages were outdated since then.
I'm not sure if there is a proper api to do this check, but I don't
expect the string we check for to change often so this should work fine.
Change-Id: I6e7c1f206551722d6bfe1631b9c1da8d34d85ba8
One might be wondering what happened if the build job failed, but
building the binary packages succeeded. Since we run 'make check' in
all cases. Add this short explanation:
The build job(s) failed, but deb/rpm jobs passed.
We don't enable external/vty tests when building
packages, so maybe those failed. Check the logs.
Related: OS#2385
Change-Id: I70027ec970a220c3ddfa766302faec7bd8752118
Instead of having the description of the nightly repository (copy paste
error), put in "osmocom:master". This gets printed a few times in the
log of the gerrit-binpkgs-rpm jenkins job and may lead to confusion -
the nightly repository is not used, it's a separate master repository.
Related: OS#2385
Change-Id: Ia5a88b064da66d7bf5cebe910961f752262b1bac
It's useful to be able to retrigger the pipeline, for example if the
build failed because a depending patch was not merged yet and later it
is merged. Add the link where the "Retrigger" button is to make this
convenient.
Related: OS#2385
Change-Id: I647efac9e79d755282b78f14bd27244c99ef7f11
Instead of notifying everybody listed as reviewers and in CC:
* notify the owner if the build failed
* notify nobody if the build was successful
Related: OS#2385
Change-Id: I7c6c183b98624eb75d5dccd9766ee3ff5568b06e
Let it match "Starting building: gerrit-osmo-ttcn3-hacks-build #5". It
failed on the 3 in ttcn3 before.
Related: OS#2385
Change-Id: I247af55e2c0e3a2dd088ab1c951d8535cfc93229
Allow running build_srcpkg.py with "osmo_dia2gsup" instead of
"erlang/osmo_dia2gsup", because that's how it gets passed along from
jobs/gerrit-verifications.yml for the binary package build
verifications.
Related: OS#2385
Change-Id: I09304b219e7002495bd3fcce61bc68c34d5ffcd8
Instead of using "--docker centos:8" as argument and translating it
later on in the code to use "almalinux:8" as base distribution, use
"--docker almalinux:8" as argument. It was brought up in code review
that this makes it less confusing.
Related: OS#2385
Change-Id: Id8298e8bafe065010f7bc00f1ff261aa6431ed4c
Installing osmo-gsm-manuals-dev plus depends takes a long time. Don't do
this for every build, instead do it once when building a second docker
container and then use that.
Related: OS#2385
Change-Id: I8475bd954352b572197795ad4cd9461e39896d48
Configure dpkg to not extract man pages. Otherwise it will spend some
time regenerating the man page index whenever installing build
dependencies before starting to build a package.
Related: OS#2385
Change-Id: I1c9e3883b976e023c96dfd59eb147770f7ad99a7
Add a script to build deb/rpm packages, as it would be done on
obs.osmocom.org. This will be used by jenkins to verify deb and rpm
builds for each submitted gerrit patch.
I have attempted to use 'osc build' instead of directly calling
apt-get build-dep and dpkg-buildpackage (and rpm equivalents). Using
'osc build' would have the advantage that the build works as close to
the OBS build as possible. However it would try to install dependencies
with sudo, so we would need to have sudo available in the docker
container that builds the untrusted code from gerrit. Let's not do that.
Related: OS#2385
Change-Id: I4c6b5d61af35df98cbc70d9ddc8ad36d38a9ce18
Don't use os.path.basename on the script_path parameter passed to the
function. A future patch will pass a script that is inside the data
subdirectory to this function, therefore this is needed.
Related: OS#2385
Change-Id: Ide78d976f9af445c4c8d8748bc274d7289064769
Rename Dockerfile to build_srcpkg.Dockerfile and adjust related code to
allow using a different Dockerfile for building the binary packages in a
future patch.
Related: OS#2385
Change-Id: I8ef7944a4a81acd6c915998f37139eebad2b2d3e
No need to pass args here. It was only used for args.verbose, and that
isn't necessary since we don't need to restore the verbose mode.
Related: OS#2385
Change-Id: I620cdef46e18f4c66644f14003caf2183c89686f
Put it on an extra line like the other arguments to prevent it from
getting overlooked.
Related: OS#2385
Change-Id: Ie7dea2734e7c47766dfe64b7091bc4f5f72d53a5
Add one new job for building source packages and sending them to
obs.osmocom.org. Trigger it from all master-* jobs.
I've also considered adding one job per existing master job that would
only update one package at a time (master-libosmocore-obs,
master-osmo-bsc-obs, ...). With some additional development effort it
should be possible, and it would make each individual master OBS job
faster. But given that with the current implementation it only takes
20s to 30s for *all* packages if there are no changes, as it compares
git remote HEAD with the version currently on OBS before starting to
clone repositories and building the source packages (similar to
Osmocom_OBS_latest_obs.osmocom.org), it didn't seem worth optimizing.
Set concurrent to false as the triggers from master-builds will likely
cause it to run multiple times in parallel otherwise.
Related: https://jenkins.osmocom.org/jenkins/view/OBS/job/Osmocom_OBS_master_obs.osmocom.org/
Related: https://obs.osmocom.org/project/show/osmocom:master
Related: OS#2385
Change-Id: I53a494f13f81ae837f2d362c54e1bdf13f121db3
Add a new master feed, where packages are updated as soon as patches get
merged to master. Upload a commit_$COMMIT.txt file for each package in
this feed and use it to determine if the package needs to be updated or
not.
In most packages the commit is already part of the version in the dsc
file, e.g. "libosmocore_1.7.0.38.c3b90.dsc", but that's harder to parse
and is more likely to have a hash collision (just 5 characters).
Related: OS#2385
Change-Id: I3b0b4f4876b8c1eeb61f20d903a6f2cac6e99638
Harald Welte
Oliver Smith