Rewrite the osmocom-release-manuals script (previous version is in
* Instead of using a separate docker container for this, that also lists
all dependencies for all packages (as needed to pass ./configure), use
debian-bookworm-build and install missing packages at time of
generating the tarballs with "apt-get build-dep". Missing dependencies
are typically other Osmocom libraries.
* This allows removing the debian 11 based release-tarball-build-dist
container. As the script doesn't depend on a separate docker container
anymore, move it to osmo-ci.git.
* Make it similar to scripts/manuals/publish-manuals-for-tags.sh, so it
is easier to maintain both.
Add a jenkins job that automatically builds manuals for release tags, if
they don't exist on the server already.
Change repo-install-test to run inside of qemu instead of docker. This
job needs to run systemd to verify that the systemd services start up
properly. Running systemd inside docker was never officially supported,
it worked with cgroups1 but does not work anymore with cgroups2.
An alternative approach was running inside podman instead of docker
(running systemd inside of podman is officially supported). However we
would have needed various workarounds with podman and wouldn't be able
to test all Osmocom systemd services in the end, due to lack of
permissions (see review of I394918fc61de36acce65ffb33defcb8fc21801c4).
By running with a separate kernel inside qemu we can run all Osmocom
Harald requested that the OBS scripts should not stop if building one
specific source package fails, instead it should keep going and report
at the end a non-success exit code.
Given that the shell script code has historically grown and became hard
to maintain, I decided to rewrite the scripts for implementing this
feature. This rewrite solves additional problems:
* No full checkout of an OBS project like network:osmocom:latest
anymore, with lots of packages that won't get updated (e.g. the uhd
package has a uhd-images_220.127.116.11.tar.xz file that is 108 MB). With
the old code, developers had to wait minutes during the checkout
before the script reaches code that is currently being developed. Now
only single packages get checked out right before they get updated.
* No need to clone git repositories over and over. With the new code,
git repos only get cloned if needed (for latest it is not needed if
the remote git tag is the same as the version in OBS). During
development, the cloned git repositories are cached.
* Output from commands like "git tag -l" is not written to the log
unless they failed. This makes the log more readable, which is
especially important when a package fails to build, we keep going and
need to spot the build error in the middle of the log later on.
* No more duplicated code for nightly and latest scripts that worked
similar but had slight differences. Also the list of packages is not
duplicated for nightly and latest anymore; nightly uses all packages
and latest uses packages that have at least one git tag.
* Building source packages is decoupled from uploading them. A separate
script build_srcpkg.py can be used to just build the deb + rpm spec
source packages, without interacting with the OBS server.
* The scripts can optionally run in docker with a command-line switch,
and this is used by jenkins. This way we don't need to install
more dependencies on the host such as rebar3 which is now needed for
* Add erlang/osmo_dia2gsup and run its generate_build_dep.sh (SYS#6006)
I have done the new implementation in python to make use of argparse
and to be able to use try/except and print a trace when building one
Clone docker-playground.git, source its jenkins-common.sh and run
docker_images_require from there. This will make it possible to run
osmocom-release-tarballs.sh in a docker container, for which the
Dockerfile is stored in docker-playground.git.
Build old releases of Osmocom programs and libraries against
"master of the day" to detect breakage.
Redirect each build's output to its own log file, so it is easy to see
what is currently getting build, and what failed. On error, print the
end of the new failing build logs, along with a note to find the full
logs in the jenkins artifacts.
This initial configuration builds the last three release tags of the
Osmocom repositories against master. The configuration can be changed
Indicate known failures with "err" instead of "ERR" in the output, do
not cause them to fail the build and do not print the beginning of the
error log (it is still in the artifacts). This way, new errors stand out
and don't get overlooked among the known errors.
The osmo-depcheck script consists of multiple Python files. Python
creates a __pycache__ folder, so let's make sure that does not get
added by accident into the git repository.
jenkins job builder is a python library to write jenkins jobs in .yml or
.json including templating and basic substition operation.
To update the job call:
Create your own jenkins_jobs.ini based on the exmaple and call
jenkins-jobs --conf ./jenkins_jobs.ini update osmocom-nightly-nitb-split.yml