We started using git submodules with osmo-trx. Adjust the obs scripts
to actually make the git submodules part of the source packages.
Note that this didn't fail in jenkins before with the rpm build
verification, as jenkins updates git submodules on its own.
Fix for:
[ 165s] Makefile.am:32: error: required directory ./osmocom-bb/src/host/trxcon does not exist
Change-Id: I51b423f3885d6ead5c21a83bdf8ef6051dc34fe3
After building packages with the OBS scripts, show their contents. This
allows easy checking if config files were correctly packaged etc. By
adding it here, it will also show up at the end of the related jenkins
jobs for gerrit verifications.
Related: OS#5817
Change-Id: Ie30b07f35f7e41990fa352523427d86458291d4d
With recent changes in open5gs, building the source packages we
generated fails:
[ 45s] ../lib/metrics/meson.build:52:4: ERROR: Automatic wrap-based subproject downloading is disabled
[ 45s] dh_auto_configure: error: cd obj-x86_64-linux-gnu && LC_ALL=C.UTF-8 meson .. --wrap-mode=nodownload --buildtype=plain --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=lib/x86_64-linux-gnu --libexecdir=lib/x86_64-linux-gnu returned exit code 1
Fix this by downloading all subprojects. This is now possible as I added
a meson.build file to the prometheus-client-c repository that open5gs
uses: https://github.com/open5gs/prometheus-client-c/pull/2
Change-Id: If3910b520382b177a77b216c93771ea88414723d
The number of builders connected to the OBS server seems to fluctuate a
bit (used to be 14, then 17 now 16). Make sure that we have at least 10
builders connected. Fix that the test didn't exit with 1 on error.
Related: https://obs.osmocom.org/monitor
Change-Id: Iedd506601a5450550e21bf701309b4ea79a3d897
Add the "%ext_man .gz" macro from OBS, so building rpms with man pages
does not fail with errors like:
File not found: /home/user/rpmbuild/BUILDROOT/osmo-ggsn-1.9.0.3.0d3b-0.x86_64/usr/share/man/man8/osmo-ggsn.8
Fixes: OS#5737
Change-Id: Ib6950fb46e9f94aabae98021b215f69838557045
Add a job that checks if the amount of connected builders to the OBS
server matches what we expect. This should prevent what we had today,
that it only was noticed by chance several days after all builders
failed to connect and the packages were outdated since then.
I'm not sure if there is a proper api to do this check, but I don't
expect the string we check for to change often so this should work fine.
Change-Id: I6e7c1f206551722d6bfe1631b9c1da8d34d85ba8
Instead of having the description of the nightly repository (copy paste
error), put in "osmocom:master". This gets printed a few times in the
log of the gerrit-binpkgs-rpm jenkins job and may lead to confusion -
the nightly repository is not used, it's a separate master repository.
Related: OS#2385
Change-Id: Ia5a88b064da66d7bf5cebe910961f752262b1bac
Allow running build_srcpkg.py with "osmo_dia2gsup" instead of
"erlang/osmo_dia2gsup", because that's how it gets passed along from
jobs/gerrit-verifications.yml for the binary package build
verifications.
Related: OS#2385
Change-Id: I09304b219e7002495bd3fcce61bc68c34d5ffcd8
Instead of using "--docker centos:8" as argument and translating it
later on in the code to use "almalinux:8" as base distribution, use
"--docker almalinux:8" as argument. It was brought up in code review
that this makes it less confusing.
Related: OS#2385
Change-Id: Id8298e8bafe065010f7bc00f1ff261aa6431ed4c
Installing osmo-gsm-manuals-dev plus depends takes a long time. Don't do
this for every build, instead do it once when building a second docker
container and then use that.
Related: OS#2385
Change-Id: I8475bd954352b572197795ad4cd9461e39896d48
Configure dpkg to not extract man pages. Otherwise it will spend some
time regenerating the man page index whenever installing build
dependencies before starting to build a package.
Related: OS#2385
Change-Id: I1c9e3883b976e023c96dfd59eb147770f7ad99a7
Add a script to build deb/rpm packages, as it would be done on
obs.osmocom.org. This will be used by jenkins to verify deb and rpm
builds for each submitted gerrit patch.
I have attempted to use 'osc build' instead of directly calling
apt-get build-dep and dpkg-buildpackage (and rpm equivalents). Using
'osc build' would have the advantage that the build works as close to
the OBS build as possible. However it would try to install dependencies
with sudo, so we would need to have sudo available in the docker
container that builds the untrusted code from gerrit. Let's not do that.
Related: OS#2385
Change-Id: I4c6b5d61af35df98cbc70d9ddc8ad36d38a9ce18
Don't use os.path.basename on the script_path parameter passed to the
function. A future patch will pass a script that is inside the data
subdirectory to this function, therefore this is needed.
Related: OS#2385
Change-Id: Ide78d976f9af445c4c8d8748bc274d7289064769
Rename Dockerfile to build_srcpkg.Dockerfile and adjust related code to
allow using a different Dockerfile for building the binary packages in a
future patch.
Related: OS#2385
Change-Id: I8ef7944a4a81acd6c915998f37139eebad2b2d3e
No need to pass args here. It was only used for args.verbose, and that
isn't necessary since we don't need to restore the verbose mode.
Related: OS#2385
Change-Id: I620cdef46e18f4c66644f14003caf2183c89686f
Put it on an extra line like the other arguments to prevent it from
getting overlooked.
Related: OS#2385
Change-Id: Ie7dea2734e7c47766dfe64b7091bc4f5f72d53a5
Add one new job for building source packages and sending them to
obs.osmocom.org. Trigger it from all master-* jobs.
I've also considered adding one job per existing master job that would
only update one package at a time (master-libosmocore-obs,
master-osmo-bsc-obs, ...). With some additional development effort it
should be possible, and it would make each individual master OBS job
faster. But given that with the current implementation it only takes
20s to 30s for *all* packages if there are no changes, as it compares
git remote HEAD with the version currently on OBS before starting to
clone repositories and building the source packages (similar to
Osmocom_OBS_latest_obs.osmocom.org), it didn't seem worth optimizing.
Set concurrent to false as the triggers from master-builds will likely
cause it to run multiple times in parallel otherwise.
Related: https://jenkins.osmocom.org/jenkins/view/OBS/job/Osmocom_OBS_master_obs.osmocom.org/
Related: https://obs.osmocom.org/project/show/osmocom:master
Related: OS#2385
Change-Id: I53a494f13f81ae837f2d362c54e1bdf13f121db3
Add a new master feed, where packages are updated as soon as patches get
merged to master. Upload a commit_$COMMIT.txt file for each package in
this feed and use it to determine if the package needs to be updated or
not.
In most packages the commit is already part of the version in the dsc
file, e.g. "libosmocore_1.7.0.38.c3b90.dsc", but that's harder to parse
and is more likely to have a hash collision (just 5 characters).
Related: OS#2385
Change-Id: I3b0b4f4876b8c1eeb61f20d903a6f2cac6e99638
Make it slightly easier to read, before extending the latest branch with
master too in a follow-up patch.
Related: OS#2385
Change-Id: I4f1d4e250eb4d8163f1ded29106a44deb0d0feff
Refactor checkout_default_branch so the get_default_branch logic can be
used from another function in a future patch.
Related: OS#2385
Change-Id: If3f0ab500ff165d81f61b19ad41af90bd39dce8f
While generating srcpkg we're not really interested in entire
project history (which can be quite heavy) - let's only get
the bare minimum from git.
Change-Id: I71629c244d2c29728a88871e83b20493341321c8
Having an option to explicitly checkout sources for package from gerrit
using numeric ID comes in handy for testing changes.
Related: OS#4107
Change-Id: I403c2083ef8c3f6f966c5b376fb13ea46d8a7a2b
Not all the required programs are actually required to build subset of packages.
Add option to ignore the default check.
Change-Id: I8b5eaedab87b586fb1ca4c22746be4dfbb3c2176
Don't pass stdin to the programs, as we expect them to run
non-interactively and also don't show the program's output unless -v is
used or the exit code is not 0.
Change-Id: I7e893101c2a3e7b005659ec72aa44fa932b7ccd9
Check if the oscrc exists, as otherwise the osc command will
interactively prompt for the credentials. If running without verbose
mode (the default), the command output is hidden and so it would just
appear to be hanging without this patch if the config doesn't exist.
Change-Id: I6f140b2c1d171d3019fd259399e6a46da9c6451f
When running in verbose mode, it's enough that the output is displayed
while the command is running. Don't print it again on error.
Change-Id: I4fcf2421a6f82bd94ba098b39f58e5d3f18aa8fd
Deb and rpm packages don't build for osmo-upf so disable it again until
it is fixed.
Related: OS#5655, OS#5654
Change-Id: Idf18159b786c51154fda9b40758b08a8656fc215
If there's an empty debian/changelog, packaging needs to be fixed in the
git repository of the project. Print a meaningful error instead of:
File "/obs/lib/srcpkg.py", line 68, in get_version_for_feed
if ":" in ret:
TypeError: argument of type 'NoneType' is not iterable
Change-Id: I294d844ccccfa12599b6ba1def087e4a32d289e7
As Neels pointed out, with the current code the 2022q1 and similar feeds
do not get checked out properly if:
* the repository was not freshly cloned (like jenkins would do)
* the repository was fetched (--git-fetch)
Instead of the last revision of that branch from origin, we get the
revision that was previously checked out on the branch. Fix this by
prepending origin/.
Change-Id: I2e265fc84abd4596c386bb56cbabb2af121dc7ca
Looks like I copied the list of packages from the old latest script, so
these two that were only in nightly are missing. Once added, they will
be built for nightly again and not for latest as there's no tag matching
the patterin in the git repositories.
Change-Id: I4b96f15e95c70f1dca12a292964e6f12b04e4a04
Apparently a \n at the end of debian/source/format is not required, so
don't fix it if it is missing. This avoids printing this confusing line:
osmo-trx: fixing debian/source/format (3.0 (native) => 3.0 (native))
Change-Id: I7f9bb22a389a2109109f7fecd3b7ae0413fe6f5b
Harald requested that the OBS scripts should not stop if building one
specific source package fails, instead it should keep going and report
at the end a non-success exit code.
Given that the shell script code has historically grown and became hard
to maintain, I decided to rewrite the scripts for implementing this
feature. This rewrite solves additional problems:
* No full checkout of an OBS project like network:osmocom:latest
anymore, with lots of packages that won't get updated (e.g. the uhd
package has a uhd-images_3.14.1.1.tar.xz file that is 108 MB). With
the old code, developers had to wait minutes during the checkout
before the script reaches code that is currently being developed. Now
only single packages get checked out right before they get updated.
* No need to clone git repositories over and over. With the new code,
git repos only get cloned if needed (for latest it is not needed if
the remote git tag is the same as the version in OBS). During
development, the cloned git repositories are cached.
* Output from commands like "git tag -l" is not written to the log
unless they failed. This makes the log more readable, which is
especially important when a package fails to build, we keep going and
need to spot the build error in the middle of the log later on.
* No more duplicated code for nightly and latest scripts that worked
similar but had slight differences. Also the list of packages is not
duplicated for nightly and latest anymore; nightly uses all packages
and latest uses packages that have at least one git tag.
* Building source packages is decoupled from uploading them. A separate
script build_srcpkg.py can be used to just build the deb + rpm spec
source packages, without interacting with the OBS server.
* The scripts can optionally run in docker with a command-line switch,
and this is used by jenkins. This way we don't need to install
more dependencies on the host such as rebar3 which is now needed for
erlang/osmo_dia2gsup.
* Add erlang/osmo_dia2gsup and run its generate_build_dep.sh (SYS#6006)
I have done the new implementation in python to make use of argparse
and to be able to use try/except and print a trace when building one
package fails.
Example output:
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly_obs.osmocom.org/48/console
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_latest_obs.osmocom.org/46/console
Change-Id: I45a555d05a9da808c0fe0145aae665f583cb80d9
Oliver Smith
Max
Neels Hofmeyr
Vadim Yanitskiy