scripts/obs: rewrite pushing source pkgs to OBS

Harald requested that the OBS scripts should not stop if building one
specific source package fails, instead it should keep going and report
at the end a non-success exit code.

Given that the shell script code has historically grown and became hard
to maintain, I decided to rewrite the scripts for implementing this
feature. This rewrite solves additional problems:

* No full checkout of an OBS project like network:osmocom:latest
  anymore, with lots of packages that won't get updated (e.g. the uhd
  package has a uhd-images_3.14.1.1.tar.xz file that is 108 MB). With
  the old code, developers had to wait minutes during the checkout
  before the script reaches code that is currently being developed. Now
  only single packages get checked out right before they get updated.

* No need to clone git repositories over and over. With the new code,
  git repos only get cloned if needed (for latest it is not needed if
  the remote git tag is the same as the version in OBS). During
  development, the cloned git repositories are cached.

* Output from commands like "git tag -l" is not written to the log
  unless they failed. This makes the log more readable, which is
  especially important when a package fails to build, we keep going and
  need to spot the build error in the middle of the log later on.

* No more duplicated code for nightly and latest scripts that worked
  similar but had slight differences. Also the list of packages is not
  duplicated for nightly and latest anymore; nightly uses all packages
  and latest uses packages that have at least one git tag.

* Building source packages is decoupled from uploading them. A separate
  script build_srcpkg.py can be used to just build the deb + rpm spec
  source packages, without interacting with the OBS server.

* The scripts can optionally run in docker with a command-line switch,
  and this is used by jenkins. This way we don't need to install
  more dependencies on the host such as rebar3 which is now needed for
  erlang/osmo_dia2gsup.

* Add erlang/osmo_dia2gsup and run its generate_build_dep.sh (SYS#6006)

I have done the new implementation in python to make use of argparse
and to be able to use try/except and print a trace when building one
package fails.

Example output:
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly_obs.osmocom.org/48/console
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_latest_obs.osmocom.org/46/console

Change-Id: I45a555d05a9da808c0fe0145aae665f583cb80d9
changes/82/28882/2
Oliver Smith 5 months ago
parent 871ff5c685
commit 35030b7618
  1. 4
      .gitignore
  2. 21
      jobs/osmocom-obs.yml
  3. 146
      scripts/common-obs-conflict.sh
  4. 326
      scripts/common-obs.sh
  5. 48
      scripts/obs/build_srcpkg.py
  6. 35
      scripts/obs/data/Dockerfile
  7. 26
      scripts/obs/data/Release.key
  8. 5
      scripts/obs/data/rpmlintrc
  9. 145
      scripts/obs/lib/__init__.py
  10. 103
      scripts/obs/lib/config.py
  11. 109
      scripts/obs/lib/debian.py
  12. 62
      scripts/obs/lib/docker.py
  13. 109
      scripts/obs/lib/git.py
  14. 104
      scripts/obs/lib/metapkg.py
  15. 120
      scripts/obs/lib/osc.py
  16. 84
      scripts/obs/lib/rpm_spec.py
  17. 157
      scripts/obs/lib/srcpkg.py
  18. 215
      scripts/obs/update_obs_project.py
  19. 247
      scripts/osmocom-latest-packages.sh
  20. 6
      scripts/osmocom-next-packages.sh
  21. 282
      scripts/osmocom-nightly-packages.sh

4
.gitignore vendored

@ -6,12 +6,10 @@ tokens.txt
jenkins_jobs.ini
jenkins-jobs.ini
__pycache__/
_cache/
_temp/
_deps/
_release_tarballs/
_docker_playground
_repo_install_test_data/
_repo_install_test_cache/
# osmocom-nightly-packages.sh
nightly-3g_*

@ -4,9 +4,12 @@
jobs:
- Osmocom_OBS_{type}_{server}
type:
- nightly
- latest
# - next (disabled: OS#5322)
- nightly:
# For nightly we don't provide ABI compatibility, make sure packages
# from different build dates are not mixed by accident
conflict_version: "$(date +%Y%m%d%H%M)"
- latest:
conflict_version: ""
server:
- build.opensuse.org:
proj: "network:osmocom"
@ -28,10 +31,16 @@
default: 'refs/remotes/origin/master'
builders:
- shell: |
export PROJ={proj}:{type}
export OBS_SERVER={server}
export PYTHONUNBUFFERED=1
./scripts/osmocom-{type}-packages.sh
./scripts/obs/update_obs_project.py \
--apiurl {server} \
--conflict-version {conflict_version} \
--docker \
--feed {type} \
--git-fetch \
--meta \
{proj}:{type}
scm:
- git:
branches:

@ -1,146 +0,0 @@
#!/bin/sh
# Create conflicting dummy packages in OBS (opensuse build service), so users can't mix packages
# built from different branches by accident
OSMO_OBS_CONFLICT_PKGVER="${OSMO_OBS_CONFLICT_PKGVER:-1.0.0}"
# Create the conflicting package for debian
#
# $1: name of dummy package (e.g. "osmocom-nightly")
# $2-*: name of conflicting packages (e.g. "osmocom-latest")
#
# Generates the following directory structure:
# debian
# ├── changelog
# ├── compat
# ├── control
# ├── copyright
# ├── rules
# └── source
# └── format
osmo_obs_prepare_conflict_deb() {
local pkgname="$1"
shift
local oldpwd="$PWD"
mkdir -p "debian/source"
cd "debian"
# Fill control
cat << EOF > control
Source: ${pkgname}
Section: unknown
Priority: optional
Maintainer: Oliver Smith <osmith@sysmocom.de>
Build-Depends: debhelper (>= 9)
Standards-Version: 3.9.8
Package: ${pkgname}
Depends: \${misc:Depends}
Architecture: any
EOF
printf "Conflicts: " >> control
first=1
for i in "$@"; do
if [ "$first" -eq 1 ]; then
first=0
else
printf ", " >> control
fi
printf "%s" "$i" >> control
done
printf "\n" >> control
cat << EOF >> control
Description: Dummy package, which conflicts with: $@
EOF
# Fill changelog
cat << EOF > changelog
${pkgname} (${OSMO_OBS_CONFLICT_PKGVER}) unstable; urgency=medium
* Dummy package, which conflicts with: $@
-- Oliver Smith <osmith@sysmocom.de> Thu, 13 Jun 2019 12:50:19 +0200
EOF
# Fill rules
cat << EOF > rules
#!/usr/bin/make -f
%:
dh \$@
EOF
# Finish up debian dir
chmod +x rules
echo "9" > compat
echo "3.0 (native)" > source/format
touch copyright
cd "$oldpwd"
}
# Create the conflicting package for rpm (e.g. contrib/osmocom-nightly.spec.in). The remaining
# placeholders are replaced in osmo_obs_add_rpm_spec().
#
# $1: name of dummy package (e.g. "osmocom-nightly")
# $2-*: name of conflicting packages (e.g. "osmocom-latest")
osmo_obs_prepare_conflict_rpm() {
local pkgname="$1"
shift
local spec_in="contrib/$pkgname.spec.in"
mkdir -p contrib
cat << EOF > "$spec_in"
Name: $pkgname
Version: @VERSION@
Release: 0
Summary: Dummy package, which conflicts with: $@
License: AGPL-3.0-or-later
Group: Hardware/Mobile
Source: @SOURCE@
EOF
for i in "$@"; do
echo "Conflicts: $i" >> "$spec_in"
done
cat << EOF >> "$spec_in"
%description
Dummy package, which conflicts with: $@
%files
EOF
}
# Print names of packages that the conflict package from the current feed
# (e.g. osmocom-nightly) should conflict with (e.g. osmocom-latest,
# osmocom-next, osmocom-2021q1)
osmo_obs_prepare_conflict_args() {
for i in $FEEDS_ALL; do
if [ "$i" != "$FEED" ]; then
echo "osmocom-$i"
fi
done
}
# Create conflicting packages, based on global $FEED and $FEEDS_ALL vars
osmo_obs_prepare_conflict() {
local pkgname="osmocom-$FEED"
local conflict_args="$(osmo_obs_prepare_conflict_args)"
local oldpwd="$PWD"
mkdir -p "$pkgname"
cd "$pkgname"
osmo_obs_prepare_conflict_deb "$pkgname" $conflict_args
osmo_obs_prepare_conflict_rpm "$pkgname" $conflict_args
# Put in git repository
git init .
git add -A
git commit -m "auto-commit: $pkgname dummy package" || true
git tag -f "$OSMO_OBS_CONFLICT_PKGVER"
cd "$oldpwd"
}

@ -1,326 +0,0 @@
#!/bin/sh
# Various common code used in the OBS (opensuse build service) related osmo-ci shell scripts
. "$(dirname "$0")/common-obs-conflict.sh"
FEEDS_ALL="
2021q1
2021q4
2022q1
2022q2
latest
next
nightly
"
osmo_cmd_require \
dch \
dh \
dpkg-buildpackage \
gbp \
git \
meson \
mktemp \
osc \
patch \
sed
if [ -z "$PROJ" ]; then
echo "PROJ environment variable is not set"
exit 1
fi
if [ -z "$OBS_SERVER" ]; then
echo "OBS_SERVER environment variable is not set"
exit 1
fi
# Related configuration sections are in .oscrc (OS#5557)
echo "Using OBS server: $OBS_SERVER"
shopt -s expand_aliases
alias osc="osc -A '$OBS_SERVER'"
# Add dependency to all (sub)packages in debian/control and commit the change.
# $1: path to debian/control file
# $2: package name (e.g. "libosmocore")
# $3: dependency package name (e.g. "osmocom-nightly")
# $4: dependency package version (optional, e.g. "1.0.0.202101151122")
osmo_obs_add_depend_deb() {
local d_control="$1"
local pkgname="$2"
local depend="$3"
local dependver="$4"
if [ "$pkgname" = "$depend" ]; then
echo "NOTE: skipping dependency on itself: $depend"
return
fi
if [ -n "$dependver" ]; then
depend="$depend (= $dependver)"
fi
# Note: adding the comma at the end should be fine. If there is a Depends: line, it is most likely not empty. It
# should at least have ${misc:Depends} according to lintian.
sed "s/^Depends: /Depends: $depend, /g" -i "$d_control"
git -C "$(dirname "$d_control")" commit -m "auto-commit: debian: depend on $depend" .
}
# Add dependency to all (sub)packages in rpm spec file
# $1: path to rpm spec file
# $2: package name (e.g. "libosmocore")
# $3: dependency package name (e.g. "osmocom-nightly")
# $4: dependency package version (optional, e.g. "1.0.0.202101151122")
osmo_obs_add_depend_rpm() {
local spec="$1"
local pkgname="$2"
local depend="$3"
local dependver="$4"
if [ "$pkgname" = "$depend" ]; then
echo "NOTE: skipping dependency on itself: $depend"
return
fi
if [ -n "$dependver" ]; then
depend="$depend = $dependver"
fi
( while IFS= read -r line; do
echo "$line"
case "$line" in
# Main package
"Name:"*)
echo "Requires: $depend"
;;
# Subpackages
"%package"*)
echo "Requires: $depend"
;;
# Build recipe
"%build"*)
if [ -n "$dependver" ]; then
cat << EOF
# HACK: don't let rpmlint abort the build when it finds that a library depends
# on a package with a specific version. The path used here is listed in:
# https://build.opensuse.org/package/view_file/devel:openSUSE:Factory:rpmlint/rpmlint-mini/rpmlint-mini.config?expand=1
# Instead of writing to the SOURCES dir, we could upload osmocom-rpmlintrc as
# additional source for each package. But that's way more effort, not worth it.
echo "setBadness('shlib-fixed-dependency', 0)" \\
> "%{_sourcedir}/osmocom-rpmlintrc"
EOF
fi
;;
esac
done < "$spec" ) > "$spec.new"
mv "$spec.new" "$spec"
}
# Copy a project's rpm spec.in file to the osc package dir, set the version/source, depend on the conflicting dummy
# package and 'osc add' it
# $1: oscdir (path to checked out OSC package)
# $2: repodir (path to git repository)
# $3: package name (e.g. "libosmocore")
# $4: dependency package name (e.g. "osmocom-nightly")
# $5: dependency package version (optional, e.g. "1.0.0.202101151122")
osmo_obs_add_rpm_spec() {
local oscdir="$1"
local repodir="$2"
local name="$3"
local depend="$4"
local dependver="$5"
local spec_in="$(find "$repodir" -name "$name.spec.in")"
local spec="$oscdir/$name.spec"
local tarball
local version
local epoch
if [ -z "$spec_in" ]; then
echo "WARNING: RPM spec missing: $name.spec.in"
return
fi
cp "$spec_in" "$spec"
osmo_obs_add_depend_rpm "$spec" "$name" "$depend" "$dependver"
# Set version and epoch from "Version: [EPOCH:]VERSION" in .dsc
version="$(grep "^Version: " "$oscdir"/*.dsc | cut -d: -f2- | xargs)"
case $version in
*:*)
epoch=$(echo "$version" | cut -d : -f 1)
version=$(echo "$version" | cut -d : -f 2)
;;
esac
if [ -n "$epoch" ]; then
sed -i "s/^Version:.*/Version: $version\nEpoch: $epoch/g" "$spec"
else
sed -i "s/^Version:.*/Version: $version/g" "$spec"
fi
# Set source file
tarball="$(cd "$oscdir" && ls -1 *_*.tar.*)"
sed -i "s/^Source:.*/Source: $tarball/g" "$spec"
osc add "$spec"
}
# Get the path to a distribution specific patch, either from osmo-ci.git or from the project repository.
# $PWD must be the project repository dir.
# $1: distribution name (e.g. "debian8")
# $2: project repository (e.g. "osmo-trx", "limesuite")
osmo_obs_distro_specific_patch() {
local distro="$1"
local repo="$2"
local ret
ret="$OSMO_CI_DIR/obs-patches/$repo/build-for-$distro.patch"
if [ -f "$ret" ]; then
echo "$ret"
return
fi
ret="debian/patches/build-for-$distro.patch"
if [ -f "$ret" ]; then
echo "$ret"
return
fi
}
# Check if checkout or build of a given package should be skipped, based on the
# PACKAGES environment variable.
# $1: package name (e.g. "libosmocore")
osmo_obs_skip_pkg() {
local pkgname="$1"
if [ -z "$PACKAGES" ]; then
# Don't skip
return 1
fi
for i in "osmocom-$FEED" $PACKAGES; do
if [ "$i" = "$pkgname" ]; then
return 1
fi
done
# Skip
return 0
}
# Copy an already checked out repository dir and apply a distribution specific patch.
# $PWD must be where all repositories are checked out in subdirs.
# $1: distribution name (e.g. "debian8")
# $2: project repository (e.g. "osmo-trx", "limesuite")
osmo_obs_checkout_copy() {
local distro="$1"
local repo="$2"
local patch
if osmo_obs_skip_pkg "$repo"; then
return
fi
echo
echo "====> Checking out $repo-$distro"
# Verify distro name for consistency
local distros="
debian8
debian10
"
local found=0
local distro_i
for distro_i in $distros; do
if [ "$distro_i" = "$distro" ]; then
found=1
break
fi
done
if [ "$found" -eq 0 ]; then
echo "ERROR: invalid distro name: $distro, should be one of: $distros"
exit 1
fi
# Copy
if [ -d "$repo-$distro" ]; then
rm -rf "$repo-$distro"
fi
cp -a "$repo" "$repo-$distro"
cd "$repo-$distro"
# Commit patch
patch="$(osmo_obs_distro_specific_patch "$distro" "$repo")"
if [ -z "$patch" ]; then
echo "ERROR: no patch found for distro=$distro, repo=$repo"
exit 1
fi
patch -p1 < "$patch"
git commit -m "auto-commit: apply $patch" debian/
cd ..
}
# Run git-version-gen inside Osmocom repositories, so the .tarball-version
# becomes part of the source repository. Usually this would be done with
# "make dist", but we use git-buildpackage instead.
osmo_obs_git_version_gen() {
if [ -x ./git-version-gen ]; then
./git-version-gen . > .tarball-version 2>/dev/null
fi
}
# Return a version based on the latest tag and commit (e.g. "1.5.1.93.47cc")
# or fall back to the last debian version (e.g. "2.2.6"). Run
# osmo_obs_git_version_gen before. $PWD must be inside a git repository.
osmo_obs_get_commit_version() {
local version=""
if [ -e ".tarball-version" ]; then
version=$(cat .tarball-version)
# debian doesn't allow '-' in version.
version=$(echo "$version" | sed 's/-/./g' )
fi
# deb version
deb_version=$(head -1 debian/changelog | cut -d ' ' -f 2 | sed 's,(,,' | sed 's,),,')
if [ -z "$version" ] || [ "$version" = "UNKNOWN" ]; then
version="$deb_version"
else
# add epoch from debian/changelog
case $deb_version in
*:*)
epoch=$(echo "$deb_version" | cut -d : -f 1)
version=$epoch:$version
;;
esac
fi
echo -n "$version"
}
# Verify that $FEED is in $FEEDS and $FEEDS_ALL
osmo_obs_verify_feed() {
local i
local j
for i in $FEEDS; do
if [ "$i" != "$FEED" ]; then
continue
fi
for j in $FEEDS_ALL; do
if [ "$j" = "$i" ]; then
return
fi
done
echo "feed found in FEEDS but not FEEDS_ALL: $FEED"
exit 1
done
echo "unsupported feed: $FEED"
exit 1
}

@ -0,0 +1,48 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import argparse
import lib
import lib.config
import lib.docker
import lib.git
import lib.metapkg
import lib.srcpkg
def main():
parser = argparse.ArgumentParser(
description="Clone the git repository and build the debian source"
" package as well as an rpm .spec file. This is the same"
" code that runs to generate source packages which we"
" upload to https://obs.osmocom.org."
f" Output dir: {lib.config.path_temp}/srcpkgs")
lib.add_shared_arguments(parser)
parser.add_argument("package", nargs="?",
help="package name, e.g. libosmocore or open5gs")
args = parser.parse_args()
if not args.meta and not args.package:
print("ERROR: specify -m and/or a package. See -h for help.")
exit(1)
lib.set_cmds_verbose(args.verbose)
if args.docker:
lib.docker.run_in_docker_and_exit(__file__, args)
lib.check_required_programs()
if args.package:
lib.check_package(args.package)
lib.remove_temp()
if args.meta:
lib.metapkg.build(args.feed, args.conflict_version)
if args.package:
lib.srcpkg.build(args.package, args.feed, args.conflict_version,
args.git_fetch)
if __name__ == "__main__":
main()

@ -0,0 +1,35 @@
FROM debian:bullseye
ARG UID
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y --no-install-recommends \
ca-certificates \
gnupg2 \
&& \
apt-get clean
COPY Release.key /tmp/Release.key
RUN apt-key add /tmp/Release.key && \
rm /tmp/Release.key && \
echo "deb https://downloads.osmocom.org/packages/osmocom:/latest/Debian_11/ ./" \
> /etc/apt/sources.list.d/osmocom-latest.list
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y --no-install-recommends \
debhelper \
dh-python \
dpkg-dev \
fakeroot \
git \
meson \
osc \
python3-setuptools \
rebar3 \
sed \
&& \
apt-get clean
RUN useradd --uid=${UID} -m user
USER user

@ -0,0 +1,26 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.5 (GNU/Linux)
mQENBGKzE1QBCADFcM3ZzggvgxNRNNqDGWf5xIDAiK5qzFLdGes7L6F9VCHdaPy0
RAOB5bFb/Q1tSDFNEBLtaauXKz+4iGL6qMVjZcyjzpB5w4jKN+kkrFRhjDNUv/SH
BX6d+P7v5WBGSNArNgA8D1BGzckp5a99EZ0okMJFEqIcN40PD6OGugpq5XnVV5Nk
e93fLa2Cu8vhFBcVn6CuHeEhsmuMf6NLbQRNfNNCEEUYaZn7beMYtpZ7t1djsKx5
1xGm50OzI22FLu8lELQ9d7qMVGRG3WHYawX9BDteRybiyqxfwUHm1haWazRJtlGt
UWyzvwAb80BK1J2Nu5fbAa3w5CoEPAbUuCyrABEBAAG0JW9zbW9jb20gT0JTIFBy
b2plY3QgPG9zbW9jb21Ab3Ntb2NvbT6JAVQEEwEIAD4WIQRrKp83ktFetw1Oao+G
pzC2U3JZcwUCYrMV4wIbAwUJBB6yjwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAK
CRCGpzC2U3JZc4FRCACQQkKIrnvQ7n2u7GSmyVZa3I+oLoFXSGqaGyey5TW/nrMm
vFDKU3qliHiuNSmUY35SnAhXUsvqOYppxVRoO1MLrqUvzMOnIWqkJpf8mtjGUnsW
jyVeto7Rsjs75y2i1Hk+e7ljb/V65J3NlfrfEYWbqR9AKd53ReNXTdrQ0J05A38N
GdI4Ld/2lNISAwaBmGhqdeKsLHpQw/JERU1TApVJR1whFiIwDF1rOCg9GPnNKIk7
yRZdK267XzztrainX/cbPILyzUZEDhYs6wQuyACyQ1YUxZIxrwVfk7PMNay8CrLH
z42B73Ne5IAj8+op/3iJafFONLm7YXiDUFN+QDYAiQEzBBMBCAAdFiEExoiYhHND
S7aVYlnqa51NyAUyjdsFAmKzE1UACgkQa51NyAUyjdvuZgf+OXmr//i7u7Gg7eWB
7e0qUsyCId9lXS8J437x3K6ciJfD7/6RSy8TFW5Nglm/uSkbyq582I8t+SoOirMD
E6cg9U/5+h5s46bAf+Kd2XS/6tLGeNLM18i4el8CP06NpFzDrsKu76uYFpyRiiHD
otBdtgxeLJ83LugGfZslF+/5cigJkAJMhAdVvGO8h85R6fba8ZSOKtMKkaQRfi76
nhyOrJPlLuS+DLEnHwdkOFgtKnxHdjM97K+Tx0gisb6uwaWroXfSLnhP8RTLLZZy
Z+noU1Hw3c+mn4c/NYbcC/uwHYHKRzuf9gHnQ3dGgv0Z5sbeLRVo92hjGj7Ftlyd
4hmKBg==
=HxK4
-----END PGP PUBLIC KEY BLOCK-----

@ -0,0 +1,5 @@
# Don't abort the build when finding a library that depends on a package with
# a specific version. This is intentional for nightly builds, we don't want
# libraries from different build dates to be mixed as they might have ABI
# incompatibilities.
setBadness('shlib-fixed-dependency', 0)

@ -0,0 +1,145 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import importlib
import os
import shutil
import subprocess
import sys
import tempfile
import lib.config
cmds_verbose = False
def add_shared_arguments(parser):
parser.add_argument("-f", "--feed", help="package feed (default: nightly)",
metavar="FEED", default="nightly",
choices=lib.config.feeds)
parser.add_argument("-d", "--docker",
help="run in docker to avoid installing required pkgs",
action="store_true")
parser.add_argument("-g", "--git-fetch",
help="fetch already cloned git repositories",
action="store_true")
parser.add_argument("-m", "--meta", action="store_true",
help="build a meta package (e.g. osmocom-nightly)")
parser.add_argument("-c", "--conflict-version", nargs="?",
help="Of the generated source packages, all Osmocom"
" packages (not e.g. open5gs, see lib/config.py"
" for full list) depend on a meta-package such as"
" osmocom-nightly, osmocom-latest, osmocom-2021q1"
" etc. These meta-packages conflict with each"
" other to ensure that one does not mix e.g."
" latest and nightly packages by accident."
" With this -c argument, it is possible to let"
" these packages depend on a meta-package of a"
" specific version. This is used for nightly and"
" 20YYqX packages to ensure they are not mixed"
" from different build dates (ABI compatibility"
" is only on latest).")
parser.add_argument("-v", "--verbose", action="store_true",
help="always print shell commands and their output,"
" instead of only printing them on error")
def set_cmds_verbose(new_val):
global cmds_verbose
cmds_verbose = new_val
def check_required_programs():
ok = True
for program in lib.config.required_programs:
if not shutil.which(program):
print(f"ERROR: missing program: {program}")
ok = False
for module in lib.config.required_python_modules:
if not importlib.find_loader(module):
print(f"ERROR: missing python3 module: {module}")
ok = False
if not ok:
print("Either install them or use the -d argument to run in docker")
exit(1)
def check_package(package):
if package in lib.config.projects_osmocom:
return
if package in lib.config.projects_other:
return
print(f"ERROR: unknown package: {package}")
print("See packages_osmocom and packages_other in obs/lib/config.py")
exit(1)
def exit_error_cmd(completed, error_msg):
""" :param completed: return from run_cmd() below """
print()
print(f"ERROR: {error_msg}")
print()
print(f"*** command ***\n{completed.args}\n")
print(f"*** returncode ***\n{completed.returncode}\n")
print(f"*** output ***\n{completed.output}")
print("*** python trace ***")
raise RuntimeError("shell command related error, find details right above"
" this python trace")
def run_cmd(cmd, check=True, *args, **kwargs):
""" Like subprocess.run, but has check=True and text=True by default and
allows capturing the output while displaying it at the same time. By
default the output is hidden unless there's an error, with -v the
output gets written to stdout.
:returns: subprocess.CompletedProcess instance, but with combined
stdout + stderr written to ret.output
:param check: stop with error if exit code is not 0 """
global cmds_verbose
if cmds_verbose:
print(f"+ {cmd}")
with tempfile.TemporaryFile(encoding="utf8", mode="w+") as output_buf:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True, bufsize=1,
*args, **kwargs)
while True:
out = p.stdout.read(1)
if out == "" and p.poll() is not None:
break
if out != "":
output_buf.write(out)
if cmds_verbose:
sys.stdout.write(out)
sys.stdout.flush()
output_buf.seek(0)
setattr(p, "output", output_buf.read())
if p.returncode == 0 or not check:
return p
exit_error_cmd(p, "command failed unexpectedly")
def remove_temp():
run_cmd(["rm", "-rf", lib.config.path_temp])
def remove_cache_extra_files():
""" dpkg-buildpackage outputs all files to the top dir of the package
dir, so it will always put them in _cache when building e.g. the debian
source package of _cache/libosmocore. Clear all extra files from _cache
that don't belog to the git repositories which we actually want to
cache. """
run_cmd(["find", lib.config.path_cache, "-maxdepth", "1", "-type", "f",
"-delete"])
def get_output_path(project):
return f"{lib.config.path_temp}/srcpkgs/{os.path.basename(project)}"

@ -0,0 +1,103 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import os
# Lists are ordered alphabetically.
path_top = os.path.normpath(f"{os.path.realpath(__file__)}/../..")
path_cache = f"{path_top}/_cache"
path_temp = f"{path_top}/_temp"
# Keep in sync with packages installed in data/Dockerfile
required_programs = [
"dh",
"dh_python3",
"dpkg-buildpackage",
"fakeroot",
"find",
"git",
"meson",
"osc",
"rebar3",
"sed",
]
required_python_modules = [
"setuptools",
]
feeds = [
"2022q1",
"2022q2",
"latest",
"nightly",
]
# Osmocom projects: generated source packages will depend on a meta package,
# such as osmocom-nightly, osmocom-latest or osmocom-2022q1. This meta package
# prevents that packages from different feeds are mixed by accident.
projects_osmocom = [
"erlang/osmo_dia2gsup",
"libasn1c",
"libgtpnl",
"libosmo-abis",
"libosmo-dsp",
"libosmo-netif",
"libosmo-sccp",
"libosmocore",
"libsmpp34",
"libusrp",
"osmo-bsc",
"osmo-bts",
"osmo-cbc",
"osmo-e1d",
"osmo-gbproxy",
"osmo-ggsn",
"osmo-gsm-manuals",
"osmo-hlr",
"osmo-hnbgw",
"osmo-hnodeb",
"osmo-iuh",
"osmo-mgw",
"osmo-msc",
"osmo-pcap",
"osmo-pcu",
"osmo-remsim",
"osmo-sgsn",
"osmo-sip-connector",
"osmo-smlc",
"osmo-sysmon",
"osmo-trx",
"osmo-uecups",
"python/osmo-python-tests",
"rtl-sdr",
"simtrace2",
]
projects_other = [
"limesuite",
"neocon",
"open5gs",
]
git_url_default = "https://gerrit.osmocom.org" # /project gets appended
git_url_other = {
"libosmo-dsp": "https://gitea.osmocom.org/sdr/libosmo-dsp",
"limesuite": "https://github.com/myriadrf/LimeSuite",
"neocon": "https://github.com/laf0rge/neocon",
"open5gs": "https://github.com/open5gs/open5gs",
"rtl-sdr": "https://gitea.osmocom.org/sdr/rtl-sdr",
}
git_branch_default = "master"
git_branch_other = {
"open5gs": "main",
}
git_latest_tag_pattern_default = "^[0-9]*\\.[0-9]*\\.[0-9]*$"
git_latest_tag_pattern_other = {
"limesuite": "^v[0-9]*\\.[0-9]*\\.[0-9]*$",
"open5gs": "^v[0-9]*\\.[0-9]*\\.[0-9]*$",
}
docker_image_name = "debian-bullseye-osmocom-obs"

@ -0,0 +1,109 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import datetime
import os
import shlex
import lib
import lib.git
def control_add_depend(project, pkgname, version):
""" :param pkgname: of the meta-package to depend on (e.g. osmocom-nightly)
:param version: of the meta-pkgname to depend on or None """
repo_path = lib.git.get_repo_path(project)
if version:
depend = f"{pkgname} (= {version})"
else:
depend = pkgname
cmd = ["sed", f"s/^Depends: /Depends: {depend}, /", "-i", "debian/control"]
lib.run_cmd(cmd, cwd=repo_path)
def changelog_add_entry(project, feed, version):
""" :param version: for the new changelog entry """
repo_path = lib.git.get_repo_path(project)
changelog_path = f"{repo_path}/debian/changelog"
changelog_old = open(changelog_path).read()
# Package name might be different from project name, read it from changelog
pkgname = changelog_old.split(" ", 1)[0]
assert pkgname
# Debian doesn't allow '-' in version
version = version.replace("-", ".")
# Debian changelog requires this specific date format
date = datetime.datetime.now(datetime.timezone.utc)
date_str = date.strftime("%a, %d %b %Y %H:%M:%S %z")
# Add new changelog entry
with open(changelog_path, "w") as f:
f.write(f"{pkgname} ({version}) unstable; urgency=medium\n")
f.write("\n")
f.write(" * Automatically generated changelog entry for building the"
f" Osmocom {feed} feed\n")
f.write("\n")
f.write(f" -- Osmocom OBS scripts <info@osmocom.org> {date_str}\n")
f.write("\n")
f.write(changelog_old)
def fix_source_format(project):
""" Always use format "3.0 (native)" (e.g. limesuite has "3.0 (quilt)")."""
repo_path = lib.git.get_repo_path(project)
format_path = f"{repo_path}/debian/source/format"
if not os.path.exists(format_path):
return
expected = "3.0 (native)\n"
current = open(format_path, "r").read()
if current == expected:
return
print(f"{project}: fixing debian/source/format ({current.rstrip()} =>"
f" {expected.rstrip()})")
open(format_path, "w").write(expected)
def get_last_version_from_changelog(project):
repo_path = lib.git.get_repo_path(project)
changelog_path = f"{repo_path}/debian/changelog"
if not os.path.exists(changelog_path):
return None
changelog = open(changelog_path).read()
if not changelog:
return None
return changelog.split("(", 1)[1].split(")", 1)[0]
def changelog_add_entry_if_needed(project, feed, version):
""" Adjust the changelog if the version in the changelog is different from
the given version. """
version_changelog = get_last_version_from_changelog(project)
if version_changelog == version:
return
print(f"{project}: adding debian/changelog entry ({version_changelog} =>"
f" {version})")
changelog_add_entry(project, feed, version)
def build_source_package(project):
fix_source_format(project)
print(f"{project}: building debian source package")
lib.run_cmd(["dpkg-buildpackage", "-S", "-uc", "-us", "-d"],
cwd=lib.git.get_repo_path(project))
def move_files_to_output(project):
path_output = lib.get_output_path(project)
lib.run_cmd(f"mv *.tar* *.dsc {shlex.quote(path_output)}", shell=True,
cwd=lib.config.path_cache)

@ -0,0 +1,62 @@
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import os
import shutil
import subprocess
import sys
import lib
import lib.config
def build_image():
print(f"docker: building image {lib.config.docker_image_name}")
lib.run_cmd(["docker", "build",
"--build-arg", f"UID={os.getuid()}",
"-t", lib.config.docker_image_name,
f"{lib.config.path_top}/data"])
def get_oscrc():
ret = os.path.expanduser("~/.oscrc")
if "OSCRC" in os.environ:
ret = os.environ["OSCRC"]
if os.path.exists(ret):
return os.path.realpath(ret)
print("ERROR: couldn't find ~/.oscrc. Put it there or set OSCRC.")
exit(1)
def run_in_docker_and_exit(script_path, args, add_oscrc=False):
if "INSIDE_DOCKER" in os.environ:
return
if not shutil.which("docker"):
print("ERROR: docker is not installed")
exit(1)
oscrc = None
if add_oscrc:
oscrc = get_oscrc()
# Build the docker image. Unless it is up-to-date, this will take a few
# minutes or so, therefore print the output.
lib.set_cmds_verbose(True)
build_image()
lib.set_cmds_verbose(args.verbose)
cmd = ["docker", "run",
"-e", "INSIDE_DOCKER=1",
"-e", "PYTHONUNBUFFERED=1",
"--rm", "-v", f"{lib.config.path_top}:/obs"]
if oscrc:
cmd += ["-v", f"{oscrc}:/home/user/.oscrc"]
script_path = f"/obs/{os.path.basename(script_path)}"
cmd += [lib.config.docker_image_name, script_path] + sys.argv[1:]
print(f"docker: running: {os.path.basename(script_path)} inside docker")
ret = subprocess.run(cmd)
exit(ret.returncode)

@ -0,0 +1,109 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import os
import re
import lib.config
def get_repo_path(project):
return f"{lib.config.path_cache}/{os.path.basename(project)}"
def get_repo_url(project):
if project in lib.config.git_url_other:
return lib.config.git_url_other[project]
return f"{lib.config.git_url_default}/{project}"
def get_latest_tag_pattern(project):
if project in lib.config.git_latest_tag_pattern_other:
return lib.config.git_latest_tag_pattern_other[project]
return lib.config.git_latest_tag_pattern_default
def clone(project, fetch=False):
repo_path = get_repo_path(project)
url = get_repo_url(project)
if os.path.exists(repo_path):
if fetch:
print(f"{project}: cloning {url} (cached, fetching)")
lib.run_cmd(["git", "fetch"], cwd=repo_path)
else:
print(f"{project}: cloning {url} (cached, not fetching)")
return
print(f"{project}: cloning {url}")
os.makedirs(lib.config.path_cache, exist_ok=True)
lib.run_cmd(["git", "clone", url, repo_path])
lib.run_cmd(["git", "config", "user.name", "Osmocom OBS scripts"],
cwd=repo_path)
lib.run_cmd(["git", "config", "user.email", "info@osmocom.org"],
cwd=repo_path)
def clean(project):
repo_path = get_repo_path(project)
lib.run_cmd(["git", "clean", "-ffxd"], cwd=repo_path)
def checkout(project, branch):
repo_path = get_repo_path(project)
print(f"{project}: checking out {branch}")
lib.run_cmd(["git", "checkout", "-f", branch], cwd=repo_path)
lib.run_cmd(["git", "reset", "--hard", branch], cwd=repo_path)
def checkout_default_branch(project):
branch = lib.config.git_branch_default
if project in lib.config.git_branch_other:
branch = lib.config.git_branch_other[project]
checkout(project, f"origin/{branch}")
def get_latest_tag(project):
pattern_str = get_latest_tag_pattern(project)
pattern = re.compile(pattern_str)
repo_path = get_repo_path(project)
git_tag_ret = lib.run_cmd(["git", "tag", "-l", "--sort=-v:refname"],
cwd=repo_path)
for line in git_tag_ret.output.split('\n'):
line = line.strip('\r')
if pattern.match(line):
return line
lib.exit_error_cmd(git_tag_ret, f"couldn't find latest tag for {project},"
f" regex used on output: {pattern_str}")
def get_latest_tag_remote(project):
pattern_str = get_latest_tag_pattern(project)
pattern = re.compile(pattern_str)
print(f"{project}: getting latest tag from git remote")
ls_remote = lib.run_cmd(["git", "ls-remote", "--tags", "--sort=-v:refname",
get_repo_url(project)])
for line in ls_remote.output.split('\n'):
# Tags are listed twice, skip the ones with ^{} at the end
if "^{}" in line:
continue
if "refs/tags/" not in line:
continue
line = line.rstrip().split("refs/tags/")[1]
if pattern.match(line):
return line
# No tag found probably means the repository was just created and doesn't
# have a release tag yet
return None
def checkout_latest_tag(project):
checkout(project, get_latest_tag(project))

@ -0,0 +1,104 @@
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import os
import lib
import lib.config
import lib.debian
import lib.rpm_spec
def get_conflicts(feed):
ret = []
for f in lib.config.feeds:
if f == feed:
continue
ret += [f"osmocom-{f}"]
return ret
def prepare_source_dir(feed):
path = f"{lib.config.path_cache}/osmocom-{feed}"
if os.path.exists(path):
lib.run_cmd(["rm", "-rf", path])
os.makedirs(f"{path}/debian")
os.makedirs(f"{path}/contrib")
def generate_debian_pkg(feed, version):
path = f"{lib.config.path_cache}/osmocom-{feed}"
conflicts = get_conflicts(feed)
with open(f"{path}/debian/control", "w") as f:
f.write(f"Source: osmocom-{feed}\n")
f.write("Section: unknown\n")
f.write("Priority: optional\n")
f.write("Maintainer: Osmocom OBS scripts <info@osmocom.org>\n")
f.write("Build-Depends: debhelper (>= 10)\n")
f.write("Standards-Version: 3.9.8\n")
f.write("\n")
f.write(f"Package: osmocom-{feed}\n")
f.write("Depends: ${misc:Depends}\n")
f.write("Architecture: any\n")
f.write(f"Conflicts: {', '.join(conflicts)}\n")
f.write(f"Description: Dummy package, conflicts with {conflicts}\n")
with open(f"{path}/debian/changelog", "w") as f:
f.write(f"osmocom-{feed} ({version}) unstable; urgency=medium\n")
f.write("\n")
f.write(f" * Dummy package, which conflicts with: {conflicts}\n")
f.write("\n")
f.write(" -- Osmocom OBS scripts <info@osmocom.org> Tue, 25 Jul 2022"
" 15:48:00 +0200\n")
with open(f"{path}/debian/rules", "w") as f:
f.write("#!/usr/bin/make -f\n")
f.write("%:\n")
f.write("\tdh $@\n")
lib.run_cmd(["chmod", "+x", f"{path}/debian/rules"])
with open(f"{path}/debian/compat", "w") as f:
f.write("10\n")
def generate_rpm_spec(feed, version):
print(f"osmocom-{feed}: generating rpm spec file")
path = (f"{lib.config.path_cache}/osmocom-{feed}/contrib/osmocom-{feed}"
".spec.in")
conflicts = get_conflicts(feed)
with open(path, "w") as f:
f.write(f"Name: osmocom-{feed}\n")
f.write(f"Version: {version}\n")
f.write(f"Summary: Dummy package, conflicts with: {conflicts}\n")
f.write("Release: 0\n")
f.write("License: AGPL-3.0-or-later\n")
f.write("Group: Hardware/Mobile\n")
for conflict in conflicts:
f.write(f"Conflicts: {conflict}\n")
f.write("%description\n")
f.write(f"Dummy package, which conflicts with: {conflicts}\n")
f.write("%files\n")
def build(feed, conflict_version):
pkgname = f"osmocom-{feed}"
version = conflict_version if conflict_version else "1.0.0"
print(f"{pkgname}: generating meta package {version}")
prepare_source_dir(feed)
generate_debian_pkg(feed, version)
os.makedirs(lib.get_output_path(pkgname))
lib.remove_cache_extra_files()
lib.debian.build_source_package(pkgname)
lib.debian.move_files_to_output(pkgname)
generate_rpm_spec(feed, version)
lib.rpm_spec.copy_to_output(pkgname)
lib.remove_cache_extra_files()
return version

@ -0,0 +1,120 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import glob
import os
import shlex
import shutil
import lib
import lib.config
apiurl = None
def check_proj(obs_project):
if ":" in obs_project:
return
print(f"ERROR: this doesn't look like a valid OBS project: {obs_project}")
exit(1)
def set_apiurl(url):
global apiurl
apiurl = url
def run_osc(cmd, *args, **kwargs):
global apiurl
# For some osc commands like 'osc add *' it makes sense to use a flat
# string and shell=True, hence support both list and string in this wrapper
if isinstance(cmd, str):
if apiurl:
cmd = f"osc -A {shlex.quote(apiurl)} {cmd}"
else:
cmd = f"osc {cmd}"
else:
if apiurl:
cmd = ["osc", "-A", apiurl] + cmd
else:
cmd = ["osc"] + cmd
return lib.run_cmd(cmd, *args, **kwargs)
def get_remote_pkgs(proj):
print(f"OBS: getting packages in {proj}")
ret = run_osc(["list", proj])
return ret.output.rstrip().split("\n")
def get_package_version(proj, package):
print(f"{package}: getting OBS version")
ret = run_osc(["list", proj, os.path.basename(package)])
# Empty OBS package
if ret.output == '\n':
return "0"
# Extract the version from the dsc filename
for line in ret.output.split('\n'):
line = line.rstrip()
if line.endswith(".dsc"):
return line.split("_")[-1][:-4]
lib.exit_error_cmd(ret, "failed to find package version on OBS by"
" extracting the version from the .dsc filename")
def create_package(proj, package):
print(f"{package}: creating new OBS package")
# cut off repository prefix like in "python/osmo-python-tests"
package = os.path.basename(package)
path_meta = f"{lib.config.path_temp}/_meta"
path_meta_obs = f"source/{proj}/{package}/_meta"
with open(path_meta, "w") as f:
f.write(f'<package name="{package}" project="{proj}">\n')
f.write(f'<title>{package}</title>\n')
f.write('<description></description>\n')
f.write('</package>\n')
run_osc(["api", "-X", "PUT", "-T", path_meta, path_meta_obs])
os.unlink(path_meta)
def remove_temp_osc():
lib.run_cmd(["rm", "-rf", f"{lib.config.path_temp}/osc"])
def update_package(proj, package, version):
print(f"{package}: updating OBS package")
# cut off repository prefix like in "python/osmo-python-tests"
package = os.path.basename(package)
path_output = lib.get_output_path(package)
path_temp_osc = f"{lib.config.path_temp}/osc"
path_temp_osc_pkg = f"{path_temp_osc}/{proj}/{package}"
remove_temp_osc()
os.makedirs(path_temp_osc)
run_osc(["checkout", proj, package], cwd=path_temp_osc)
if glob.glob(f"{path_temp_osc_pkg}/*"):
run_osc("del *", shell=True, cwd=path_temp_osc_pkg)
lib.run_cmd(f"mv * {shlex.quote(path_temp_osc_pkg)}", shell=True,
cwd=path_output)
run_osc("add *", shell=True, cwd=path_temp_osc_pkg)
run_osc(["commit", "-m", f"upgrade to {version}"], cwd=path_temp_osc_pkg)
remove_temp_osc()

@ -0,0 +1,84 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
import glob
import os
import shutil