Commit Graph

1401 Commits

Author SHA1 Message Date
Oliver Smith a13ce691d1 scripts/docker-cleanup: buildkit cache too
In newer docker versions, a buildkit cache was introduced. It gets used
while building images. Clean it up as well.

Related: https://osmocom.org/projects/osmocom-servers/wiki/Docker_cache_clean_up
Change-Id: Icf5237def75d4bcef6b0065f3f1f1da2ff362322
2023-11-21 13:01:03 +00:00
Harald Welte b301da1369 ansible: add raspbian11build container on lx2 machine
Change-Id: I739e40cf6ef8d77585dfeb067ed75e258d94bbb3
Related: OS#5873
2023-11-18 15:22:43 +01:00
Harald Welte a05542cf58 ansible/jenkins-slave/osmo-ci: Install 'make' required for osmo-ci
TASK [osmocom-jenkins-slave : rebuild osmocom docker images for jenkins build tests] ******************
fatal: [lx2-raspbian11build-ansible]: FAILED! => {
    "changed": true,
    "cmd": "./scripts/osmo-ci-docker-rebuild.sh",
    "delta": "0:00:04.129667",
    "end": "2023-11-18 14:09:52.142684",
    "rc": 1,
    "start": "2023-11-18 14:09:48.013017"
}

STDERR:

+ make -C ../debian-bookworm-build-arm BUILD_ARGS=--pull UPSTREAM_DISTRO=debian:bookworm DISTRO=debian-bookworm IMAGE=osmocom-build/debian-bookworm-build-arm OSMO_TTCN3_BRANCH=master
./scripts/osmo-ci-docker-rebuild.sh: 203: make: not found

Change-Id: I785e4224e729accec15beaa3aef8e01f12624d9d
2023-11-18 15:22:43 +01:00
Harald Welte 506e58eb5b ttcn3-slave: Install sudo
The sudo tool is required by the ansible 'become_user' used later in ttcn3-slave.yml

TASK [osmocom-jenkins-slave : ensure jenkins folder exists] *******************************************
fatal: [lx2-raspbian11build-ansible]: FAILED! => {
    "changed": false,
    "rc": 127
}

MODULE_STDOUT:
/bin/sh: 1: sudo: not found

Related: OS#5873
Change-Id: I2d50ef2dee8e060ff4f84810f2bba62d4801d638
2023-11-18 15:13:18 +01:00
Harald Welte f61843c0e8 hosts: Add build4-obsworker
Change-Id: I0c663fa281042aa98822d27a67add8ba07dca172
2023-11-18 11:38:46 +01:00
Harald Welte ced38b31bc obs-worker: support more recent opensuse versions
We don't want to install new builders with 15.3, and the playbook
should just detect the installed version rather than making assumptions

Change-Id: I96f9896c0eeab120633d5800198d8b93371f7a5a
2023-11-18 11:38:08 +01:00
Harald Welte 26ae81f6bd obs-mirror: rsync over from remote server via rsync
We no longer run the archive on the same machine/lxc as the normal
downloads server

Change-Id: Id490690394db9eb773f945ad4d7e65fe7fd5e17a
Related: OS#6257
2023-11-15 15:17:07 +01:00
Harald Welte 0afdbcf8bd obs-mirror: keep nightly builds for two years
As we're moving the archive to a machine with sufficient storage,
let's extend the time we keep an archive of nightly builds from
one month to two years.

Change-Id: Ic8529ee5679f15c4e4940a5f095374605c2e5c79
Related: OS#6257
2023-11-14 21:23:59 +01:00
Oliver Smith 38707a473c tarballs: git archive: make proper tar.bz2
Compress the tarball from "git archive" with bzip2 to turn it into a
proper .tar.bz2. Previously the archive was named .tar.bz2, but it was
just a regular tarball as git archive didn't guess the type from the
filename (supposedly this works for .tar.gz and some others) and just
defaulted to tar.

Move the code that adds build_dep.tar.gz to the archive before the bzip2
call, because tar can't add files to the archive if it is compressed.

Change-Id: I5830f9123150540a6467c19c64461d2fa1972a52
2023-11-07 12:23:20 +01:00
Oliver Smith 0c79a15162 tarballs: git archive: put files in directory
Make the releases created from the "git archive" code path more similar
to the autotools releases, by having all files in a subdirectory with
the project name and tag name instead of having them directly in the
root of the tarball.

  $ tree osmo_dia2gsup-0.1.1 | head
  osmo_dia2gsup-0.1.1
  ├── build_dep.tar.gz
  ├── contrib
  │   ├── generate_build_dep.sh
  │   ├── jenkins.sh
  │   └── systemd
  │       └── osmo_dia2gsup.service
  ├── debian
  │   ├── changelog
  │   ├── compat

Change-Id: If37dcdc04278de0d4479cf72e0ca33c90b7deea8
2023-11-07 11:56:37 +01:00
Oliver Smith 88d62e3c69 tarballs: add osmo_dia2gsup
Change-Id: I9cbd3d3ce27a4b4c9c8e19b2876386d170d4f0a6
2023-11-07 11:40:06 +01:00
Oliver Smith a9e9c366b1 repo-install-test: ignore limesuite-images
Don't attempt to install limesuite-images. It runs a post-install script
that downloads files from an external server and fails currently, as
there are no images for 23.10. While we have limesuite-images packages
in our OBS repository, this is just a side-effect of building limesuite.
What we are really interested in is liblimesuite for osmo-trx, as I
understand.

Add --no-install-recommends to the apt-get install line in
install_repo_packages_debian, because the main limesuite package has
limesuite-images in recommends and would pull it in otherwise.

Change-Id: I237408c805977c831f352a57a301ea45753d1ac1
2023-11-02 21:42:19 +00:00
Oliver Smith 8ba48eb28a repo-install-test: fix ca-certificates-java error
The repo-install-test started to fail on debian 10 with the following
error. Apparently this happens when installing ca-certificates-java
after installing java. Add a workaround.

  Setting up ca-certificates-java (20190405) ...
  head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory
  Exception in thread "main" java.lang.InternalError: Error loading java.security file

Change-Id: I00b9c97d9d85fb37ba33a48caa732cd50de99683
2023-11-02 21:42:19 +00:00
Oliver Smith 25f74d3d51 Update git tag patterns for rtl-sdr, osmo-fl2k
With the v2.0.0 release of rtl-sdr, it was decided to include the v in
the tag pattern again, so adjust it in the related scripts. Adjust
osmo-fl2k too in the OBS scripts because there the tag pattern was not
set.

Change-Id: If1bb64722bd24b0079d2427ea3b0f5b5c87ef7ac
2023-11-02 13:53:22 +01:00
Oliver Smith 867f2d28fd OBS: add workaround for limesuite 23.10 release
The previous workaround for building on debian 12 is not needed anymore,
as the related patch has been merged. Replace it with a new workaround
needed for building the limesuite 23.10 release.

Change-Id: I569642ebb13ac3eb3e241642e299be0a0a2ab15d
2023-10-31 12:04:33 +01:00
Oliver Smith b365c1df68 scripts/tarballs: gapk: add install depends
Now that we run ./configure before building the tarball, make sure the
dependencies are installed.

Related: OS#6227
Change-Id: I82eb0a87f6e294fb258003ebe76f850a99952938
2023-10-26 12:31:52 +02:00
Oliver Smith b59c791d13 scripts/tarballs: gapk > v1.0: use make dist-bzip2
Use autotools logic to build tarballs for gapk above version v1.0.

Related: OS#6227
Change-Id: If9a04b458553e311cf8b891b204244db2b1ae2d8
2023-10-26 11:05:22 +02:00
Oliver Smith 3d2c3dd14c scripts/repo-install-test: fix sftp compat, part 2
Copy files inside repo-install-test, instead of copying the directory
again. Otherwise it creates a /repo-install-test/repo-install-test/ path
inside qemu and then fails to run the script inside the directory:

Fix for:
  sh: 0: cannot open /repo-install-test/run-inside-env.sh: No such file

Fixes: a7791217 ("scripts/repo-install-test: fix sftp compat")
Change-Id: I04d8c0e0fd31ee5d2a0912300dc93f5339bbef11
2023-10-23 12:57:54 +02:00
Oliver Smith a779121725 scripts/repo-install-test: fix sftp compat
When running scp, Debian 12 uses the SFTP protocol by default instead of
the legacy SCP protocol. For some reason, this means it doesn't create
the /repo-install-test directory inside QEMU:

  + … scp … -r scripts/repo-install-test root@127.0.0.1:/repo-install-test
  Warning: Permanently added '[127.0.0.1]:22793' (ED25519) to the list of known hosts.
  scp: realpath /repo-install-test: No such file
  scp: upload "/repo-install-test": path canonicalization failed
  scp: failed to upload directory scripts/repo-install-test to /repo-install-test

Changing to the legacy protocol with "scp -O" would make it work again,
but instead of doing that add the mkdir so it works with the new
protocol too.

Related: OS#6186
Change-Id: I740affad2ea794094c8faa89a89bcad094e0075d
2023-10-20 12:19:30 +02:00
Oliver Smith 48b3edb0cc scripts/tarballs: create release dir if missing
Fix for:
rsync: [Receiver] change_dir#3 "/downloads/home/releases/web-files/gapk" failed: No such file or directory (2)

Change-Id: I46310d633302028d486dd356147c9c4f35b24940
2023-10-19 11:37:53 +02:00
Oliver Smith 7fbb49f87d scripts/tarballs: add osmo-fl2k, rtl-sdr
Closes: OS#6226, OS#6225
Change-Id: Idff299c65902b6818e409a409a3d303e8fccc4f9
2023-10-19 11:09:43 +02:00
Oliver Smith 5597e82d55 scripts/tarballs: add gapk
Closes: OS#6227
Change-Id: I6f26b6f5953701b383acb3b93058b66a286f7989
2023-10-19 11:07:06 +02:00
Oliver Smith 4e0a34b26e scripts/common: osmo_git_last_commit…: add vars
Add variable names for $1, $2, $3 so the logic is easier to follow.

Related: OS#6227
Change-Id: I7abbcefc8d5bd4c5b84c30a7e38ca3dcb855dd56
2023-10-19 11:07:06 +02:00
Oliver Smith a498ca6f2a scripts/tarballs: add osmo-e1-recorder
Closes: OS#6228
Change-Id: I6bd8387f99c258226125b33705645c43aadd875f
2023-10-19 11:02:18 +02:00
Oliver Smith 4e3b7babda ansible: qemu: netcat -> netcat-openbsd
Make the ansible rule compatible with debian 12. The same package is
already in debian 11 and debian 10.

Fix for:
  E: Package 'netcat' has no installation candidate

Change-Id: Idcc90ad7a2a104a9b4f5a76c6c89d74604651530
2023-10-19 07:30:20 +00:00
Oliver Smith 367f550da9 ansible: jenkins: debian: add more packages
Add python3-setuptools, as it is needed to run the
update-osmo-python-on-slaves jenkins job.

Add bc, as it is needed in docker-playground.git's network_create() in
jenkins-common.sh.

Related: OS#6186
Change-Id: I1ff0711b7b29ba4dc78f51f257b986d7a0994a96
2023-10-19 07:30:20 +00:00
Oliver Smith 7edd780403 ansible: jenkins: debian: sort packages
Related: OS#6186
Change-Id: If4ce784821cc40a2e795afafa452c99bf0588253
2023-10-19 07:30:20 +00:00
Oliver Smith 6d4b0effa6 jobs/update-osmo-*: add build4-deb12build-ansible
Related: OS#6186
Change-Id: I1f732718df87ed3b850332dbf1a7f6b2f854f280
2023-10-19 07:30:20 +00:00
Oliver Smith 119bc1f44a ansible: jenkins: use default-jre-headless
Use default-jre-headless instead of giving the openjdk version, so we
don't need to adjust it to whatever the current version in debian is.

The default-jre-headless package is at least available since debian 10:
https://packages.debian.org/search?suite=buster&arch=any&searchon=names&keywords=default-jre-headless

Related: OS#6186
Change-Id: I6c5c74ff3d75dca2f89765d95c4b4c6d67ace703
2023-10-19 07:30:20 +00:00
Oliver Smith 7d90d5b17f ansible: jenkins: debian: remove qemu package
This is a dummy package that isn't available anymore in debian 12.

https://packages.debian.org/buster/qemu says:
> This package can safely be removed.

We install qemu-user-static and qemu-system-arm, which contain qemu
binaries.

Related: OS#6186
Change-Id: I4d5bd376b1e370d9636b9a2e9e710f5f20b9ae98
2023-10-19 07:30:20 +00:00
Oliver Smith 89c0b50a08 ansible: hosts: add build4-deb12build-ansible
Related: OS#6186
Change-Id: Ia4ce2ab074941dda6b7c820673108d19d4136664
2023-10-19 07:30:20 +00:00
Pau Espin 7be78df106 jobs/ttcn3-testsuites.yml: Add ttcn3-epdg-test
Related: OS#6204
Change-Id: I4b566c1e767c7a6d6cb474a257b0a2632a2afe92
2023-10-18 12:29:16 +02:00
Oliver Smith 45de5a99d9 ansible: known_hosts: update
After deploying with ansible to the jenkins nodes, these entries were
added.

Change-Id: I3e3841d9f1f5581067b7dd90cdfcaf8337700e6f
2023-10-16 11:25:12 +02:00
Oliver Smith 81b714b636 ansible: hosts: remove gtp0-deb(10,11)build
Remove nodes that don't exist anymore according to:
https://jenkins.osmocom.org/jenkins/computer/

Change-Id: I8e8ae59d4e4c8be85bba527ac89cce04b15fcaea
2023-10-16 11:21:56 +02:00
Oliver Smith 975edb5099 ansible: ttcn3: install netcat-openbsd
Make nc available on jenkins nodes running ttcn3 tests, as it is now
used by ttcn3-hss-test-ogs.

Related: docker-playground Iebeefd9689ae2efa54678b99ff929ca255ec64d1
Change-Id: I397d4fe33813b083b61cdabb1d00ab4ea63a456b
2023-10-16 11:17:34 +02:00
Pau Espin 35a1155a42 jobs/ttcn3-testsuites.yml: Add ttcn3-hss-test-ogs
Related: SYS#6588
Change-Id: Ic98a5a22b2c10b32052e613d0a46fbc7640c60d9
2023-10-10 19:26:08 +02:00
Oliver Smith 15777116dd obs: build_binpkg: no network during build
Simulate that we don't have network during package builds in gerrit
verifications, like it is the case when the packages will be built by
OBS.

I've also considered using network namespaces via "unshare -n" or
bubblewrap, but it wouldn't be possible to create a loopback device
inside the network namespace inside docker (maybe with --privileged, but
not worth it) and e.g. the osmo-mgw tests rely on lo.

Depends: osmo-epdg I2bb0b5f608cd5d9a24c2a367425c524447bd002c
Change-Id: I00282552e9988d66fdc269c6230f0e1432ace360
2023-10-06 14:05:51 +02:00
Oliver Smith 2e809f4637 jobs/master-builds: add erlang projects
Add the erlang projects found in gerrit-verifications to master-builds
too, as we have it with most other Osmocom projects.

Change-Id: I6cf5a3c1e52ee73ad63eb2d7d5b1af19a9809026
2023-10-06 08:21:48 +00:00
Oliver Smith 193e2a4ce6 jobs/master: use same url scheme as gerrit jobs
Make it easier to copy entries from gerrit-verifications.yml to
master-builds.yml by using the same url schema in both.

Change-Id: Ibe992dd8027a8b8df8623abc57a590972443449f
2023-10-06 08:21:48 +00:00
Oliver Smith f8e2ee24ed obs: srcpkg: support having no git tags
Instead of failing when a project has no git-version-gen script, and no
git tags, use 0.0.0 as version. This is needed for the new osmo-epdg
project, which doesn't have tags yet.

Change-Id: I2f0f409feb96611095dfa511a33ea730e5bbf3e4
2023-10-05 18:05:41 +02:00
Pau Espin 76a67bc8d7 obs: Fix typo in log line informing variable to be modified
Change-Id: I381f30ad2443bbd86a23d2808a5f8e3c650953b7
2023-10-05 17:36:42 +02:00
Pau Espin 325162a1fe obs: add erlang/osmo-epdg to projects_osmocom
Change-Id: I5c5537e2d96752cb68fb251552bd444a9abf8cd1
2023-10-05 17:36:42 +02:00
Oliver Smith 3f037f4a73 obs: srcpkg.py: Run generate_build_dep.sh for all projects
Adjust srcpkg.py to run a generate_build_dep.sh not only for
osmo_dia2gsup, but for any Osmocom project that has such a script. This
is in preparation for osmo-epdg, which will also have one.

Change-Id: Icd9dc734ec938398668640f77ec83aae4a1ee0f4
2023-10-05 17:35:02 +02:00
Pau Espin 9b9b2a2780 jobs/gerrit-verifications: Add osmo-epdg
Change-Id: I84accfc5065ed2e780c4ecb77885af61bc6fb72f
2023-10-05 16:22:49 +02:00
Oliver Smith b206b2f1d2 scripts/docker-cleanup: remove containers > 24h
Remove containers starting with jenkins- or having ttcn3 in the name, if
they have been running for more than 24 hours. This can happen with the
ttcn3 testsuites, as they typically start multiple docker containers in
the background (one per Osmocom program) before they start the testsuite
docker container in the foreground. Usually the clean up trap makes sure
that all containers get killed, but we have seen that a few containers
have been running for a few months. One reason for this could be
temporary loss of connection between the jenkins server and the node
running the job.

Extend the clean script to remove the containers that were not properly
removed by the clean up trap.

Historically we used to kill docker containers of the same name before
starting a testsuite, but this had the downside that we could not start
the same testsuite multiple times in parallel. This was refactored in
docker-playground Ifcd384272c56d585e220e2588f2186dc110902ed.

Change-Id: I58c17b57c998eaba411658e83b7295d7cfcf9a23
2023-10-04 17:53:51 +02:00
Oliver Smith 25db4a639e ansible/util/known_hosts: new file
Add the known hosts file that ansible creates.

Change-Id: Ibef171ac2de5193c2335d4d28ef9347cc88b4c49
2023-09-20 11:09:14 +02:00
Oliver Smith 9aab3f54ad ansible: increase docker_max_image_space
The purpose of the limit is to remove images that are not being used in
a long time. With the current limit of 40 GB it leads to images being
removed just a few hours after they were last used, resulting in lots
of needless rebuilds. Also unexpected errors with images that we just
assume to be there (debian-bookworm-build).

Increases the limit for the nodes that run most of the jobs to 80 GB.

I have verified that the nodes have enough free space.

Related: OS#6183
Change-Id: I8cfb21a86510e43b8347c7f17159575b695ce9ba
2023-09-20 11:06:49 +02:00
Oliver Smith 1e748b38a4 OBS: add git tag pattern for wireshark
Since previous patch 62693be1 ("OBS: fix generating wrong limesuite
version"), the scripts rely on having the correct pattern for the tag
in the config. Add the pattern for wireshark.

Fix for:
  fatal: No tags can describe '6d412b991d8c2f50b7854611b1a9ef567bab9962'.

Change-Id: Ic8e83b4d5d897cb6b94b04b1232486288f3a1f9b
2023-09-18 10:41:41 +02:00
Oliver Smith 9df253bb34 repo-install-test: skip soapysdr-module-lms7
Fix for:
  dpkg: error processing archive /tmp/apt-dpkg-install-4OjX3P/0924-soapysdr0.8-module-lms7_22.09.1+dfsg-1+b2_amd64.deb (--unpack):
   trying to overwrite '/usr/lib/x86_64-linux-gnu/SoapySDR/modules0.8/libLMS7Support.so', which is also in package soapysdr0.7-module-lms7:amd64 22.09.1.17.4828.202309142026

Change-Id: I51f3541c62edc78842d4c461362dd8e5ea2f5d94
2023-09-15 10:46:51 +02:00
Oliver Smith 62693be148 OBS: fix generating wrong limesuite version
Instead of using git-version-gen in all git repositories, only use it
where it is part of the repository. Use git directly to build the
version string otherwise. To fix generating the wrong version string for
limesuite, use "git describe --tags" instead of "git describe" as in
git-version-gen. This is needed because limesuite releases > 20.01 only
have lightweight (non-annotated) tags.

Fix for:
  limesuite: WARNING: version from changelog (22.09.1-1) is higher than
  version based on git tag (20.01.0.182-4828.202309140026), using
  version from changelog (git tag not pushed yet?)

Change-Id: I00d97d5cd12b2938a002d3e5c8ada6503f976e36
2023-09-14 15:40:53 +02:00