Commit Graph

922 Commits

Author SHA1 Message Date
Oliver Smith 88521fbc14 scripts/docker-cleanup.sh: conditional img clean
Only run the simple image clean code if docuum is not running. It works
well enough in most cases, but has the drawbacks that it never deletes
"latest" images or images not matching "^osmocom-build", and may delete
images that are still being used (OS#5447). With the other tool, all
images are considered for removal, and the ones that have not been used
the longest time are removed first.

Related: OS#5477, OS#5066, SYS#5827
Change-Id: I1cef0833c096de0fa5acf77156bb5dd362e2ef9c
2022-02-11 15:44:44 +01:00
Oliver Smith b5ebf6ea6b scripts/docker-cleanup.sh: use "docker system prune"
Do not only clean up dangling images, but also containers, volumes and
networks.

Related: SYS#5827
Change-Id: If441b251de50063f0229d36fb1bc260a4cb1dd87
2022-02-11 15:44:16 +01:00
Oliver Smith 2b77e64c48 scripts/docker-cleanup.sh: delete containers too
Related: SYS#5827
Change-Id: I73b2f13875286c1aaa5424809edab2202f41768b
2022-02-11 15:44:16 +01:00
Oliver Smith e94b29e837 scripts/docker-cleanup.sh: use set -x
Change-Id: Iba170128e55a9778467c3d3bcf33a91321a8c29f
2022-02-11 15:44:16 +01:00
Oliver Smith 17f82a260d ansible/roles/docker: install docuum
Allow setting a max amount of space that docker images should take up,
and delete the least recently used images when the amount is exceeded.

Related: https://osmocom.org/projects/osmocom-servers/wiki/Docker_cache_clean_up
Change-Id: I640b1e607feca87e7a578946ae4b8332ce854ab1
2022-02-11 15:44:16 +01:00
Oliver Smith f07b5c8a64 ansible/setup-jenkins-slave: accept rel-change
Fix for:
  W:This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
  E:Repository 'http://raspbian.raspberrypi.org/raspbian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'

Related: https://github.com/ansible/ansible/issues/48352
Change-Id: I90ee8dd49001d6810f2118e9feff3375e87052ea
2022-02-11 15:44:16 +01:00
Oliver Smith 6695a91c0b ansible/ansible.cfg: make command output readable
Have a readable output, instead of the default that wraps the output
with stripped new lines inside json.

Related: https://github.com/ansible/ansible/issues/27078#issuecomment-364560173
Change-Id: I88d584a1808d82d75906b350e3bffe9bc73a8c67
2022-02-11 15:44:16 +01:00
Oliver Smith 41e218a355 lint: catch missing --rm flag for 'docker run'
Not having the --rm flag causes docker containers to be stored forever,
so make sure we always add it to the "docker run" commands that we run
on jenkins via scripts in various repositories.

Depends: docker-playground I48b01c43fedf379b8a565eaab0369806d7831bd8
Related: https://osmocom.org/projects/osmocom-servers/wiki/Docker_cache_clean_up
Related: SYS#5827, OS#5099
Change-Id: I8ab9c291504475d670bdefc50c4524c5bdd4c880
2022-02-11 15:44:16 +01:00
Oliver Smith 2e88fa2663 scripts/osmocom-packages-docker: docker run --rm
Change-Id: I357ae8071d27088a2c0a74f962ac1589bff43a70
2022-02-11 15:44:16 +01:00
Oliver Smith 0b22171bc3 editorconfig: new file
Auto-configure editors with editorconfig plugin installed to indent yaml
files with two spaces in this repository.

Change-Id: I008ee040e2c15db11a35a061faa4270bacdbd10d
2022-02-11 15:44:16 +01:00
Vadim Yanitskiy f6f49d23a1 tasks/ttcn3-slave.yml: also install libosmocore-utils
This package provides osmo-config-merge, which is going to be used
by docker-playground for generating configuration files.

Change-Id: I87e54440a6b399aa0f0461ebf668991b5cdd0b07
Related: https://gerrit.osmocom.org/c/docker-playground/+/20622
2022-02-11 13:59:06 +00:00
Harald Welte 5cff7924ec jobs/master-builds.yml: Add master-pysim to publish manual
We only had gerrit build verification for pysim so far, but no 'master'
job that periodically builds/tests master (and publishes the manual).

As a reult, the pySim manual was still the original version from April
2021, while the code has of course progressed quite a bit in git since.

Change-Id: I4dce05cdb5f2c574256378d5fb66cff1c329c141
Closes: OS#5271
2022-02-09 17:48:04 +01:00
Pau Espin 1982f52f8e Deploy osmo-ci to ogt nodes and update it periodically
We need osmo-ci in those nodes in order for the docker old images to be
cleaned through cron entry installed by ansible docker role, which uses
/home/jenkins/osmo-ci/scripts/docker-cleanup.sh.

Related: OS#5441
Change-Id: Ieb6db0b24dce6099cfa24871a957471ca2c8bcee
2022-02-07 11:29:26 +01:00
Oliver Smith 7ad596e4d9 jobs/master-builds-dahdi.yml: new file
Related: OS#5407
Depends: dahdi-linux I904ab66a1ecd72492642ac2cc4cb102c7283c590
Change-Id: I8a479d0a214ff56c5b66f468a1c2b1b2f674e091
2022-02-04 15:05:04 +01:00
Oliver Smith b3cef938c9 jobs/gerrit-verifications-dahdi.yml: new file
Add gerrit verifications for the out-of-tree dahdi-linux kernel module.
Put this in a separate file (not gerrit-verifications.yml), because:
* not only gerrit git repo gets cloned, but also linux.git, and caching
  needs to be done properly for it
* so we can have one separate job per linux version that gets tested

Test v4.19 and v5.10 for now, as suggested in the issue.

Related: OS#5407
Depends: docker-playground Id72d19ad08681cd7cb3194de2226292f19e96df5
Change-Id: I65c7663a6b8045796536a727e4adf5554f16b2a9
2022-02-04 14:40:17 +01:00
Oliver Smith b7bf1b735d jobs/gerrit-lint: add dahdi-linux
Related: OS#5407
Change-Id: I7fdb40e40fb8cf8c9072a4e898aeffa43d1d7ae0
2022-02-04 12:36:28 +01:00
Oliver Smith 68d8fab771 jobs/registry-update-base-images: centos8 -> alma
Follow up on the change of using almalinux instead of centos8 in
docker-playground, with providing the almalinux base images in the
Osmocom registry.

Related: SYS#5818
Related: docker-playground I30e1a773b901b1d2187214445116c7f2aecc4e36
Change-Id: I93a8892da628b8d2f9cbb2a6453e8ab3153a9c4a
2022-02-03 09:31:58 +01:00
Oliver Smith c04fb387ea jobs/{master-builds,gerrit-*}: add osmo-bsc-nat
Related: SYS#5560
Change-Id: I240cb614bf3905d04bccc6a00c72567a2ad722d8
2022-01-31 14:05:34 +01:00
Daniel Willmann a96eeaade9 Disable ttcn3-hnbgw-test for now
Change-Id: Ia385c4155ffed761c5887fc4d6c76163ca732527
2022-01-26 11:44:46 +01:00
Harald Welte 8a3cd203f8 jenkins-jobs: add master build for opencm3 projects
Change-Id: I95f84079478ce57d8d5c0cffd71bc67e1605e68e
2022-01-18 14:58:41 +00:00
Oliver Smith 6350b2f537 Revert "remove linter jobs for osmo-e1d and osmo-e1-firmware"
Enable the linter again, as we decided to use a repo-specific checkpatch
config in these.

This reverts commit 2c6aa82c9a.

Related: OS#5399
Change-Id: Ib79a20658bfae64d9b718e53e6a9393708a4e694
2022-01-18 09:47:10 +01:00
Oliver Smith 3ee13bd9cd jobs: explicitly enable ansi color wrapper
For some reason (jenkins upgrade?), output is not colored anymore.
Explicitly enable the ansi color wrapper in the jjb configs, where we
have such colors in the output.

Change-Id: I2d3faaf608ae9d12573f5c0f740ff7ee4b33358e
2022-01-17 16:36:03 +01:00
Pau Espin 6942211175 jobs: ogt: Rename osmo-iuh->osmo-hnbgw build jobs
The real target was not osmo-iuh itself but osmo-hnbgw. Since osmo-hnbgw
is inits own repository now, let's rename it.

Change-Id: I72e93683ea355cf0886f08a3f31568f58ff42473
2022-01-12 10:34:06 +01:00
Neels Hofmeyr 09a4c75e2a jobs: disable ttcn3-upf-test
Change-Id: I0518a65ee68fc45bf65e10f327b1d590d7456d88
2022-01-12 03:53:25 +01:00
Harald Welte 685a362e0f cosmetic: Fix typo (no canes involved)
Change-Id: I4b3dbe6fdb6053a8dd2a6b4b3d06552df1107fdf
2022-01-10 16:16:07 +01:00
Harald Welte 2c6aa82c9a remove linter jobs for osmo-e1d and osmo-e1-firmware
tnt will remain to use his slightly not-really-osmocom coding style,
as expressed in https://gerrit.osmocom.org/c/osmo-e1-hardware/+/26841

Change-Id: If2b98bd19ce1d89970dc266119e83d2a1c648454
Related: OS#5399
2022-01-10 16:12:16 +01:00
Pau Espin 2362892acb Add new repo osmo-hnbgw (latest) to build jobs and scripts
Change-Id: I634ebbfa7f471342ffcf253fdb8a316f5e71089f
2022-01-05 12:29:40 +01:00
Pau Espin 702fd8ff94 Add new repo osmo-hnbgw (nightly) to build jobs and scripts
Change-Id: I2fdbd651e4b50191e1a67e183590de99d6366ca5
2022-01-05 11:39:41 +01:00
Pau Espin 7f89d91666 jobs/master-builds: Place osmo-hnodeb properly in dependency tree
Change-Id: Id66a8be3f665ff4dc88a754d072939fbda16d719
2022-01-04 19:22:37 +01:00
Harald Welte 4bf60895e9 obs-mirror: Keep more 'latest' distributions
'latest' doesn't change very often, so we can keep history for more
distros.

Change-Id: I3c8ac31bdf394f8b94ed643413ee2a1379eb76a2
2021-12-30 16:32:04 +01:00
Harald Welte fe8c4552d8 obs-mirror-archive: keep raspbian + debian 11 (instead of 10)
Debian / Raspbian 10 are getting a bit old, and our test suites also
all have shifted to 11 by now, so let's rather keep a nightly archive of
11 as opposed to 10

Change-Id: Id08acf82d9c959553684d0d95b31a8c0379115a6
2021-12-30 16:32:04 +01:00
Harald Welte cce962cdc0 obs-mirror.sh: Keep log files of rsync job
Before this commit, the obs-mirror.log was overwritten every night,
meaning we have no history of rsync log files.

Change-Id: I26392ed8615b9f4a6aa23e92b82ad9aff01eac4b
2021-12-29 17:07:22 +01:00
Neels Hofmeyr 0c5869f359 jobs: add osmo-upf (gerrit, master builds, ttcn3 tests)
Change-Id: I43cd55c9395f2c706369afe1a9a7f475be47339f
2021-12-28 15:57:54 +01:00
Oliver Smith 37fe0ed2f4 OBS, ttcn3-testsuites: add 2021q4
Related: SYS#5726
Change-Id: I4913218b192cf0cb1fb74b239ced9a0501c2eb48
2021-12-23 09:15:48 +00:00
Oliver Smith d7fe8fd047 lint: checkpatch_osmo: ignore TRACING_LOGGING
Don't recommend to use ftrace instead of using __func__ inside printf.
This only applies to the linux kernel, not to the Osmocom code base.

Related: OS#5087
Related: https://gerrit.osmocom.org/c/libosmocore/+/6357
Change-Id: Ied23be32d3371a8b9ac9938a336474f71e7a3ccd
2021-12-22 15:10:03 +01:00
Pau Espin 82f21fa6fc repo-install-test: Validate osmo-hnodeb --version
Change-Id: I87c9c375fa810223daadd5768d6a54e9935c34eb
2021-12-21 10:57:11 +01:00
Oliver Smith 958224ea62 contrib/obs-mirror: log DRY var in rm script
Related: SYS#5764
Change-Id: I705e52030d538e6d86c72f2bccb0eb97e6dab1de
2021-12-20 13:35:19 +00:00
Oliver Smith 9b694744ea contrib/obs-mirror: redirect all output to a log file
Related: SYS#5764
Change-Id: Ie56c901877a3b0a1ad566cb243ab698f292407d5
2021-12-20 13:35:19 +00:00
Oliver Smith e3aaae199f contrib/obs-mirror: stop script on rsync error
Sync files into a temporary directory first, and only rename the temp
dir to the final dir name if rsync succeeds. Otherwise, exit with error.

Related: SYS#5764
Change-Id: Icfefd8e4bae1fd6c73445c7427aaa842c0391b2d
2021-12-20 13:35:19 +00:00
Oliver Smith 68896ff0ae contrib/obs-mirror: adjust includelist path
Point at obs-mirror-include.txt from this git repository.

Related: OS#4862
Change-Id: I4c6a6e424fd727de4e8058c916aa7eb43e7eb4b2
2021-12-20 13:35:19 +00:00
Oliver Smith 05902bd35e contrib/obs-mirror: import
Import scripts from OS#4862 to have them properly versioned. We use this
to replicate Osmocom related packages from OBS to:
https://download.osmocom.org/obs-mirror

Put it in contrib/, because it doesn't need to get installed to the
jenkins nodes (as it would happen with everything in scripts/).

Related: OS#4862
Change-Id: I3f47cddbcee6c9b85fde77a039dcd5b54dd89f2d
2021-12-20 13:35:19 +00:00
Pau Espin 9832c6aa4e jobs: Automatically push ttcn3-bts-tests to osmocom registry
This image used to be pushed to the sysmocom registry, before the
osmocom one existed. Let's make it available in the osmocom one now that
we have it instead.

The new job triggers upon conclusion of the
registry-rebuild-upload-titan job, since anyway the ttcn3-bts-tests is
based on that image and hence requires it to be built prior to it.

Related: SYS#5765
Change-Id: I65e4e5a9e539de5bfd881e08d7c5842f2266ecee
2021-12-20 11:39:26 +01:00
Oliver Smith 37f480d441 registry-rebuild-upload-titan: s/stretch/bullseye/
Depends: docker-playground Ib3bdfa3bec8f8ef42c55ca61cdee8fbca923874f
Related: OS#4969
Change-Id: Ic117725f765a83cf3d2955e19441795fe6415dd2
2021-12-16 11:01:34 +01:00
Oliver Smith cc3e7efe1a jobs/registry-update-base-images: add d:bullseye
Related: OS#4969
Change-Id: I2f30c9ff3558e3fad60117d951c85c3548eddf1b
2021-12-16 10:44:39 +01:00
Oliver Smith f0d09c50af jobs/registry-update-base-images: one image per line
Make the list of images more git friendly, and sort the debian releases
by date.

Related: OS#4969
Change-Id: Ic724aa4b2c71aa398ad7f7049e8f3406a3883eaa
2021-12-16 10:44:39 +01:00
Oliver Smith 521bfea795 jobs/ttcn3: disable TTCN3-centos-remsim
It seems that TTCN3-centos-remsim-test was disabled manually via the web
UI and only got enabled recently after rolling out unrelated changes in
jobs/ttcn3-testsuites.yml. The tests do not run properly with the centos
version and it seems that nobody is using it, so disable them again for
now.

Change-Id: I4885333d63944af3510fdf0519233989808944ff
2021-12-15 17:18:37 +00:00
Oliver Smith 31edfd7116 jobs/octsim_osmo-ccid-firmware: new job
Import the existing (non jenkins-job-builder) job with
jenkins-job-wrecker. Change it to fetch the git repository, add missing
mkdir so it works with a fresh workspace too.

Fixes: SYS#5738
Change-Id: I19d18b24a6ae5948aeaf7c46083748c70cdb3b7b
2021-12-14 13:22:04 +00:00
Oliver Smith 2b1085886c lint: checkpatch: tweak FSF_MAILING_ADDRESS error
Change-Id: Ic76132c40a9c30a167c24c4c4b53180c4cc0fb86
2021-12-14 12:52:16 +01:00
Oliver Smith 444e048693 osmocom-release-tarballs: fix missing st2 clone
Clone simtrace2.git before trying to create the tarball with git.
Fix for:
  simtrace2
    simtrace2-0.1.tar.bz2 (creating)
  + cd /osmo-ci/_temp/repos/simtrace2
  /osmo-ci/scripts/osmocom-release-tarballs.sh: 195: cd: can't cd to /osmo-ci/_temp/repos/simtrace2

Related: OS#5347
Fixes: 0221a0 ("OSMO_RELEASE_REPOS: add simtrace2, osmo-remsim")
Change-Id: I0a845549ba1fe9f0d9ab55a5c5c7bf5b8f57caae
2021-12-14 08:53:18 +00:00
Oliver Smith b5617e2f84 jobs/registry-update-base-images: add centos7
Related: SYS#5754
Change-Id: I64ce913b79dad063fd34848be538b088d6b3402d
2021-12-13 16:36:15 +01:00