osmo-ci/jobs
Oliver Smith 35030b7618 scripts/obs: rewrite pushing source pkgs to OBS
Harald requested that the OBS scripts should not stop if building one
specific source package fails, instead it should keep going and report
at the end a non-success exit code.

Given that the shell script code has historically grown and became hard
to maintain, I decided to rewrite the scripts for implementing this
feature. This rewrite solves additional problems:

* No full checkout of an OBS project like network:osmocom:latest
  anymore, with lots of packages that won't get updated (e.g. the uhd
  package has a uhd-images_3.14.1.1.tar.xz file that is 108 MB). With
  the old code, developers had to wait minutes during the checkout
  before the script reaches code that is currently being developed. Now
  only single packages get checked out right before they get updated.

* No need to clone git repositories over and over. With the new code,
  git repos only get cloned if needed (for latest it is not needed if
  the remote git tag is the same as the version in OBS). During
  development, the cloned git repositories are cached.

* Output from commands like "git tag -l" is not written to the log
  unless they failed. This makes the log more readable, which is
  especially important when a package fails to build, we keep going and
  need to spot the build error in the middle of the log later on.

* No more duplicated code for nightly and latest scripts that worked
  similar but had slight differences. Also the list of packages is not
  duplicated for nightly and latest anymore; nightly uses all packages
  and latest uses packages that have at least one git tag.

* Building source packages is decoupled from uploading them. A separate
  script build_srcpkg.py can be used to just build the deb + rpm spec
  source packages, without interacting with the OBS server.

* The scripts can optionally run in docker with a command-line switch,
  and this is used by jenkins. This way we don't need to install
  more dependencies on the host such as rebar3 which is now needed for
  erlang/osmo_dia2gsup.

* Add erlang/osmo_dia2gsup and run its generate_build_dep.sh (SYS#6006)

I have done the new implementation in python to make use of argparse
and to be able to use try/except and print a trace when building one
package fails.

Example output:
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly_obs.osmocom.org/48/console
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_latest_obs.osmocom.org/46/console

Change-Id: I45a555d05a9da808c0fe0145aae665f583cb80d9
2022-08-01 16:29:20 +02:00
..
README.adoc jobs: use "cmd: |", not "# keep first line ..." 2019-04-17 15:35:27 +00:00
coverity.yml jobs/coverity.yml: add parameter for git branch 2022-07-12 14:45:09 +02:00
gerrit-lint.yml add libosmo-pfcp 2022-06-17 16:33:10 +02:00
gerrit-verifications-dahdi.yml dahdi-linux: Add gerrit build verification against master 2022-04-24 19:48:47 +02:00
gerrit-verifications.yml jobs/gerrit-verifications: add osmo-opencm3-projects 2022-07-12 14:33:30 +00:00
master-builds-dahdi.yml jobs: stretch -> buster 2022-07-06 15:20:22 +00:00
master-builds.yml jobs/master-builds: enable osmo-opencm3-projects 2022-07-12 14:33:30 +00:00
octsim_osmo-ccid-firmware.yml jobs/octsim_osmo-ccid-firmware: new job 2021-12-14 13:22:04 +00:00
osmo-gsm-manuals-trigger.yml jobs/osmo-gsm-manuals-trigger: new job 2021-01-13 12:38:12 +01:00
osmo-gsm-tester-builder.yml jobs: ogt: Rename osmo-iuh->osmo-hnbgw build jobs 2022-01-12 10:34:06 +01:00
osmo-gsm-tester-runner.yml jobs: ogt: Rename osmo-iuh->osmo-hnbgw build jobs 2022-01-12 10:34:06 +01:00
osmo-gsm-tester-trigger.yml jobs: Add OGT open5gs build jobs 2021-03-15 19:13:07 +01:00
osmo-gsm-tester_run-gerrit.sh jobs: osmo-gsm-tester: Drop use of OSMO_GSM_TESTER_CONF for default conf dir 2020-05-07 19:16:15 +02:00
osmo-gsm-tester_run-prod.sh jobs: osmo-gsm-tester: Drop use of OSMO_GSM_TESTER_CONF for default conf dir 2020-05-07 19:16:15 +02:00
osmo-gsm-tester_run-rnd.sh jobs: osmo-gsm-tester: Drop use of OSMO_GSM_TESTER_CONF for default conf dir 2020-05-07 19:16:15 +02:00
osmo-gsm-tester_ttcn3.sh jobs: osmo-gsm-tester_ttcn3: Fix path to jenkins script 2020-06-09 14:10:46 +02:00
osmocom-build-tags-against-master.yml jobs: add osmocom-build-tags-against-master.yml 2019-03-26 15:54:46 +01:00
osmocom-depcheck.yml jenkins: Send build failures to jenkins-notifications mailing list 2019-03-03 16:04:41 +01:00
osmocom-list-commits.yml jobs: add osmocom-list-commits.yml 2019-03-20 11:41:28 +01:00
osmocom-obs.yml scripts/obs: rewrite pushing source pkgs to OBS 2022-08-01 16:29:20 +02:00
osmocom-release-tarballs.yml jobs/osmocom-release-tarballs: move to dp 2022-07-01 15:50:37 +00:00
registry-rebuild-upload-fpga-build.yml add registry-rebuild-upload-fpga-build.yml 2020-12-17 15:54:41 +01:00
registry-rebuild-upload-titan.yml jobs: Automatically push ttcn3-bts-tests to osmocom registry 2021-12-20 11:39:26 +01:00
registry-triggers.yml jobs/registry-triggers: add for osmo-ttcn3-hacks 2021-02-11 18:03:38 +00:00
registry-update-base-images.yml jobs/registry-update-base-images: drop old images 2022-07-06 15:20:22 +00:00
repo-install-test.yml jobs/repo-install-test: update debian versions 2022-07-06 15:20:22 +00:00
ttcn3-testsuites-kernel-git.yml jobs: explicitly enable ansi color wrapper 2022-01-17 16:36:03 +01:00
ttcn3-testsuites.yml jobs/ttcn3-testsuites: remove 2022q1 2022-07-15 13:39:54 +00:00
update-osmo-ci-on-slaves.yml jobs/update-osmo-*: change mail recipients 2022-06-29 09:33:30 +00:00
update-osmo-python-on-slaves.yml jobs/update-osmo-*: change mail recipients 2022-06-29 09:33:30 +00:00

README.adoc

These jenkins.osmocom.org job definitions, managed by
https://docs.openstack.org/infra/jenkins-job-builder/index.html[Jenkins Job Builder]

*Prepare:*

Install jenkins-job-builder:

For debian 9, see troubleshooting below. Otherwise:

  apt-get install jenkins-job-builder

Have a jenkins-job-builder.ini file. One of

    ~/.config/jenkins_jobs/jenkins_jobs.ini
    /etc/jenkins_jobs/jenkins_jobs.ini

or place one in here and pass it to jenkins-jobs using the --conf file.

Make sure the file not world readable to minimally safeguard your jenkins password.
Instead of using your jenkins password, use an *API Token*. To retrieve your token go
to Jenkins via a Webbrowser, click on your Username in the right corner, click on configure,
click on *Show API Toke...*.

jenkins_jobs.ini:

    [jenkins]
    user=my_user_name
    password=my_api_token
    url=https://jenkins.osmocom.org/jenkins

and

    chmod go-rwx jenkins_jobs.ini

*Update a single job on jenkins.osmocom.org:*

    jenkins-jobs --conf jenkins_jobs.ini update gerrit-verifications.yml gerrit-osmo-msc

NOTE: when you supply a name not defined in that yml file, you will not get an
error message, just nothing will happen.

*Update all jobs of one file:*

    jenkins-jobs --conf jenkins_jobs.ini update gerrit-verifications.yml

*Update all jobs in all files:*

    jenkins-jobs --conf jenkins_jobs.ini update ./

*Troubleshooting:*

- 'jenkins.JenkinsException: create[gerrit-osmo-msc] failed'

jenkins.osmocom.org is not reachable, or URL in the config file is erratic.
Make sure it is exactly

    url=https://jenkins.osmocom.org/jenkins

- newlines:

Use 'key: |' to keep new lines in multiline values, e.g.:
  - shell: |
      echo hello
      echo world

See also:
* https://yaml-multiline.info/
* https://stackoverflow.com/a/21699210

- jobs named on cmdline are not updated:

Make sure the job name is correct, or just issue an entire yml file without
individual job names.

Also be aware that jobs are only actually updated when anything changed.

- debian 9

For jenkins-job-builder to work on debian 9:

add 'testing' to /etc/apt/sources.list like

  deb http://ftp.de.debian.org/debian/ testing main

then

    sudo apt-get update
    sudo apt-get -t testing install python-jenkins
    sudo apt-get install jenkins-job-builder

and make sure to *not* pick the python3 versions
(or you'll need to add catching of HTTPError to jenkins/__init__.py in
'def maybe_add_crumb')

*Jenkins labels*

The build jobs are using labels to specify on which nodes the job should be run.
Please use the following labels

- osmocom-master-debian8: are nodes which can build osmocom-master and is a debian 8
- osmocom-master-debian9: are nodes which can build osmocom-master and is a debian 9
- osmocom-gerrit-debian8: are nodes which can build osmocom-gerrit and is a debian 8
- osmocom-gerrit-debian9: are nodes which can build osmocom-gerrit and is a debian 9