Harald requested that the OBS scripts should not stop if building one
specific source package fails, instead it should keep going and report
at the end a non-success exit code.
Given that the shell script code has historically grown and became hard
to maintain, I decided to rewrite the scripts for implementing this
feature. This rewrite solves additional problems:
* No full checkout of an OBS project like network:osmocom:latest
anymore, with lots of packages that won't get updated (e.g. the uhd
package has a uhd-images_3.14.1.1.tar.xz file that is 108 MB). With
the old code, developers had to wait minutes during the checkout
before the script reaches code that is currently being developed. Now
only single packages get checked out right before they get updated.
* No need to clone git repositories over and over. With the new code,
git repos only get cloned if needed (for latest it is not needed if
the remote git tag is the same as the version in OBS). During
development, the cloned git repositories are cached.
* Output from commands like "git tag -l" is not written to the log
unless they failed. This makes the log more readable, which is
especially important when a package fails to build, we keep going and
need to spot the build error in the middle of the log later on.
* No more duplicated code for nightly and latest scripts that worked
similar but had slight differences. Also the list of packages is not
duplicated for nightly and latest anymore; nightly uses all packages
and latest uses packages that have at least one git tag.
* Building source packages is decoupled from uploading them. A separate
script build_srcpkg.py can be used to just build the deb + rpm spec
source packages, without interacting with the OBS server.
* The scripts can optionally run in docker with a command-line switch,
and this is used by jenkins. This way we don't need to install
more dependencies on the host such as rebar3 which is now needed for
erlang/osmo_dia2gsup.
* Add erlang/osmo_dia2gsup and run its generate_build_dep.sh (SYS#6006)
I have done the new implementation in python to make use of argparse
and to be able to use try/except and print a trace when building one
package fails.
Example output:
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly_obs.osmocom.org/48/console
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_latest_obs.osmocom.org/46/console
Change-Id: I45a555d05a9da808c0fe0145aae665f583cb80d9
Fix the pattern that gets the source tarball with ls in order to update
the "Source:" line in the spec file. The previous assumption was that it
would always match:
ls -1 "${name}_"*".tar."*
But this isn't the case anymore since a spec file was added to
libosmo-dsp where the package name ($name) is libosmo-dsp, but the
tarball doesn't have the minus character:
libosmodsp_0.4.0.1.d7d9.202207150002.tar.xz
Therefore just change the pattern to:
ls -1 *_*.tar.*
We can do this because there is always only one tarball in the directory
and we wouldn't use ".tar." in a package name.
Fixes: OS#5619
Change-Id: Ic704f480102ab6f90f3936a3be30e3020402e750
Make the variable available, as it's needed by the scripts that
osmocom-packages-docker.sh calls since
Ib271e16e93ebc58ed1ecb0ea881e1f1370e40928.
Change-Id: I8e53523b9b1ceafa45fe7b3dca2731d4e1c74959
Mention that new projects should be added to the components list, as a
lot of projects were not listed there. I've just added all missing
ones. Put it as comment in prepare_source_Osmocom.sh because that's
probably the most likely spot where people will see it while adding a
new project.
Change-Id: I48630f4eb5b4f2b7b714697d15432c0d71f136f9
Build debian-bullseye-erlang on x86_64 nodes only, as the image doesn't
get used on nodes with other arches. The image is only used by
gerrit-verifications jobs for osmo_dia2gsum and osmo_gsup, which are
restriected to run on these nodes:
https://jenkins.osmocom.org/jenkins/label/osmocom-gerrit-debian9/
Remove the check and comments for not building for debian 10 x32, as
that node has been replaced with a debian 11 one.
Related: OS#5453
Change-Id: I5636dfc9e06a44b096f9dc55528d677ea2dfc352
Allow configuring a different git branch before starting the build. This
is especially useful to reproduce a failure that only happens when using
coverity tools during a build.
Change-Id: I5f7c615ff5af4bf1f34ccef62767b5ca71cd2d8d
Run the coverity/jenkins.sh from the osmo-ci.git repository that jenkins
checks out at the beginning of the build, instead of using a copy in ~/
that is updated whenever pushing to master. This makes it less
confusing, I configured the job to use a WIP branch and didn't realize
that it still used the code from master.
Change-Id: Ie992d71d2afec6982447b8fe3626c821a9e48d19
* no need to mention "(jenkins-job-builder)" in the display-name
* set mail to jenkins-notifications ML like most other jobs
Change-Id: I0284dd7354157e057903f53cde149ce652e76550
Revert 3038c19155, which adds a second
osmocom-coverity-deps job that apparently was never deployed (or was
deployed and then deleted). The mentioned curl dependency shouldn't go
here either.
Change-Id: I5d4709c0019ef4bae6ccbd11c28eeb23d75692c1
The coverity jenkins job has been failing since we tried to build tests
for all projects (and in an earlier attempt, build and run tests for all
projects). Make the coverity job work again by skipping build of tests
for asn1c. I looked into it earlier and the cause wasn't obvious to me,
and it's currently not worth spending time on.
Fix for:
libtool: error: cannot find the library '../../skeletons/libasn1cskeletons.la' or unhandled argument '../../skeletons/libasn1cskeletons.la'
Change-Id: Iba0bfe0e753812058436eaf515391eb1f4b09951
osmo-python-tests was added to latest just like it was in nightly, by
adding the following two lines to build_osmocom():
checkout python/osmo-python-tests
...
build osmo-python-tests
But the build fails, because the "checkout" function clones into a
"python/osmo-python-tests" directory in latest, and not into a
"osmo-python-tests" directory like in nightly. Make it behave the same
by adjusting the "checkout" function in nightly.
(This code has historically grown and should be refactored imho, but
let's fix this bug first.)
Fix for:
====> Building osmo-python-tests
+ cd /home/osmocom-build/jenkins/workspace/Osmocom_OBS_latest_obs.osmocom.org/osmo-python-tests
./scripts/osmocom-latest-packages.sh: 99: cd: can't cd to .../osmo-python-tests
Change-Id: I97074ee953cadac476fe2e4b686689c26bac1ea1
No need to mirror these images if we don't use them anymore.
Remove debian 8 and 9, and ubuntu zesty. The latter is the codename for
ubuntu 17.04, which is a non-LTS and was unsupported since 2018-01-13.
Related: OS#5601
Change-Id: If1c198038b0fd5faf0763172447962969448940e
In change 82265bc46d [1] I attempted
to fix aaeab8c976 [2], however both
pushd and popd commands are not available in plain shell.
Change-Id: Ib63ded66781c774e16773537c3b943237904b03e
Fixes: [1] I92b648cd1e80399561067eae99426faed1ff1e25
Fixes: [2] Idf0714a59356633021be21acc47f9471d059b7fe
Compiling unit tests would allow Coverity to analyze them too and thus
find more bugs. The problem is that binaries listed in check_PROGRAMS
are not compiled during 'make all', but only during 'make check'.
One approach [1] is to run 'make check' in do_build(), however it breaks
'Coverity-Upload' job because some unit tests are failing. This is why
I had to revert bf84ba9f69 together with
ae41942214 that was supposed to fix it.
Even if all unit tests were passing, we still don't want to execute them
because running tests is beyond the scope of 'Coverity-Upload' job.
Another approach is to patch Makefile.am files replacing check_PROGRAMS
with noinst_PROGRAMS, so that the test binaries are compiled during
'make all' but not executed.
Change-Id: Idf0714a59356633021be21acc47f9471d059b7fe
Related: [1] I3f3149a61df96646798c72d0ad31c41b97e6b81f
The purpose of "cut -d / -f 2-" is to strip off leading "./" in
the paths. The same can be achieved by doing "-printf '%P\n'".
Matching "*.c" and "*.h" files can be expressed as "-name *.[hc]".
Change-Id: Iae3fc5c8842df6926e6ff16a41be5663f1dedd1b
Move the long shell command into a jenkins.sh in docker-playground, and
move the osmocom-release-tarballs.sh script there as well. This makes it
easier to test the script, and only patching one repository is needed
from now on when changing to a newer debian version.
Depends: docker-playground I4cf66097f508c029f688ec8af1cab2238d42f84f
Change-Id: I15f6c08b807f1462f91e95b624cdc1aad0592dd6
These two projects have been released recently as part of Osmocom CNI
202206. Let's enable testing them since version in latest is good enough
for testing.
Change-Id: I1587e8400eff2d8b1ec26ab2fff1fbfa772fe8f8
When running these scripts locally while developing, it's more useful to
not fetch the sources and clean the git repository instead. Add
environment variables that allow doing this.
Change-Id: Id6d3cdabb776435df36c0145d93e514693062fca
Don't attempt to clone layer1-api if it exists already. Jenkins deletes
the sources for every run, but for development it is useful to be able
to run this script multiple times without getting errors.
Change-Id: I17bd76a5ff6abafd4bd0189073930f45de20afe8
'make check' does not pass for asn1c, it fails with the following. I
looked into it and the cause is not obvious to me, so in order to make
the jenkins job pass again I've changed it back to just run 'make'
instead.
Fix for:
...
fatal: making test-suite.log: failed to create ../tests/98-attribute-class-OK.asn1.log
fatal: making test-suite.log: failed to create ../tests/99-class-sample-OK.asn1.trs
fatal: making test-suite.log: failed to create ../tests/99-class-sample-OK.asn1.log
Makefile:735: recipe for target 'test-suite.log' failed
make[3]: *** [test-suite.log] Error 1
make[3]: Leaving directory '/home/osmocom-build/osmo-ci/coverity/source-Osmocom/asn1c/libasn1fix'
Makefile:841: recipe for target 'check-TESTS' failed
Disable 'make check' for libusrp too as it also didn't pass:
Making check in usrp2
make[4]: Entering directory '/tmp/coverity/source-Osmocom/libusrp/firmware/src/usrp2'
test -f `basename 'eeprom_boot.a51'` || ln -s 'eeprom_boot.a51' .
test -f ../common/`basename 'eeprom_boot.a51'` -o \
\! -f `dirname 'eeprom_boot.a51'`/../common/`basename 'eeprom_boot.a51'` \
|| ln -s `dirname 'eeprom_boot.a51'`/../common/`basename 'eeprom_boot.a51'` ../common/`basename 'eeprom_boot.a51'`
sdas8051 -plosgff `basename 'eeprom_boot.a51'`
sdcc -mmcs51 --no-xinit-opt -I../../../firmware/include -I../../../firmware/src/usrp2 -I../../../firmware/src/common -I../../../firmware/src/common -DHAVE_USRP2 \
-c -o eeprom_init.rel `test -f 'eeprom_init.c' || echo './'`eeprom_init.c
test -f `basename '_startup.a51'` || ln -s '_startup.a51' .
test -f ../common/`basename '_startup.a51'` -o \
\! -f `dirname '_startup.a51'`/../common/`basename '_startup.a51'` \
|| ln -s `dirname '_startup.a51'`/../common/`basename '_startup.a51'` ../common/`basename '_startup.a51'`
sdas8051 -plosgff `basename '_startup.a51'`
make[4]: *** No rule to make target '../../lib/libfx2.lib', needed by 'eeprom_boot.ihx'. Stop.
All other 'make check' calls do work as expected, the build script runs
through again with this patch.
Fixes: bf84ba9f ("coverity/build_Osmocom.sh: run 'make check' in do_build()")
Change-Id: I6b9eb8e73c51d925897579fb51bc0a1e86e77270
This enables building and running tests (check_PROGRAMS), so that
Coverity will be able to find bugs in their code too.
Change-Id: I3f3149a61df96646798c72d0ad31c41b97e6b81f
Currently there is only one jenkins node labeled as
osmo-gsm-tester-build. Build the osmo-gsm-tester container only there,
not on all x86_64 nodes.
The /var/tmp/osmo-gsm-tester/state directory is set up by
ansible/roles/gsm-tester/tasks/main.yml.
Related: https://jenkins.osmocom.org/jenkins/label/osmo-gsm-tester-build/
Change-Id: I9191d6d54d516ce75f3f576f34463dcaad22c85a
Starting in February 2022, coverity stopped to support the old
2019.03 version that we were using. Unfortunately that didn't
generate any e-mail notification to the project admins so it
remained undetected until now :(
What sucks is that the filename states 2021.12 but then the
actual directory created upon extraction is 2021.12.1, meaning
our ansible rule breaks. We have to rename the tarball to 2021.12.1
before running the playbook...
Closes: OS#5594
Change-Id: I950834cc2ad70979a53a563610bbc9feef1f5f20
Split the OBS related jenkins jobs into two, one that uploads to the
new obs.osmocom.org and one that uploads to build.opensuse.org.
Set an alias for "osc" that sets the right server, and set the
expand_aliases shopt so the bash scripts that source common-obs.sh will
not ignore the alias.
Related: OS#5557
Change-Id: Ib271e16e93ebc58ed1ecb0ea881e1f1370e40928
Oliver Smith
Harald Welte
Vadim Yanitskiy
Pau Espin