Commit Graph

1180 Commits (6dde3dcd3851adc52658af340084653496b0f438)

Author SHA1 Message Date
Neels Hofmeyr 7057a9f55d obs: tweak logging output
Change-Id: I203cb7f0aa21b59b06973e8f329cffdc64aa8d47
2022-08-17 15:32:07 +02:00
Neels Hofmeyr 39cde7f098 obs: add README
Explain how to use the obs/ scripts to test a private git branch.

Change-Id: I4d1303e0c04e827200b48a9fe4aea3680c9c9f84
2022-08-17 15:32:07 +02:00
Neels Hofmeyr 60fe5c86b5 obs: expand help for --feed
Change-Id: Iabd165e4f7a8c17b30f01ad27abd1a878233dabf
2022-08-17 15:32:07 +02:00
Neels Hofmeyr 872dbbd58b obs: add explicit --git-branch arg
Change-Id: I612c18e533542f442e0571927fee8b8be9fb8c2b
2022-08-17 15:32:07 +02:00
Oliver Smith 71bafe74d3 jobs/master-builds: archive core, testsuite.log
In case one of the master-buils is segfaulting or has an error in the
testsuite, collect the relevant files and store them as artifacts.

It was asked in the related issue to do this only on error, however
while there's an option to archive only artifacts when a build is
successful, there is not an option to do it only on failure. So just do
it always, the testsuite.log file is typically just a few kb in size.
core files should only get generated on error.

Closes: OS#5642
Change-Id: I91d4b995b16c8e028a5dff19b0e0f09c01d8c8c6
2022-08-17 08:10:12 +00:00
Neels Hofmeyr c90ee51321 obs: add libosmo-pfcp, osmo-upf
Change-Id: I7192f4c9e77035cebb147b8186296bcd741d419b
2022-08-16 23:14:57 +02:00
Harald Welte 5455b42405 update git URLs (git -> https; gitea/gerrit)
Unencrypted git:// protocol offers no integrity or authentication,
making it subject to tampering.  Use https:// instead.

Change-Id: Ia44e4127ff87a3d4d747225d3258188a0a891f1b
2022-08-13 04:02:51 +07:00
Harald Welte 942f5ef96c master-opencm3-projects: Build from gitea, not git is a mirror of, let's build from
the actual master repo, not from the (possibly slightly outdated) mirror

Change-Id: I12d090a9ee7fde09892567ada739826484a4a35b
2022-08-12 12:11:05 +02:00
Vadim Yanitskiy ac0b663951 {jobs,coverity}: add a new project - libosmo-gprs
Change-Id: I83e86084e9f7a7431bf5a70c372da1f9ccd28436
2022-08-10 04:39:36 +07:00
Vadim Yanitskiy 9f36fd75a4 OSMO_RELEASE_REPOS: add missing libosmo-pfcp and osmo-upf
Change-Id: Ib7431281ff78686087d23bb6d600e6c68faef6e5
2022-08-10 04:32:00 +07:00
Vadim Yanitskiy 33c09f4344 jobs/master-builds: fix libosmo-pfcp related trigger mess
* master-libosmo-pfcp is triggered directly by libosmocore, not
  by master-libosmo-netif, nor master-libosmo-abis, so
** fix comment explaining the hierarchy.

* master-osmo-upf is triggered by master-libosmo-pfcp, so
** it should not be triggered directly by libosmocore.

Change-Id: Ie4a5cb47464e9abfc9d04e63e9e71619f33ffd42
Fixes: Icb75adec886cf1053b80c968fea59e597ed289a7
2022-08-10 04:08:30 +07:00
Vadim Yanitskiy 6d81605790 coverity: add missing libosmo-pfcp, dependency of osmo-upf
Since recently [1] osmo-upf depends on libosmo-{gtlv,pfcp}.so, which
are provided by libosmo-pfcp.git.  We need to build libosmo-pfcp
before attempting to build osmo-upf, otherwise it fails:

  configure: error: Package requirements (libosmo-gtlv >= 0.1.0) were not met:

  Package libosmo-gtlv was not found in the pkg-config search path.
  Perhaps you should add the directory containing `libosmo-gtlv.pc'
  to the PKG_CONFIG_PATH environment variable
  Package 'libosmo-gtlv', required by 'world', not found

Change-Id: Ic4fe7b40cf9a57e0462b2c0806f8810fcc04a10c
2022-08-09 15:19:19 +07:00
Oliver Smith 8b6447a1be obs-mirror: use packages as source
Change the rsync command to use the packages from
/downloads/home/packages/web-files/osmocom: on the same server as
source, which is where is pushing the packages.

Remove xUbuntu_19.04 from the directories to sync, as we don't build for
it anymore. It's not an LTS release (unlike 18.04, 20.04) and it's EOL.

Related: OS#5557
Change-Id: Id6be69530a7aa65b0cf3866fc7e65af79c9d9d45
2022-08-05 08:16:11 +00:00
Oliver Smith 3f4ee14898 obs: add osmo-bsc-nat, osmo-fl2k
Looks like I copied the list of packages from the old latest script, so
these two that were only in nightly are missing. Once added, they will
be built for nightly again and not for latest as there's no tag matching
the patterin in the git repositories.

Change-Id: I4b96f15e95c70f1dca12a292964e6f12b04e4a04
2022-08-04 14:17:40 +00:00
Oliver Smith 7340fa7561 exclude dotfiles
There's a .~tmp~ file on the remote that currently causes the rsync
command to fail. Ignore all files starting with a dot.

Fix for:
  rsync: [sender] opendir "/opensuse/repositories/network:/osmocom:/nightly/Raspbian_11/.~tmp~" (in opensuse-full-really-everything-including-repositories) failed: Permission denied (13)

It's time to update the script to sync the packages
instead of the ones from, I'll do that in the next
patch. Nevertheless, I think it makes sense to add this exclude argument
in case OBS (our instance) also creates temporary files like this.

Change-Id: Iacee2ea4acada60d8de960592fb828911a55781e
2022-08-04 13:41:10 +00:00
Oliver Smith c1f64b40d1 Cosmetic: jobs/ttcn3: debian-stretch -> debian
Remove reference to stretch (debian 9), as the jobs run on a newer
debian version now. Remove the reference altogether so we don't need to
update it here. The current version can be looked up in

Change-Id: I06c74830b0f6fe0baed44c300f9e1023fb0522de
2022-08-04 12:43:15 +00:00
Oliver Smith 72bab749e5 obs: fix_source_format: ignore missing \n
Apparently a \n at the end of debian/source/format is not required, so
don't fix it if it is missing. This avoids printing this confusing line:
  osmo-trx: fixing debian/source/format (3.0 (native) => 3.0 (native))

Change-Id: I7f9bb22a389a2109109f7fecd3b7ae0413fe6f5b
2022-08-03 15:24:24 +02:00
Oliver Smith 35030b7618 scripts/obs: rewrite pushing source pkgs to OBS
Harald requested that the OBS scripts should not stop if building one
specific source package fails, instead it should keep going and report
at the end a non-success exit code.

Given that the shell script code has historically grown and became hard
to maintain, I decided to rewrite the scripts for implementing this
feature. This rewrite solves additional problems:

* No full checkout of an OBS project like network:osmocom:latest
  anymore, with lots of packages that won't get updated (e.g. the uhd
  package has a uhd-images_3.14.1.1.tar.xz file that is 108 MB). With
  the old code, developers had to wait minutes during the checkout
  before the script reaches code that is currently being developed. Now
  only single packages get checked out right before they get updated.

* No need to clone git repositories over and over. With the new code,
  git repos only get cloned if needed (for latest it is not needed if
  the remote git tag is the same as the version in OBS). During
  development, the cloned git repositories are cached.

* Output from commands like "git tag -l" is not written to the log
  unless they failed. This makes the log more readable, which is
  especially important when a package fails to build, we keep going and
  need to spot the build error in the middle of the log later on.

* No more duplicated code for nightly and latest scripts that worked
  similar but had slight differences. Also the list of packages is not
  duplicated for nightly and latest anymore; nightly uses all packages
  and latest uses packages that have at least one git tag.

* Building source packages is decoupled from uploading them. A separate
  script can be used to just build the deb + rpm spec
  source packages, without interacting with the OBS server.

* The scripts can optionally run in docker with a command-line switch,
  and this is used by jenkins. This way we don't need to install
  more dependencies on the host such as rebar3 which is now needed for

* Add erlang/osmo_dia2gsup and run its (SYS#6006)

I have done the new implementation in python to make use of argparse
and to be able to use try/except and print a trace when building one
package fails.

Example output:

Change-Id: I45a555d05a9da808c0fe0145aae665f583cb80d9
2022-08-01 16:29:20 +02:00
Oliver Smith 871ff5c685 obs-mirror-include.txt: drop latest/Debian_9.0
Don't attempt to sync Debian_9.0, as we are not building for it anymore
and it doesn't exist anymore.

This should fix the verify-obs-mirror job.

Change-Id: I25ba2790a8ae6a54068b8ad4035a669f48795509
2022-08-01 11:50:12 +02:00
Oliver Smith 7d299912ca scripts: osmo_obs_add_rpm_spec: fix tar pattern
Fix the pattern that gets the source tarball with ls in order to update
the "Source:" line in the spec file. The previous assumption was that it
would always match:
  ls -1 "${name}_"*".tar."*

But this isn't the case anymore since a spec file was added to
libosmo-dsp where the package name ($name) is libosmo-dsp, but the
tarball doesn't have the minus character:

Therefore just change the pattern to:
  ls -1 *_*.tar.*

We can do this because there is always only one tarball in the directory
and we wouldn't use ".tar." in a package name.

Fixes: OS#5619
Change-Id: Ic704f480102ab6f90f3936a3be30e3020402e750
2022-07-18 10:56:38 +02:00
Oliver Smith 54eef69a40 scripts/osmocom-packages-docker: pass OBS_SERVER
Make the variable available, as it's needed by the scripts that calls since

Change-Id: I8e53523b9b1ceafa45fe7b3dca2731d4e1c74959
2022-07-15 16:30:28 +02:00
Oliver Smith 444ad3677e jobs/ttcn3-testsuites: remove 2022q1
Related: SYS#6007
Change-Id: Ic4cfd2dbf85fb08394166605c37f1778bf7dcb44
2022-07-15 13:39:54 +00:00
Oliver Smith b55a41f2c6 lint/typedefs_osmo: add A_SEQUENCE_OF, A_SET_OF
Fix need consistent spacing around '*' lint errors where these are used.

Change-Id: I50e8e1ddcf6a4f927f533094accf7f8a18b523d1
2022-07-15 11:30:32 +02:00
Oliver Smith 0b2de357a9 Cosmetic: coverity: how to add a new project
Mention that new projects should be added to the components list, as a
lot of projects were not listed there. I've just added all missing
ones. Put it as comment in because that's
probably the most likely spot where people will see it while adding a
new project.

Change-Id: I48630f4eb5b4f2b7b714697d15432c0d71f136f9
2022-07-13 07:47:14 +00:00
Harald Welte d5b6cba4e9 ansible: Use instead of
Change-Id: Ib60eb87c7b0c17d20bbbfa8b94d8262f0c165b1c
Related: OS#5557
2022-07-13 07:41:55 +00:00
Harald Welte ac694eacb6 ansible/hosts: Replace failed OBS/qemu with real arm machine
We gave up on trying to fix OBS cross-building via qemu, and instead
bought a real ARM64 machine.

Change-Id: I5f3ffe451d029ba459575881bf88f2527fe9fd15
2022-07-13 07:41:55 +00:00
Oliver Smith 2bb0a98756 jobs/gerrit-verifications: add osmo-opencm3-projects
Related: SYS#6041
Change-Id: Ic514e96ff515c5703f39f5a31f419ff344179368
2022-07-12 14:33:30 +00:00
Oliver Smith 86c77d3958 jobs/master-builds: enable osmo-opencm3-projects
Use debian-bullseye-jenkins to have picolibc-arm-none-eabi available.

Related: OS#6041
Change-Id: Iefff6e3c3b18f4af99079635a45320faaac54e50
2022-07-12 14:33:30 +00:00
Oliver Smith 6754a89e4c docker-rebuild: add debian-bullseye-jenkins
Related: OS#6041
Depends: docker-playground I849568826b1993fd9fb652779e631188af4a319e
Change-Id: I22f95de0bd131f0f0cfd5e2a24e17bed22e5d248
2022-07-12 14:33:30 +00:00
Oliver Smith fa8a74dfde docker-rebuild: build erlang img on x86_64 only
Build debian-bullseye-erlang on x86_64 nodes only, as the image doesn't
get used on nodes with other arches. The image is only used by
gerrit-verifications jobs for osmo_dia2gsum and osmo_gsup, which are
restriected to run on these nodes:

Remove the check and comments for not building for debian 10 x32, as
that node has been replaced with a debian 11 one.

Related: OS#5453
Change-Id: I5636dfc9e06a44b096f9dc55528d677ea2dfc352
2022-07-12 14:33:30 +00:00
Oliver Smith e1540eee9e jobs/coverity.yml: add parameter for git branch
Allow configuring a different git branch before starting the build. This
is especially useful to reproduce a failure that only happens when using
coverity tools during a build.

Change-Id: I5f7c615ff5af4bf1f34ccef62767b5ca71cd2d8d
2022-07-12 14:45:09 +02:00
Oliver Smith ff94d1def5 jobs/coverity: don't cd ~/osmo-ci/coverity
Run the coverity/ from the osmo-ci.git repository that jenkins
checks out at the beginning of the build, instead of using a copy in ~/
that is updated whenever pushing to master. This makes it less
confusing, I configured the job to use a WIP branch and didn't realize
that it still used the code from master.

Change-Id: Ie992d71d2afec6982447b8fe3626c821a9e48d19
2022-07-12 14:44:52 +02:00
Oliver Smith e58c208706 jobs/coverity: modernize
* no need to mention "(jenkins-job-builder)" in the display-name
* set mail to jenkins-notifications ML like most other jobs

Change-Id: I0284dd7354157e057903f53cde149ce652e76550
2022-07-12 14:03:05 +02:00
Oliver Smith 451d687036 Revert "WIP: coverity: Make sure curl is installed, as our coverity upload job needs it"
Revert 3038c19155, which adds a second
osmocom-coverity-deps job that apparently was never deployed (or was
deployed and then deleted). The mentioned curl dependency shouldn't go
here either.

Change-Id: I5d4709c0019ef4bae6ccbd11c28eeb23d75692c1
2022-07-12 14:02:59 +02:00
Oliver Smith c42a20a8f1 coverity: don't build tests for all
The coverity jenkins job has been failing since we tried to build tests
for all projects (and in an earlier attempt, build and run tests for all
projects). Make the coverity job work again by skipping build of tests
for asn1c. I looked into it earlier and the cause wasn't obvious to me,
and it's currently not worth spending time on.

Fix for:
libtool:   error: cannot find the library '../../skeletons/' or unhandled argument '../../skeletons/'

Change-Id: Iba0bfe0e753812058436eaf515391eb1f4b09951
2022-07-12 13:44:06 +02:00
Oliver Smith 56d1a172d6 scripts/repo-install-test: add debian 11, drop 9
Change-Id: I3e154def9051076bd8091b3f027ee76bbb94cb18
2022-07-08 15:35:14 +02:00
Oliver Smith 6135bf1bc4 scripts/OBS latest: fix checkout with dir in path
osmo-python-tests was added to latest just like it was in nightly, by
adding the following two lines to build_osmocom():

  checkout python/osmo-python-tests
  build osmo-python-tests

But the build fails, because the "checkout" function clones into a
"python/osmo-python-tests" directory in latest, and not into a
"osmo-python-tests" directory like in nightly. Make it behave the same
by adjusting the "checkout" function in nightly.

(This code has historically grown and should be refactored imho, but
let's fix this bug first.)

Fix for:
  ====> Building osmo-python-tests
  + cd /home/osmocom-build/jenkins/workspace/
  ./scripts/ 99: cd: can't cd to .../osmo-python-tests

Change-Id: I97074ee953cadac476fe2e4b686689c26bac1ea1
2022-07-08 14:44:43 +02:00
Oliver Smith 6116c0dfe2 scripts/ build osmo-python-tests
Build the source package too, don't just checkout the sources.

Related: OS#5604
Change-Id: I5b7c6b6849edbca44f45041a6e506f10933b5d43
2022-07-07 09:19:32 +00:00
Oliver Smith 2b272e0e83 jobs/registry-update-base-images: drop old images
No need to mirror these images if we don't use them anymore.
Remove debian 8 and 9, and ubuntu zesty. The latter is the codename for
ubuntu 17.04, which is a non-LTS and was unsupported since 2018-01-13.

Related: OS#5601
Change-Id: If1c198038b0fd5faf0763172447962969448940e
2022-07-06 15:20:22 +00:00
Oliver Smith dc00f325c7 ci-docker-rebuild: drop debian-stretch-jenkins
Related: OS#5601
Change-Id: I572ff9624fba25d7d96af79d1229b614be1c11ba
2022-07-06 15:20:22 +00:00
Oliver Smith 095a110506 jobs/repo-install-test: update debian versions
Run against debian 10 and 11, and not against debian 9 anymore.

Related: OS#5601
Change-Id: Iaf7c596e0910ca2ce0fafd2819e79d4659b92514
2022-07-06 15:20:22 +00:00
Oliver Smith 8424c5986f jobs: stretch -> buster
Related: OS#5601
Change-Id: Ieb2bfc426563e48b9b8cd3910efbebdf09b41f46
2022-07-06 15:20:22 +00:00
Oliver Smith 7bc57b7ef1 scripts/osmocom-latest-packages: add osmo-python-tests
Add it in the same place as in

Closes: OS#5604
Change-Id: I187c73f8dff6800abaaeb6126ace13142a8ae20d
2022-07-06 16:08:41 +02:00
Vadim Yanitskiy 2be279a9e0 fixup: coverity: compile tests, but not execute them
In change 82265bc46d [1] I attempted
to fix aaeab8c976 [2], however both
pushd and popd commands are not available in plain shell.

Change-Id: Ib63ded66781c774e16773537c3b943237904b03e
Fixes: [1] I92b648cd1e80399561067eae99426faed1ff1e25
Fixes: [2] Idf0714a59356633021be21acc47f9471d059b7fe
2022-07-04 16:57:33 +07:00
Vadim Yanitskiy 82265bc46d fixup: coverity: compile tests, but not execute them
Change-Id: I92b648cd1e80399561067eae99426faed1ff1e25
Fixes: Idf0714a59356633021be21acc47f9471d059b7fe
2022-07-04 15:42:11 +07:00
Vadim Yanitskiy aaeab8c976 coverity: compile tests, but not execute them
Compiling unit tests would allow Coverity to analyze them too and thus
find more bugs.  The problem is that binaries listed in check_PROGRAMS
are not compiled during 'make all', but only during 'make check'.

One approach [1] is to run 'make check' in do_build(), however it breaks
'Coverity-Upload' job because some unit tests are failing.  This is why
I had to revert bf84ba9f69 together with
ae41942214 that was supposed to fix it.

Even if all unit tests were passing, we still don't want to execute them
because running tests is beyond the scope of 'Coverity-Upload' job.

Another approach is to patch files replacing check_PROGRAMS
with noinst_PROGRAMS, so that the test binaries are compiled during
'make all' but not executed.

Change-Id: Idf0714a59356633021be21acc47f9471d059b7fe
Related: [1] I3f3149a61df96646798c72d0ad31c41b97e6b81f
2022-07-04 14:39:30 +07:00
Vadim Yanitskiy 618f171cb1 Revert "coverity/ run 'make check' in do_build()"
This reverts commit bf84ba9f69.

Change-Id: I722f8cbb1934a4f5ce2969e243f223019f123777
2022-07-04 14:38:45 +07:00
Vadim Yanitskiy 9baa805a74 Revert "coverity/ don't do check for all"
This reverts commit ae41942214.

Change-Id: I5018e0da020f32e3a46a2772e79794e5a3a686ca
2022-07-04 14:38:45 +07:00
Vadim Yanitskiy 4f8a4954b9 lint: simplify find command in exclude_paths_common_asn1c()
The purpose of "cut -d / -f 2-" is to strip off leading "./" in
the paths.  The same can be achieved by doing "-printf '%P\n'".

Matching "*.c" and "*.h" files can be expressed as "-name *.[hc]".

Change-Id: Iae3fc5c8842df6926e6ff16a41be5663f1dedd1b
2022-07-03 04:55:54 +07:00
Vadim Yanitskiy e6ad3a2b2f lint: ignore symlinks (no newline at end of file)
Change-Id: I6db6ebb51cdfd54e8f9c2c5cc7affef60a7dec32
2022-07-03 04:49:13 +07:00