Commit Graph

1024 Commits

Author SHA1 Message Date
Vadim Yanitskiy 581a7ea879 coverity: fix prepare_source_Osmocom.sh terminating early
Since [1] was merged this script is executed with '-xe' making it
fail immediately if a command returns a non-zero code.  grep may
return 1 if pattern is not found, so let's suspend it.

Change-Id: I21a01478b64015fd699ed31634bf2fb1dc8f6750
Fixes: [1] Icf6546271a01cd9a3dab1ce0d1bc02807d08ad85
2022-10-04 17:00:01 +07:00
Vadim Yanitskiy 4bb6385a22 scripts: osmo_git_clone_url(): properly handle 'satellite/osmo-gmr'
Coverity job currently fails because it fails to fetch osmo-gmr.git.

Change-Id: I4d52dbf0f81332dc1856944a6a3c7376cb1e0638
Fixes: I17590d0b0b46274e2ceb007e20d7d1dc039f4e8d
2022-10-01 17:42:55 +07:00
Vadim Yanitskiy 6b3b90d55d coverity/prepare_source_Osmocom.sh: execute with '-xe'
Change-Id: Icf6546271a01cd9a3dab1ce0d1bc02807d08ad85
2022-10-01 17:42:39 +07:00
Oliver Smith 7fc58133d6 Replace most remaining git.osmocom.org urls
Replace the legacy git.osmocom.org clone URLs with either gerrit or
gitea in most places. Besides not using legacy urls anymore, the
advantage of using gerrit directly is that we don't have a delay of 10
min to several hours on the repository.

Not replaced:
* ansible/roles/osmocom-workstation: not sure if still used, would need
  a bit of a rewrite to use logic similar to osmo_git_clone_url().

Related: https://osmocom.org/projects/cellular-infrastructure/wiki/Git_infrastructure
Change-Id: I90409c9e7f4db4e1765647021625f9acfccaa78b
2022-09-26 15:07:59 +02:00
Oliver Smith a5521ff99e scripts/common: remove osmo_git_clone_date
This had been added for debugging purposes and isn't used anymore. Also
one less hit for git grep git.osmocom.org.

Change-Id: I10cbb598f0d5d675034ca87568a8b6f1920f2ad3
2022-09-26 14:54:01 +02:00
Oliver Smith 6c7a2076cf coverity: use osmo_git_clone_url
Stop cloning from git.osmocom.org by using the shared function.

Change-Id: I17590d0b0b46274e2ceb007e20d7d1dc039f4e8d
2022-09-26 14:50:47 +02:00
Oliver Smith 204ad54e93 coverity/prepare_source_Osmocom: remove test -d
This was redundant, mkdir -p doesn't error if the directory structure
exists.

Change-Id: Ic6253ecb9bfd5f8b2d744988b8825ef90e2e6d5e
2022-09-26 14:50:47 +02:00
Oliver Smith 3d1bbda3bc osmo_git_clone_url: git.osmocom.org -> gitea
Update the function to not use legacy URLs anymore and allow adding
future gitea URLs more easily. Update the description of the function to
reflect the new motivation for using this.

Change-Id: Ibf53ca2c444fae82af73e417410faa5b16f5de59
2022-09-26 14:41:19 +02:00
Oliver Smith ad447c5379 Revert "scripts/obs: make shallow git clones"
I didn't realize during code review that the git-version-gen script we
use in Osmocom repositories need the git history to generate a proper
version string.

For example (libosmocore):
  1.7.0.38-c3b90.202209240002
With a shallow clone, it is:
  1.7.0.202209250002

Revert the shallow git clone patch to get the proper version strings
again. This reverts commit 115262d841.

Related: https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly_obs.osmocom.org/102/console
Related: https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly_obs.osmocom.org/103/console
Change-Id: I6176e52187d12d264eee82e238051ca65e0c8f4d
2022-09-26 12:44:54 +02:00
Max 115262d841 scripts/obs: make shallow git clones
While generating srcpkg we're not really interested in entire
project history (which can be quite heavy) - let's only get
the bare minimum from git.

Change-Id: I71629c244d2c29728a88871e83b20493341321c8
2022-09-23 21:40:32 +07:00
Oliver Smith d1047b3803 lint: accept BE spelling of 'acknowledgement'
Related: https://gerrit.osmocom.org/c/libosmo-gprs/+/29402
Change-Id: Ia1a08295a7c7ed6a77f0055d66a161423d8f17f0
2022-09-22 13:15:02 +00:00
Neels Hofmeyr 5fca660c58 jobs: enable ttcn3-upf-test
Related: I7a651bb85861de4b2677a731b9f17fa0d790c621 (docker-playground)
Change-Id: I5964979b3db86268f6011df0afbee7131557f405
2022-09-22 02:12:40 +02:00
Max d19d4da914 build_srcpkg.py: add option to use git-review
Having an option to explicitly checkout sources for package from gerrit
using numeric ID comes in handy for testing changes.

Related: OS#4107
Change-Id: I403c2083ef8c3f6f966c5b376fb13ea46d8a7a2b
2022-09-21 14:27:59 +00:00
Max f2efe48110 scripts/obs/*.py: add option to ignore required programs check
Not all the required programs are actually required to build subset of packages.
Add option to ignore the default check.

Change-Id: I8b5eaedab87b586fb1ca4c22746be4dfbb3c2176
2022-09-21 14:27:42 +00:00
Oliver Smith f067b7a4d9 obs: lib.run_cmd: set stdin to subprocess.DEVNULL
Don't pass stdin to the programs, as we expect them to run
non-interactively and also don't show the program's output unless -v is
used or the exit code is not 0.

Change-Id: I7e893101c2a3e7b005659ec72aa44fa932b7ccd9
2022-09-21 14:17:37 +00:00
Oliver Smith 910cf06220 obs: update_obs_project: check if oscrc exists
Check if the oscrc exists, as otherwise the osc command will
interactively prompt for the credentials. If running without verbose
mode (the default), the command output is hidden and so it would just
appear to be hanging without this patch if the config doesn't exist.

Change-Id: I6f140b2c1d171d3019fd259399e6a46da9c6451f
2022-09-21 14:17:37 +00:00
Oliver Smith 34b97c7aa6 obs: don't print verbose output twice on error
When running in verbose mode, it's enough that the output is displayed
while the command is running. Don't print it again on error.

Change-Id: I4fcf2421a6f82bd94ba098b39f58e5d3f18aa8fd
2022-09-21 14:17:37 +00:00
Max 29b9881ff1 scripts/obs/*.py: log name of function calling command
Change-Id: If4ccc0eb44da4ea743f3858b09c762ee6a138f1d
2022-09-21 11:26:01 +07:00
Oliver Smith 4a0b42cc4f lint: ignore MACRO_WITH_FLOW_CONTROL
It seems that we don't care about this one, e.g. here:
https://gerrit.osmocom.org/c/osmo-msc/+/28848

Change-Id: I79da5a426db59031e3b16aecedeaa1498c91e847
2022-09-15 10:24:38 +02:00
Neels Hofmeyr 184cf9a33b Revert "jobs: add ttcn3-hnbgw-test-pfcp"
Instead merging into ttcn3-hnbgw-test/

This reverts commit cbdbd7b204.

Change-Id: I296a33d7db85a90bce328250d75cedc2fa04633f
2022-09-14 18:31:37 +02:00
Neels Hofmeyr cbdbd7b204 jobs: add ttcn3-hnbgw-test-pfcp
Change-Id: I56fdbd2e52bba2eca6c209a2ca89dc7e1a8f1f84
2022-09-14 15:28:12 +00:00
Oliver Smith c578f67bbe jobs/gerrit-lint: fix color output
Instead of having two "wrappers:" sections where the second overwrites
the first one, combine them into one.

Fixes: ffbf40c ("jobs/gerrit-lint: add ssh credentials")
Change-Id: Iaa0603dab4aee4e99f196eff862d76e4e8b67af9
2022-09-12 11:50:13 +02:00
Alexander Couzens f8710473d1 lint: annotate lines in gerrit
Use robot comments to add line based comments for the lint output.
Add checkpatch_json from coreboot 9cae17d0 to parse the checkpatch
output and convert it into a gerrit parsable format.

Example:
https://gerrit.osmocom.org/c/osmo-bsc-nat/+/29294

Co-Authored-By: Oliver Smith <osmith@sysmocom.de>
Change-Id: I1a48ddb976e0f53bfc0552d0be11e42ba68d9e49
2022-09-08 17:10:51 +02:00
Oliver Smith ffbf40c791 jobs/gerrit-lint: add ssh credentials
Prepare for having jenkins post gerrit review comments.

Change-Id: I27477c6c448a4b06ea49b020b6cfa534445cf31c
2022-09-08 16:43:15 +02:00
Oliver Smith d708401440 jobs/gerrit-lint: add osmo-ci branch parameter
Extend the jobs to clone the trusted osmo-ci.git repository from which
we execute the linter next to the untrusted code from gerrit, and add a
parameter for the osmo-ci branch.

Change-Id: Ic4134e6755014b3e5fbf6d3225e49c0ca5b27f9b
2022-09-08 16:43:13 +02:00
Oliver Smith 719a62ec37 jobs/gerrit-lint: hardcode shell command
Don't allow overriding cmd per repository anymore. This was used in the
gerrit-osmo-ci-lint job to run the linter from the checked out
osmo-ci.git repository (the one with the untrusted code submitted to
gerrit).

This would become problematic with a future patch that will attach the
ssh agent to the lint jobs (so jenkins can leave review comments over at
gerrit).

Change-Id: Iad23da08898205fafd1ae36256c8f7b6d7506d40
2022-09-08 16:26:59 +02:00
Oliver Smith a1027bd25c jobs/update-osmo-ci-on-slaves: run every day
Make sure scripts/osmo-ci-docker-rebuild.sh runs at least once a day on
the jenkins nodes, so the docker images are less likely to expire due to
being unused. This was the reason for master-osmo-opencm3-projects and
gerrit-osmo_dia2gsup failing.

Related: https://osmocom.org/projects/osmocom-servers/wiki/Docker_cache_clean_up
Change-Id: If5c4b3cb87100ab6c44d717a17986a4f0c6570fe
2022-08-30 14:41:24 +02:00
Oliver Smith 317e429f9e jobs/master-builds: use jenkins-notifications ML
I didn't receive mail notifications about failing
master-osmo-opencm3-projects jobs. It seems they are not getting
through on the gerrit-log mailing list, searching for jenkins gives no
results:
https://lists.osmocom.org/hyperkitty/search?mlist=gerrit-log%40lists.osmocom.org&q=jenkins

And it seems to be the wrong ML anyway, so change it to
jenkins-notifications.

Change-Id: I15a747411d928055c71603cd4475289a77c8e749
2022-08-30 14:39:01 +02:00
Oliver Smith f649cc024e repo-install-test: fix nftables errors on centos8
Don't attempt to install nftables-devel or python3-nftables in centos8
in the repo-install-test. What happens is, that dnf tries to install the
versions from the centos repositories, which conflict with the ones from
the OBS repositories. This fixes the failing
Osmocom-repo-install-centos8 jenkins job.

Change-Id: I57670069cc09a5493667971997f0245e1d6a7259
2022-08-26 12:22:59 +02:00
Neels Hofmeyr cf0d92ff21 obs/lib/config.py: add osmo-upf
Depends: libosmo-pfcp 0.1.2 I6ae6cf59e769214e11447107316d38fe5fad583d
Depends: osmo-upf 0.1.1 I1d73255f8cc30feabd3c251bdc3af1682ed4694a
Change-Id: Ia65508eaf241b799b5d4b1b81eebdeaa40f4381e
2022-08-24 16:58:42 +02:00
Oliver Smith 9d9a8661d6 scripts/osmo-deps.sh: use osmo_git_clone_url
Instead of hardcoding https://git.osmocom.org for all clones, use
the osmo_git_clone_url function. This clones via https from gerrit where
possible, it should fix the many errors we are currently seeing from
the master-openbsc job:

  + osmo-deps.sh libosmo-sccp
  + project=libosmo-sccp
  + branch=master
  + git branch -a
  + grep -c remotes/origin/master$
  + [ x1 != x0 ]
  + branch=origin/master
  + test -d libosmo-sccp
  + git clone https://git.osmocom.org/libosmo-sccp libosmo-sccp
  Cloning into 'libosmo-sccp'...
  error: HTTP/2 stream 0 was closed cleanly, but before getting  all response header fields, treated as error (curl_result = 92, http_code = 0, sha1 = d23e38020fdbb685570145acd3a35e22a5a91344)
  error: HTTP/2 stream 0 was closed cleanly, but before getting  all response header fields, treated as error (curl_result = 92, http_code = 0, sha1 = 68b450098714e3015a65b6628a7e61ac09dd4d47)
  error: Unable to find 68b450098714e3015a65b6628a7e61ac09dd4d47 under https://git.osmocom.org/libosmo-sccp
  Cannot obtain needed tree 68b450098714e3015a65b6628a7e61ac09dd4d47
  while processing commit d23e38020fdbb685570145acd3a35e22a5a91344.
  error: fetch failed.

Related: https://lists.osmocom.org/hyperkitty/list/openbsc@lists.osmocom.org/thread/RHMXTPTKU2SPGCVXF55RFJQ5GYBZ46CO/
Change-Id: I700d608ff74eca3981ed41f04ee9ced9629436aa
2022-08-22 14:07:59 +02:00
Oliver Smith 945ec9a0b3 obs: Cosmetic: verify new projects on OBS first
Related: OS#5654
Change-Id: I4a2ebf65de09e08cf3883ae4de9e7e8c952c2182
2022-08-22 12:52:07 +02:00
Oliver Smith 9d6e43a7c7 obs: disable osmo-upf
Deb and rpm packages don't build for osmo-upf so disable it again until
it is fixed.

Related: OS#5655, OS#5654
Change-Id: Idf18159b786c51154fda9b40758b08a8656fc215
2022-08-22 12:42:16 +02:00
Oliver Smith 4f766e6028 repo-install-test: adjust for obs.osmocom.org
Related: OS#5557
Change-Id: Ib53df6c40ae92634e6d93389dcb8a21c48dfb63d
2022-08-19 14:09:52 +00:00
Oliver Smith 4cb6289b51 repo-install-test: don't use next repository
For the conflict test, don't attempt to install packages from the next
feed for centos8. This was useful when we didn't have binary packages
for centos8 for latest yet, but we have them now. The next feed existed
temporarily and isn't updated anymore for a long time. We did not add it
to obs.osmocom.org.

This is in preparation for adjusting repo-install-test to use packages
from obs.osmocom.org, where we don't have a next feed.

Change-Id: I73a6c051c423eb30ddeab1688fc78fe959e60cf4
2022-08-19 14:09:25 +00:00
Oliver Smith 77a43042c0 jobs/ttcn3: use new packages for 20YYqX testsuite
Drop _MIRROR variables, as both the testsuite packages and the packages
to test are now on downloads.osmocom.org. Previously these variables
were necessary as the testsuite packages were only on
download.opensuse.org (before we set up obs.osmocom.org, which publishes
packages to downloads.osmocom.org).

Change-Id: I34050da1e2444cf684ff8bd07673e3268b4dd497
2022-08-19 14:09:03 +00:00
Oliver Smith 8c9540dd3d jobs/ttcn3: drop obs-mirror workaround for centos8
Get the centos8 packages directly from the package repository again
(https://downloads.osmocom.org/packages/) instead of getting them from
the delayed obs-mirror dir (https://downloads.osmocom.org/obs-mirror/).
Use the packages generated from obs.osmocom.org for the testsuite too.

This reverts 526f1e12d3 and
I8201f34a0f9747eea78c358a626cc3981d7407c4 which set this up to work
around sporadically missing metadata files from download.opensuse.org.

Change-Id: Ic7920ad71a0cb6362883f8b9ed3dbc9bbb992988
2022-08-19 14:08:39 +00:00
Oliver Smith a0dd1d923d jobs/master-builds: clone from gerrit.osmocom.org
Work around two issues with cloning from https://git.osmocom.org:
* cloning via https from git.osmocom.org is unreliable, leads to lots of
  mails of failed builds
* the cgit mirror of libosmo-gprs.git is out of sync

Related: https://lists.osmocom.org/hyperkitty/list/openbsc@lists.osmocom.org/message/YVYMAC423OJ3FNP52YAX55RDUQXV6FZ2/
Change-Id: I2f345be55f5c7eb42655dc346776ad8891af126d
2022-08-19 14:08:28 +00:00
Oliver Smith 65cc10c602 obs: meaningful error if debian/changelog is empty
If there's an empty debian/changelog, packaging needs to be fixed in the
git repository of the project. Print a meaningful error instead of:

    File "/obs/lib/srcpkg.py", line 68, in get_version_for_feed
      if ":" in ret:
  TypeError: argument of type 'NoneType' is not iterable

Change-Id: I294d844ccccfa12599b6ba1def087e4a32d289e7
2022-08-19 14:03:10 +00:00
Oliver Smith a8536d8847 obs: prepend origin/ to branch selected with -b
Doing this makes more sense and is less prone to errors.

Change-Id: I2fa9c457e0e17cdd1b5946dc27ae232ec13bfa25
2022-08-18 13:18:25 +00:00
Oliver Smith fb1f39e0c9 obs: make git fetch the default
As discussed with Neels, it makes more sense to fetch git repos by
default.

Change-Id: Ib8adfbcf59d66b26f322ad22b2df1ccbeaf15af7
2022-08-18 13:18:25 +00:00
Oliver Smith 3a41a4c1b8 obs: checkout_for_feed + fetch: fix for 2022q1 etc
As Neels pointed out, with the current code the 2022q1 and similar feeds
do not get checked out properly if:
* the repository was not freshly cloned (like jenkins would do)
* the repository was fetched (--git-fetch)

Instead of the last revision of that branch from origin, we get the
revision that was previously checked out on the branch. Fix this by
prepending origin/.

Change-Id: I2e265fc84abd4596c386bb56cbabb2af121dc7ca
2022-08-18 13:18:25 +00:00
Vadim Yanitskiy dbd1e02ca3 obs: add libosmo-gprs
Change-Id: I96f48cfc328f61bcbee98c4675896cc4c4e9a48f
2022-08-18 09:32:39 +00:00
Oliver Smith 43d5155b2e osmocom-list-commits: use proper git clone urls
Fix the Osmocom-list-commits jenkins job that is currently failing with:
  fatal: repository 'https://git.osmocom.org/osmo-upf/' not found

Change-Id: I2acb9880be57411f11805a2195076b514156aaf3
2022-08-18 08:27:52 +00:00
Neels Hofmeyr 7057a9f55d obs: tweak logging output
Change-Id: I203cb7f0aa21b59b06973e8f329cffdc64aa8d47
2022-08-17 15:32:07 +02:00
Neels Hofmeyr 39cde7f098 obs: add README
Explain how to use the obs/ scripts to test a private git branch.

Change-Id: I4d1303e0c04e827200b48a9fe4aea3680c9c9f84
2022-08-17 15:32:07 +02:00
Neels Hofmeyr 60fe5c86b5 obs: expand help for --feed
Change-Id: Iabd165e4f7a8c17b30f01ad27abd1a878233dabf
2022-08-17 15:32:07 +02:00
Neels Hofmeyr 872dbbd58b obs: add explicit --git-branch arg
Change-Id: I612c18e533542f442e0571927fee8b8be9fb8c2b
2022-08-17 15:32:07 +02:00
Oliver Smith 71bafe74d3 jobs/master-builds: archive core, testsuite.log
In case one of the master-buils is segfaulting or has an error in the
testsuite, collect the relevant files and store them as artifacts.

It was asked in the related issue to do this only on error, however
while there's an option to archive only artifacts when a build is
successful, there is not an option to do it only on failure. So just do
it always, the testsuite.log file is typically just a few kb in size.
core files should only get generated on error.

Closes: OS#5642
Change-Id: I91d4b995b16c8e028a5dff19b0e0f09c01d8c8c6
2022-08-17 08:10:12 +00:00
Neels Hofmeyr c90ee51321 obs: add libosmo-pfcp, osmo-upf
Change-Id: I7192f4c9e77035cebb147b8186296bcd741d419b
2022-08-16 23:14:57 +02:00