Commit Graph

75 Commits

Author SHA1 Message Date
Tobias Brunner f2d240954a testing: Skip tests with missing files, don't abort the test run
This allows simple test configs in testing/tests/local that are no
actual test cases.
2020-06-23 16:24:18 +02:00
Tobias Brunner c7a74fd3e5 testing: Allow enabling only timestamps without verbose command output
-t enables only the timestamps, -v additionally logs command output
(includes -t).
2018-11-21 14:32:25 +01:00
Tobias Brunner 2132031d0e testing: Show config files of FreeRADIUS 3.0 in test results 2018-11-21 14:32:25 +01:00
Tobias Brunner 2fbe44bef3 testing: Remove TNC@FHH dependencies and scenarios that rely on them
While we could continue to use FreeRADIUS 2.x that branch is officially EOL.
So instead of investing time and effort in updating/migrating the patches to
FreeRADIUS 3.x (the module changed quite significantly as it relies solely on
the naeap library in that release), for a protocol that is superseded anyway,
we just remove these scenarios and the dependencies.  Actually, the
complete rlm_eap_tnc module will be removed with FreeRADIUS 4.0.
2018-11-21 14:32:24 +01:00
Tobias Brunner 5cfd7311d0 testing: Print command output if test fails
This is quite helpful to debug why a pattern didn't match.

As it could produce quite a lot of output if something is not found in a
log file, the complete output is only printed in verbose mode, otherwise,
`head` is used to print the first 10 lines of output.

We only get stdout from SSH, so the stderr redirection is only really
for errors ssh itself produces.
2018-06-14 09:29:26 +02:00
Tobias Brunner 4492c9c670 testing: Ignore IP-in-IP SAs created with IPComp SAs that remain in the kernel
The kernel creates such SAs to handle uncompressed small packets.  They
are implicitly created and deleted with IPComp SAs.  The problem is that
when we delete an IPComp SA only that state is deleted and removed from
the SA lists immediately, the IP-in-IP state is not removed until the IPComp
state is eventually destroyed.  This could take a while if there are still
references to it around.  So the IP-in-IP states will keep getting reported
by ip xfrm state until that happens (we also can't flush or explicitly delete
such kernel-created states).

In kernels before 4.14 this wasn't really a problem but since
ec30d78c14a8 ("xfrm: add xdst pcpu cache") the kernel seems to keep the
references to the last used SAs around a lot longer.

Also, usually a test scenario following an IPComp scenario will create
and use new SAs and thus the cached SAs will disappear before the kernel
state is checked again.  However, if a following scenario uses different
hosts the states might remain, which caused some unrelated scenarios to
fail before adding this fix.
2018-02-01 17:10:19 +01:00
Andreas Steffen 67a97c18ae testing: Enable systemd 2017-11-10 11:49:38 +01:00
Tobias Brunner c11d13c4b9 testing: Add -v option to do-tests to prefix commands with timestamps 2017-08-07 16:55:45 +02:00
Tobias Brunner f0d051f192 testing: Also capture stderr during test cases
The output was not correct otherwise due to the reordering of commands.
2017-08-07 10:44:05 +02:00
Tobias Brunner 87c6247e0d testing: Clearly mark the tests that failed 2017-08-07 10:44:05 +02:00
Tobias Brunner 99cf64e960 testing: Add support for counting matching lines in tests
Specifying an integer instead of YES in evaltest.dat causes the number to get
compared against the actual number of lines matching the pattern.

This may be used to count matching packets or log lines.
2017-08-07 10:44:05 +02:00
Tobias Brunner 49917f0028 testing: Support running multiple tests with * as wildcard (e.g. ikev2/ocsp-*) 2017-07-07 09:23:14 +02:00
Tobias Brunner 3fb68ac211 testing: load-testconfig script loads config from source dir
It now does replace the IPs too. This way it's easier to play around
with a config (otherwise a do-tests run was required to build the
config files in the build dir).
2017-03-02 11:54:39 +01:00
Tobias Brunner 6307a18fe1 testing: Fix totals if post test checks fail 2016-09-20 15:36:14 +02:00
Tobias Brunner d8b2980aa5 testing: Log leaks and fail tests if any are detected 2016-09-20 15:36:14 +02:00
Tobias Brunner ac67aeb100 testing: Add output of iptables-save
This might be helpful to get the complete picture of the installed
rules.  `-c` is currently not used as the counters that are added in
front of every rule make the output quite hard to read and the counters
are already provided in the accompanying `iptables -v -L` output.

Fixes #2111.
2016-09-12 16:15:45 +02:00
Tobias Brunner fa36699bfa testing: List `nat` and `mangle` tables in addition to the `filter` table
This is useful in scenarios that e.g. use NAT and/or marks.

References #2111.
2016-09-12 16:15:14 +02:00
Tobias Brunner 92ccc0b412 testing: Ignore comments (lines starting with #) in pre-/eval-/posttest.dat 2016-09-09 12:19:14 +02:00
Tobias Brunner 379d94eb70 testing: Try to properly abort a test run after CTRL-C
The run is aborted after the current scenario.  Depending on which
command was interrupted it might be necessary to press CTRL-C multiple
times (e.g. if a later command depends on the interrupted one).

This should fix HTML files and get us some proper console output after
the run.
2016-08-30 16:16:42 +02:00
Tobias Brunner bdd7c42fc0 testing: Report number of tests per subdirectory in main index 2016-08-30 15:03:40 +02:00
Tobias Brunner dda06b0439 testing: Mount and serve testresults from the host
This avoids having to copy testresults, makes results of cancelled runs
browsable (runs may actually be followed live) and preserves old results
when rebuilding guest images (e.g. when using the build-strongswan script).
The number of consecutive test runs without any intermittent rebuild of the
guest images is also not limited by the image size anymore.
2016-08-29 18:11:42 +02:00
Tobias Brunner cf6042453f testing: Serve images in testresults via mod_rewrite and not a symlink 2016-08-29 18:11:42 +02:00
Tobias Brunner 63acd803f0 testing: Wait for packets to be processed by tcpdump
Sometimes tcpdump fails to process all packets during the short running
time of a scenario:

0 packets captured
18 packets received by filter
0 packets dropped by kernel

So 18 packets were captured by libpcap but tcpdump did not yet process
and print them.

This tries to use --immediate-mode if supported by tcpdump (the one
currently in jessie or wheezy does not, but the one in jessie-backports
does), which disables the buffering in libpcap.

However, even with immediate mode there are cases where it takes a while
longer for all packets to get processed.  And without it we also need a
workaround (even though the version in wheezy actually works fine).
That's why there now is a loop checking for differences in captured vs.
received packets.  There are actually cases where these numbers are not
equal but we still captured all packets we're interested in, so we abort
after 1s of retrying.  But sometimes it could still happen that packets
we expected got lost somewhere ("packets dropped by kernel" is not
always 0 either).
2016-06-16 14:36:15 +02:00
Tobias Brunner 71424a2f85 testing: Make sure tcpdump is actually terminated before analyzing/collecting logs 2016-06-16 14:01:47 +02:00
Andreas Steffen 9db530493f testing: Change sql scenarios to swanctl 2016-01-03 06:28:48 +01:00
Andreas Steffen 6789d79d46 testing: Added swanctl --list-algs output 2015-12-11 18:26:54 +01:00
Tobias Brunner 74270c8c86 vici: Don't report memory usage via leak-detective
This slowed down the `swanctl --stats` calls in the test scenarios
significantly, with not much added value.
2015-12-11 18:26:53 +01:00
Tobias Brunner e873cb5a28 testing: Add test config to create and remove a directory for DBs stored in ramfs 2015-11-09 15:18:38 +01:00
Tobias Brunner bcad0f761f testing: Report the actual strongSwan and kernel versions 2015-11-09 15:18:37 +01:00
Tobias Brunner d4908c06c1 testing: Report time required for all scenarios on test overview page 2015-11-09 15:18:37 +01:00
Tobias Brunner e22a663129 testing: Don't log anything to the console if auth.log or daemon.log do not exist 2015-11-09 15:18:36 +01:00
Tobias Brunner 12f08e07e1 testing: Simplify fetching of swanctl --list-* output 2015-11-09 15:18:36 +01:00
Tobias Brunner 9086f060d3 testing: Let test scenarios fail if IPsec SAs or policies are not removed
The IKE daemon should delete all installed SAs and policies when
everything works properly, so we fail the test if that's not the case.
2015-08-21 18:27:06 +02:00
Tobias Brunner c91682d1b8 testing: Flush state and policies before every scenario
Similar to conntrack we make sure we are working on a clean slate.
2015-08-21 18:27:06 +02:00
Tobias Brunner 008a9ad12c testing: Don't run do-tests when hosts are not running
running_any is satisfied if at least one host is running.  We could
easily add a running_all() helper to check if all hosts are running if
it turns out that's not strong enough.
2015-08-03 13:34:05 +02:00
Andreas Steffen 450c6e8dd9 testing: Added swanctl --list-authorities output to do-tests 2015-07-22 13:27:08 +02:00
Andreas Steffen db69295d2e tests: Introduced IPV6 flag in tests.conf 2015-07-21 23:17:14 +02:00
Andreas Steffen 6b265c5e5c tests: Introduced SWANCTL flag in test.conf 2015-07-21 23:17:14 +02:00
Andreas Steffen ed42874645 Added swanctl --list-pols and swanctl --stats do scenario log 2014-06-18 13:16:18 +02:00
Andreas Steffen 2721832a45 First swanctl scenario 2014-06-01 21:12:15 +02:00
Andreas Steffen 2382d45b1c Test SWID REST API ins tnc/tnccs-20-pdp scenarios 2014-05-31 21:25:46 +02:00
Tobias Brunner 7a61bf9032 testing: Run 'conntrack -F' before all test scenarios
This prevents failures due to remaining conntrack entries.
2014-04-02 11:55:05 +02:00
Tobias Brunner 053ad34959 testing: Ignore errors when searching for imcv log entries in daemon.log 2013-06-10 18:52:32 +02:00
Martin Willi b5f3c1f785 Define SSHCONF from strongswan testing directory, not TESTDIR
This fixes the use of SSHCONF in the ssh wrapper script before ./do-tests
had a chance to create the required symlinks.
2013-03-26 10:31:29 +01:00
Andreas Steffen 4a3c1cdc2b Store debug output from standalone IMC/IMVs 2013-03-22 16:45:24 +01:00
Tobias Brunner 7699a928f7 Renamed $UMLHOSTS to $VIRTHOSTS 2013-01-17 16:56:02 +01:00
Reto Buerki e3a3013323 Append seconds to TESTDATE
This avoids 'file exists' warnings when running tests multiple times in
one minute.
2013-01-17 16:55:04 +01:00
Reto Buerki 49b1655ae1 Auto-create symlink to testing directory in workdir 2013-01-17 16:55:03 +01:00
Andreas Steffen b27836412b Rename UML to KVM tests 2013-01-17 16:55:02 +01:00
Andreas Steffen 4aa32cc3fe List daemon.log on $RADIUSHOSTS 2013-01-17 16:55:02 +01:00