test: fail tests when programs are missing
Building only a subset of programs is not a very common situation, it is more likely that some feature was accidentally disabled. For that reason, fail tests by default unless a program is explicitly permitted to be missing. The '-v' test is now dropped from the Travis tests, the sole reason of adding it was to see which tests got (accidentally) skipped. Change-Id: I725f4508541d8ed980e17d69fb7aee1ad2875d73 Reviewed-on: https://code.wireshark.org/review/31660 Petri-Dish: Peter Wu <peter@lekensteyn.nl> Tested-by: Petri Dish Buildbot Reviewed-by: Peter Wu <peter@lekensteyn.nl>
This commit is contained in:
parent
70d9bfbf33
commit
0971d20452
|
@ -127,7 +127,8 @@ script:
|
||||||
# Invoke ninja (Linux/macOS, --config is ignored) or msbuild (Windows)
|
# Invoke ninja (Linux/macOS, --config is ignored) or msbuild (Windows)
|
||||||
- cmake --build . --config RelWithDebInfo
|
- cmake --build . --config RelWithDebInfo
|
||||||
- cmake --build . --config RelWithDebInfo --target test-programs
|
- cmake --build . --config RelWithDebInfo --target test-programs
|
||||||
- pytest -v
|
- if [ "$PCAP" = "OFF" ]; then export PYTEST_ADDOPTS=--skip-missing-programs=dumpcap,rawshark; fi
|
||||||
|
- pytest
|
||||||
after_script:
|
after_script:
|
||||||
- if [ -f run/tshark ]; then run/tshark --version; fi
|
- if [ -f run/tshark ]; then run/tshark --version; fi
|
||||||
- if [ -f run/RelWithDebInfo/tshark.exe ]; then run/RelWithDebInfo/tshark.exe --version; fi
|
- if [ -f run/RelWithDebInfo/tshark.exe ]; then run/RelWithDebInfo/tshark.exe --version; fi
|
||||||
|
|
|
@ -64,11 +64,9 @@ Capture tests depend on the permissions of the user running the test
|
||||||
script. We assume that the test user has capture permissions on Windows
|
script. We assume that the test user has capture permissions on Windows
|
||||||
and macOS and capture tests are enabled by default on those platforms.
|
and macOS and capture tests are enabled by default on those platforms.
|
||||||
|
|
||||||
If the program or feature dependencies of a test are not satisfied, then
|
If a feature is unavailable, the test will be skipped. For example, if
|
||||||
the test will be skipped. For example, if `dumpcap` has not been built
|
an old version of Libgcrypt is in use, then some decryption tests will
|
||||||
or if an old version of Libgcrypt is in use, then some capture or
|
be skipped while other tests can still run to completion.
|
||||||
decryption tests will be skipped while other tests can still run to
|
|
||||||
completion.
|
|
||||||
|
|
||||||
[[TestsLayout]]
|
[[TestsLayout]]
|
||||||
==== Suites, Cases, and Tests
|
==== Suites, Cases, and Tests
|
||||||
|
@ -217,6 +215,19 @@ $ pytest --collect-only
|
||||||
$ pytest --collect-only -k "dfilter and tvb"
|
$ pytest --collect-only -k "dfilter and tvb"
|
||||||
----
|
----
|
||||||
|
|
||||||
|
The test suite will fail tests when programs are missing. When only a
|
||||||
|
subset of programs are built or when some programs are disabled, then
|
||||||
|
the test suite can be instructed to skip instead of fail tests:
|
||||||
|
|
||||||
|
[source,sh]
|
||||||
|
----
|
||||||
|
# Run tests when libpcap support is disabled (-DENABLE_PCAP=OFF)
|
||||||
|
$ pytest --skip-missing-programs dumpcap,rawshark
|
||||||
|
|
||||||
|
# Run tests and ignore all tests with missing program dependencies
|
||||||
|
$ pytest --skip-missing-programs all
|
||||||
|
----
|
||||||
|
|
||||||
To open a Python debugger (PDB) on failing tests, use the `--pdb` option and
|
To open a Python debugger (PDB) on failing tests, use the `--pdb` option and
|
||||||
disable parallelism with the `-n0` option:
|
disable parallelism with the `-n0` option:
|
||||||
|
|
||||||
|
|
|
@ -19,6 +19,9 @@ def pytest_addoption(parser):
|
||||||
help='Capture interface index or name.'
|
help='Capture interface index or name.'
|
||||||
)
|
)
|
||||||
parser.addoption('--program-path', help='Path to Wireshark executables.')
|
parser.addoption('--program-path', help='Path to Wireshark executables.')
|
||||||
|
parser.addoption('--skip-missing-programs',
|
||||||
|
help='Skip tests that lack programs from this list instead of failing'
|
||||||
|
' them. Use "all" to ignore all missing programs.')
|
||||||
|
|
||||||
_all_test_groups = None
|
_all_test_groups = None
|
||||||
|
|
||||||
|
|
|
@ -72,14 +72,20 @@ def program_path(request):
|
||||||
|
|
||||||
|
|
||||||
@fixtures.fixture(scope='session')
|
@fixtures.fixture(scope='session')
|
||||||
def program(program_path):
|
def program(program_path, request):
|
||||||
|
skip_if_missing = request.config.getoption('--skip-missing-programs',
|
||||||
|
default='')
|
||||||
|
skip_if_missing = skip_if_missing.split(',') if skip_if_missing else []
|
||||||
|
dotexe = ''
|
||||||
|
if sys.platform.startswith('win32'):
|
||||||
|
dotexe = '.exe'
|
||||||
|
|
||||||
def resolver(name):
|
def resolver(name):
|
||||||
dotexe = ''
|
|
||||||
if sys.platform.startswith('win32'):
|
|
||||||
dotexe = '.exe'
|
|
||||||
path = os.path.abspath(os.path.join(program_path, name + dotexe))
|
path = os.path.abspath(os.path.join(program_path, name + dotexe))
|
||||||
if not os.access(path, os.X_OK):
|
if not os.access(path, os.X_OK):
|
||||||
fixtures.skip('Program %s is not available' % (name,))
|
if skip_if_missing == ['all'] or name in skip_if_missing:
|
||||||
|
fixtures.skip('Program %s is not available' % (name,))
|
||||||
|
raise AssertionError('Program %s is not available' % (name,))
|
||||||
return path
|
return path
|
||||||
return resolver
|
return resolver
|
||||||
|
|
||||||
|
|
|
@ -56,6 +56,9 @@ def main():
|
||||||
cap_group.add_argument('-E', '--disable-capture', action='store_true', help='Disable capture tests')
|
cap_group.add_argument('-E', '--disable-capture', action='store_true', help='Disable capture tests')
|
||||||
cap_group.add_argument('-i', '--capture-interface', help='Capture interface index or name')
|
cap_group.add_argument('-i', '--capture-interface', help='Capture interface index or name')
|
||||||
parser.add_argument('-p', '--program-path', default=os.path.curdir, help='Path to Wireshark executables.')
|
parser.add_argument('-p', '--program-path', default=os.path.curdir, help='Path to Wireshark executables.')
|
||||||
|
parser.add_argument('--skip-missing-programs',
|
||||||
|
help='Skip tests that lack programs from this list instead of failing'
|
||||||
|
' them. Use "all" to ignore all missing programs.')
|
||||||
list_group = parser.add_mutually_exclusive_group()
|
list_group = parser.add_mutually_exclusive_group()
|
||||||
list_group.add_argument('-l', '--list', action='store_true', help='List tests. One of "all" or a full or partial test name.')
|
list_group.add_argument('-l', '--list', action='store_true', help='List tests. One of "all" or a full or partial test name.')
|
||||||
list_group.add_argument('--list-suites', action='store_true', help='List all suites.')
|
list_group.add_argument('--list-suites', action='store_true', help='List all suites.')
|
||||||
|
|
Loading…
Reference in New Issue