test: fail tests when programs are missing

Building only a subset of programs is not a very common situation, it is
more likely that some feature was accidentally disabled. For that
reason, fail tests by default unless a program is explicitly permitted
to be missing.

The '-v' test is now dropped from the Travis tests, the sole reason of
adding it was to see which tests got (accidentally) skipped.

Change-Id: I725f4508541d8ed980e17d69fb7aee1ad2875d73
Reviewed-on: https://code.wireshark.org/review/31660
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Tested-by: Petri Dish Buildbot
Reviewed-by: Peter Wu <peter@lekensteyn.nl>

.travis.yml

@@ -127,7 +127,8 @@ script:
   # Invoke ninja (Linux/macOS, --config is ignored) or msbuild (Windows)
   - cmake --build . --config RelWithDebInfo
   - cmake --build . --config RelWithDebInfo --target test-programs
-  - pytest -v
+  - if [ "$PCAP" = "OFF" ]; then export PYTEST_ADDOPTS=--skip-missing-programs=dumpcap,rawshark; fi
+  - pytest
 after_script:
   - if [ -f run/tshark ]; then run/tshark --version; fi
   - if [ -f run/RelWithDebInfo/tshark.exe ]; then run/RelWithDebInfo/tshark.exe --version; fi

docbook/wsdg_src/WSDG_chapter_tests.asciidoc

@@ -64,11 +64,9 @@ Capture tests depend on the permissions of the user running the test
 script. We assume that the test user has capture permissions on Windows
 and macOS and capture tests are enabled by default on those platforms.
 
-If the program or feature dependencies of a test are not satisfied, then
-the test will be skipped. For example, if `dumpcap` has not been built
-or if an old version of Libgcrypt is in use, then some capture or
-decryption tests will be skipped while other tests can still run to
-completion.
+If a feature is unavailable, the test will be skipped. For example, if
+an old version of Libgcrypt is in use, then some decryption tests will
+be skipped while other tests can still run to completion.
 
 [[TestsLayout]]
 ==== Suites, Cases, and Tests
@@ -217,6 +215,19 @@ $ pytest --collect-only
 $ pytest --collect-only -k "dfilter and tvb"
 ----
 
+The test suite will fail tests when programs are missing. When only a
+subset of programs are built or when some programs are disabled, then
+the test suite can be instructed to skip instead of fail tests:
+
+[source,sh]
+----
+# Run tests when libpcap support is disabled (-DENABLE_PCAP=OFF)
+$ pytest --skip-missing-programs dumpcap,rawshark
+
+# Run tests and ignore all tests with missing program dependencies
+$ pytest --skip-missing-programs all
+----
+
 To open a Python debugger (PDB) on failing tests, use the `--pdb` option and
 disable parallelism with the `-n0` option:
 

test/conftest.py

@@ -19,6 +19,9 @@ def pytest_addoption(parser):
         help='Capture interface index or name.'
     )
     parser.addoption('--program-path', help='Path to Wireshark executables.')
+    parser.addoption('--skip-missing-programs',
+        help='Skip tests that lack programs from this list instead of failing'
+             ' them. Use "all" to ignore all missing programs.')
 
 _all_test_groups = None
 

test/fixtures_ws.py

@@ -72,14 +72,20 @@ def program_path(request):
 
 
 @fixtures.fixture(scope='session')
-def program(program_path):
+def program(program_path, request):
+    skip_if_missing = request.config.getoption('--skip-missing-programs',
+                                               default='')
+    skip_if_missing = skip_if_missing.split(',') if skip_if_missing else []
+    dotexe = ''
+    if sys.platform.startswith('win32'):
+        dotexe = '.exe'
+
     def resolver(name):
-        dotexe = ''
-        if sys.platform.startswith('win32'):
-            dotexe = '.exe'
         path = os.path.abspath(os.path.join(program_path, name + dotexe))
         if not os.access(path, os.X_OK):
-            fixtures.skip('Program %s is not available' % (name,))
+            if skip_if_missing == ['all'] or name in skip_if_missing:
+                fixtures.skip('Program %s is not available' % (name,))
+            raise AssertionError('Program %s is not available' % (name,))
         return path
     return resolver
 

test/test.py

@@ -56,6 +56,9 @@ def main():
     cap_group.add_argument('-E', '--disable-capture', action='store_true', help='Disable capture tests')
     cap_group.add_argument('-i', '--capture-interface', help='Capture interface index or name')
     parser.add_argument('-p', '--program-path', default=os.path.curdir, help='Path to Wireshark executables.')
+    parser.add_argument('--skip-missing-programs',
+        help='Skip tests that lack programs from this list instead of failing'
+             ' them. Use "all" to ignore all missing programs.')
     list_group = parser.add_mutually_exclusive_group()
     list_group.add_argument('-l', '--list', action='store_true', help='List tests. One of "all" or a full or partial test name.')
     list_group.add_argument('--list-suites', action='store_true', help='List all suites.')