forked from osmocom/wireshark
383 lines
13 KiB
Plaintext
383 lines
13 KiB
Plaintext
// WSDG Chapter Tests
|
|
|
|
[#ChapterTests]
|
|
== Wireshark Tests
|
|
|
|
The Wireshark sources include a collection of Python scripts that test
|
|
the features of Wireshark, TShark, Dumpcap, and other programs that
|
|
accompany Wireshark. These are located in the `test` directory of the
|
|
Wireshark source tree.
|
|
|
|
The command line options of Wireshark and its companion command line
|
|
tools are numerous. These tests help to ensure that we don't introduce
|
|
bugs as Wireshark grows and evolves.
|
|
|
|
[#TestsQuickStart]
|
|
=== Quick Start
|
|
|
|
The recommended steps to prepare for and to run tests:
|
|
|
|
* Install two Python packages, pytest: `pip install pytest pytest-xdist`
|
|
* Build programs (“wireshark”, “tshark”, etc.): `ninja`
|
|
* Build additional programs for the “unittests” suite: `ninja test-programs`
|
|
* Run tests in the build directory: `pytest`
|
|
|
|
Replace `ninja test-programs` by `make test-programs` as needed.
|
|
|
|
The test suite will attempt to test as much as possible and skip tests
|
|
when its dependencies are not satisfied. For example, packet capture
|
|
tests require a Loopback interface and capture privileges. To avoid
|
|
capture tests, pass the `--disable-capture` option.
|
|
|
|
List available tests with `pytest --collectonly`. Enable verbose output
|
|
with `pytest --verbose`. For more details, see <<ChTestsRunPytest>>.
|
|
|
|
If for whatever reason `pytest` is too old or unavailable, you could use
|
|
a more limited test runner, `test/test.py`. Use `test/test.py --help` to
|
|
see all options. For more details, see <<ChTestsRun>>.
|
|
|
|
CMake currently runs `test/test.py` when the “test” target is built.
|
|
|
|
[#ChTestsStructure]
|
|
=== Test suite structure
|
|
|
|
The following sections describes how the test suite is organized.
|
|
|
|
[#TestCoverage]
|
|
==== Test Coverage And Availability
|
|
|
|
The testing framework can run programs and check their stdout, stderr,
|
|
and exit codes. It cannot interact with the Wireshark UI. Tests cover
|
|
capture, command line options, decryption, file format support and
|
|
conversion, Lua scripting, and other functionality.
|
|
|
|
Available tests depend on the libraries with which Wireshark was built.
|
|
For example, some decryption tests depend on a minimum version of
|
|
Libgcrypt and Lua tests depend on Lua.
|
|
|
|
Capture tests depend on the permissions of the user running the test
|
|
script. We assume that the test user has capture permissions on Windows
|
|
and macOS and capture tests are enabled by default on those platforms.
|
|
|
|
If a feature is unavailable, the test will be skipped. For example, if
|
|
an old version of Libgcrypt is in use, then some decryption tests will
|
|
be skipped while other tests can still run to completion.
|
|
|
|
[#TestsLayout]
|
|
==== Suites, Cases, and Tests
|
|
|
|
The `test/test.py` script uses Python's “unittest” module. Our tests are
|
|
patterned after it, and individual tests are organized according to
|
|
suites, cases, and individual tests. Suites correspond to Python modules
|
|
that match the pattern “suite_*.py”. Cases correspond to one or more
|
|
classes in each module, and case class methods matching the pattern
|
|
”test_*” correspond to individual tests. For example, the invalid
|
|
capture filter test in the TShark capture command line options test case
|
|
in the command line options suite has the ID
|
|
“suite_clopts.case_tshark_capture_clopts.test_tshark_invalid_capfilter”.
|
|
|
|
[#TestsPytest]
|
|
==== pytest fixtures
|
|
|
|
A test has typically additional dependencies, like the path to an
|
|
executable, the path to a capture file, a configuration directory, the
|
|
availability of an optional library, and so on. The Python unittest
|
|
library is quite limited in expressing test dependencies, these are
|
|
typically specified on the class instance itself (`self`) or via globals.
|
|
|
|
https://pytest.org/[pytest] is a better test framework which has full
|
|
parallelization support (test-level instead of just suite-level),
|
|
provides nicer test reports, and allows
|
|
https://docs.pytest.org/en/latest/fixture.html[modular fixtures].
|
|
Ideally the test suite should fully switch to pytest, but in meantime a
|
|
compatibility layer is provided via the “fixtures” module.
|
|
|
|
A fixture is a function decorated with `@fixtures.fixture` and can
|
|
either call `fixtures.skip("reason")` to skip tests that depend on the
|
|
fixture, or return/yield a value.
|
|
Test functions (and other fixture functions) can receive the fixture
|
|
value by using the name of the fixture function as function parameters.
|
|
Common fixtures are available in `fixtures_ws.py` and includes
|
|
`cmd_tshark` for the path to the `tshark` executable and `capture_file`
|
|
for a factory function that produces the path to a capture file.
|
|
|
|
Each unittest test case must be decorated with
|
|
`@fixtures.uses_fixtures` to ensure that unittest test classes can
|
|
actually request fixture dependencies.
|
|
|
|
[#ChTestsRun]
|
|
=== Listing And Running Tests
|
|
|
|
Tests can be run via the `test/test.py` Python script. To run all tests,
|
|
either run `test/test.py` in the directory that contains the Wireshark
|
|
executables (`wireshark`, `tshark`, etc.), or pass the executable
|
|
path via the `-p` flag:
|
|
|
|
[source,sh]
|
|
----
|
|
$ python3 test/test.py -p /path/to/wireshark-build/run
|
|
----
|
|
|
|
You can list tests by passing one or more complete or partial names to
|
|
`test/test.py`. The `-l` flag lists tests. By default all tests are shown.
|
|
|
|
[source,sh]
|
|
----
|
|
# List all tests
|
|
$ python3 test/test.py -l
|
|
$ python3 test/test.py -l all
|
|
$ python3 test/test.py --list
|
|
$ python3 test/test.py --list all
|
|
|
|
# List only tests containing "dumpcap"
|
|
$ python3 test/test.py -l dumpcap
|
|
|
|
# List all suites
|
|
$ python3 test/test.py --list-suites
|
|
|
|
# List all suites and cases
|
|
$ python3 test/test.py --list-cases
|
|
----
|
|
|
|
If one of the listing flags is not present, tests are run. If no names or `all` is supplied,
|
|
all tests are run. Otherwise tests that match are run.
|
|
|
|
[source,sh]
|
|
----
|
|
# Run all tests
|
|
$ python3 test/test.py
|
|
$ python3 test/test.py all
|
|
|
|
# Only run tests containing "dumpcap"
|
|
$ python3 test/test.py dumpcap
|
|
|
|
# Run the "clopts" suite
|
|
$ python3 test/test.py suite_clopts
|
|
----
|
|
|
|
Run `python3 test/test.py --help` for all available options.
|
|
|
|
[#ChTestsRunPytest]
|
|
=== Listing And Running Tests (pytest)
|
|
|
|
Tests can also be run with https://pytest.org/[pytest]. Advantages include finer
|
|
test selection, full parallelism, nicer test execution summaries, better output
|
|
in case of failures (containing the contents of variables) and the ability to
|
|
open the PDB debugger on failing tests.
|
|
|
|
To get started, install pytest 3.0 or newer and
|
|
https://pypi.org/project/pytest-xdist/[pytest-xdist]:
|
|
|
|
[source,sh]
|
|
----
|
|
# Install required packages on Ubuntu 18.04 or Debian jessie-backports
|
|
$ sudo apt install python3-pytest python3-pytest-xdist
|
|
|
|
# Install required packages on other systems
|
|
$ pip install pytest pytest-xdist
|
|
----
|
|
|
|
Run `pytest` in the Wireshark build directory, Wireshark binaries are assumed to
|
|
be present in the `run` subdirectory (or `run\RelWithDebInfo` on Windows).
|
|
|
|
[source,sh]
|
|
----
|
|
# Run all tests
|
|
$ cd /path/to/wireshark/build
|
|
$ pytest
|
|
|
|
# Run all except capture tests
|
|
$ pytest --disable-capture
|
|
|
|
# Run all tests with "decryption" in its name
|
|
$ pytest -k decryption
|
|
|
|
# Run all tests with an explicit path to the Wireshark executables
|
|
$ pytest --program-path /path/to/wireshark/build/run
|
|
----
|
|
|
|
To list tests without actually executing them, use the `--collect-only` option:
|
|
|
|
[source,sh]
|
|
----
|
|
# List all tests
|
|
$ pytest --collect-only
|
|
|
|
# List only tests containing both "dfilter" and "tvb"
|
|
$ pytest --collect-only -k "dfilter and tvb"
|
|
----
|
|
|
|
The test suite will fail tests when programs are missing. When only a
|
|
subset of programs are built or when some programs are disabled, then
|
|
the test suite can be instructed to skip instead of fail tests:
|
|
|
|
[source,sh]
|
|
----
|
|
# Run tests when libpcap support is disabled (-DENABLE_PCAP=OFF)
|
|
$ pytest --skip-missing-programs dumpcap,rawshark
|
|
|
|
# Run tests and ignore all tests with missing program dependencies
|
|
$ pytest --skip-missing-programs all
|
|
----
|
|
|
|
To open a Python debugger (PDB) on failing tests, use the `--pdb` option and
|
|
disable parallelism with the `-n0` option:
|
|
|
|
[source,sh]
|
|
----
|
|
# Run decryption tests sequentially and open a debugger on failing tests
|
|
$ pytest -n0 --pdb -k decryption
|
|
----
|
|
|
|
Note that with the option `--pdb`, stray processes are not killed on
|
|
test failures since the `SubprocessTestCase.tearDown` method is not
|
|
executed. This limitation might be addressed in the future.
|
|
|
|
[#ChTestsDevelop]
|
|
=== Adding Or Modifying Built-In Tests
|
|
|
|
Tests must be in a Python module whose name matches “suite_*.py”. The
|
|
module must contain one or more subclasses of “SubprocessTestCase” or
|
|
“unittest.TestCase”. “SubprocessTestCase” is recommended since it
|
|
contains several convenience methods for running processes, normalizing
|
|
and checking output, and displaying error information. Each test case
|
|
method whose name starts with “test_” constitutes an individual test.
|
|
|
|
Success or failure conditions can be signalled using the
|
|
“unittest.assertXXX()” or “subprocesstest.assertXXX()” methods.
|
|
|
|
Test dependencies (such as programs, directories, or the environment
|
|
variables) are injected through method parameters. Commonly used
|
|
fixtures include `cmd_tshark` and `capture_file`. See also
|
|
<<TestsPytest>>.
|
|
|
|
The “subprocesstest” class contains the following methods for running
|
|
processes. Stdout and stderr is written to “<test id>.log”:
|
|
|
|
startProcess:: Start a process without waiting for it to finish.
|
|
runProcess:: Start a process and wait for it to finish.
|
|
assertRun:: Start a process, wait for it to finish, and check its exit code.
|
|
|
|
All of the current tests run one or more of Wireshark's suite of
|
|
executables and either check their return code or their output. A
|
|
simple example is “suite_clopts.case_basic_clopts.test_existing_file”,
|
|
which reads a capture file using TShark and checks its exit code.
|
|
|
|
[source,python]
|
|
----
|
|
import subprocesstest
|
|
import fixtures
|
|
|
|
@fixtures.mark_usefixtures('test_env')
|
|
@fixtures.uses_fixtures
|
|
class case_basic_clopts(subprocesstest.SubprocessTestCase):
|
|
def test_existing_file(self, cmd_tshark, capture_file):
|
|
self.assertRun((cmd_tshark, '-r', capture_file('dhcp.pcap')))
|
|
----
|
|
|
|
Program output is decoded as UTF-8 and CRLF sequences ({backslash}r{backslash}n) are converted to LFs ({backslash}n).
|
|
Output can be checked using `SubprocessTestCase.grepOutput`, `SubprocessTestCase.countOutput` or other `unittest.assert*` methods:
|
|
|
|
[source,python]
|
|
----
|
|
import subprocesstest
|
|
import fixtures
|
|
|
|
@fixtures.mark_usefixtures('test_env')
|
|
@fixtures.uses_fixtures
|
|
class case_decrypt_80211(subprocesstest.SubprocessTestCase):
|
|
def test_80211_wpa_psk(self, cmd_tshark, capture_file):
|
|
tshark_proc = self.assertRun((cmd_tshark,
|
|
'-o', 'wlan.enable_decryption: TRUE',
|
|
'-Tfields',
|
|
'-e', 'http.request.uri',
|
|
'-r', capture_file('wpa-Induction.pcap.gz'),
|
|
'-Y', 'http',
|
|
))
|
|
self.assertIn('favicon.ico', tshark_proc.stdout_str)
|
|
----
|
|
|
|
Tests can be run in parallel. This means that any files you create must
|
|
be unique for each test. “subprocesstest.filename_from_id” can be used
|
|
to generate a filename based on the current test name. It also ensures
|
|
that the file will be automatically removed after the test has run.
|
|
|
|
[#ChTestsExternal]
|
|
=== Adding Or Modifying External Tests
|
|
|
|
You can test the dissection of files outside the Wireshark source code repository by using the external test generator, which creates tests using a JSON configuration file.
|
|
The file must have the following format:
|
|
|
|
[source]
|
|
----
|
|
{
|
|
"case_name": "<test case name>",
|
|
"tests": [
|
|
{
|
|
"test_name": "<test name>",
|
|
"tshark_args": [ <tshark argument array> ],
|
|
"requirements": [ <one or more requirements> ]
|
|
}
|
|
]
|
|
}
|
|
----
|
|
|
|
`tshark_args` elements can use `${case_dir}` to specify the path to the JSON configuration file.
|
|
`requirements` can be one or more of
|
|
|
|
`[ "count", "<pattern>", <count> ]`::
|
|
Require `count` occurrences of `pattern` in the dissection output.
|
|
Equivalent to the built-in Python `assertEqual(countOutput('<pattern'), <count>)`
|
|
|
|
`[ "grep", "<pattern>" ]`::
|
|
Dissection output must contain `pattern`.
|
|
Equivalent to `assertTrue(grepOutput('<pattern>'))`.
|
|
|
|
`[ "!grep", "<pattern>" ]`::
|
|
Dissection output must _not_ contain `pattern`.
|
|
Equivalent to `assertFalse(grepOutput('<pattern>'))`.
|
|
|
|
`[ "in", "<string>", <line> ]`::
|
|
Zero-indexed line `line` of the dissection output must contain `string`.
|
|
Equivalent to `assertIn('<pattern>', lines[<line>])`.
|
|
|
|
`[ "!in", "<string>", <line> ]`::
|
|
Zero-indexed line `line` of the dissection output must _not_ contain `string`.
|
|
Equivalent to `assertNotIn('<pattern>', lines[<line>])`.
|
|
|
|
Patterns can be any valid Python regular expression.
|
|
|
|
The example below defines a single test case, named “external_example”.
|
|
The case has a single test named “dns”, which runs TShark on `tests/dns-1/dns.pcapng`, relative to the JSON configuration file.
|
|
|
|
[source,json]
|
|
----
|
|
{
|
|
"case_name": "external_example",
|
|
"tests": [
|
|
{
|
|
"test_name": "dns",
|
|
"tshark_args": [ "-r", "${case_dir}/tests/dns-1/dns.pcapng",
|
|
"-Y", "dns", "-T", "fields", "-e", "dns.qry.name"
|
|
],
|
|
"requirements": [
|
|
[ "count", "in.m.yahoo.com", 1 ],
|
|
[ "grep", "in.m.yahoo.com" ],
|
|
[ "!grep", "in.m.notyahoo.com" ],
|
|
[ "in", "in.m.yahoo.com", 0 ],
|
|
[ "!in", "in.m.notyahoo.com", 0 ]
|
|
]
|
|
}
|
|
]
|
|
}
|
|
----
|
|
|
|
You can specify external tests using the `test.py --add-external-test`.
|
|
For example, if the JSON file above is named `wireshark-tests.json` you can list its test by running the following:
|
|
|
|
[source,sh]
|
|
----
|
|
$ ./test/test.py -p ./build/run --add-external-test /path/to/wireshark-tests.json --list external
|
|
suite_external.case_external_example.test_dns
|
|
----
|