wireshark/docbook/wsdg_src/WSDG_chapter_capture.asciidoc

:toc: macro

// WSDG Chapter Capture

[[ChapterCapture]]

== Packet capturing

toc::[]

****
This chapter needs to be reviewed and extended.
****

[[ChCaptureAddLibpcap]]

=== How to add a new capture type to libpcap

The following is an updated excerpt from a developer mailing list mail about
adding ISO 9141 and 14230 (simple serial line card diagnostics) to Wireshark:

For libpcap, the first thing you’d need to do would be to get +{dlt-glob}+ values
for all the link-layer protocols you’d need. If ISO 9141 and 14230 use the same
link-layer protocol, they might be able to share a +{dlt-glob}+ value, unless the
only way to know what protocols are running above the link layer is to know
which link-layer protocol is being used, in which case you might want separate
+{dlt-glob}+ values.

For the rest of the libpcap discussion, I'll assume you're working with libpcap
1.0 or later and that this is on a UN*X platform. You probably don't want to
work with a version older than 1.0, even if whatever OS you're using happens to
include libpcap - older versions are not as friendly towards adding support for
devices other than standard network interfaces.

Then you’d probably add to the `pcap_open_live()` routine, for whatever
platform or platforms this code should work, something such as a check
for device names that look like serial port names and, if the check
succeeds, a call to a routine to open the serial port.

See, for example, the `#ifdef HAVE_DAG_API` code in _pcap-linux.c_ and
_pcap-bpf.c_.

The serial port open routine would open the serial port device, set the baud
rate and do anything else needed to open the device. It’d allocate a `pcap_t`,
set its `fd` member to the file descriptor for the serial device, set the
`snapshot` member to the argument passed to the open routine, set the `linktype`
member to one of the +{dlt-glob}+ values, and set the `selectable_fd` member to
the same value as the `fd` member. It should also set the `dlt_count` member to
the number of +{dlt-glob}+ values to support, and allocate an array of
`dlt_count` `u_int`s, assign it to the `dlt_list` member, and fill in that list
with all the +{dlt-glob}+ values.

You’d then set the various `_*_op` fields to routines to handle the operations in
question. `read_op` is the routine that’d read packets from the device. `inject_op`
would be for sending packets; if you don't care about that, you’d set it to a
routine that returns an error indication. `setfilter_op` can probably just be set
to `install_bpf_program`. `set_datalink` would just set the `linktype` member to the
specified value if it’s one of the values for OBD, otherwise it should return an
error. `getnonblock_op` can probably be set to `pcap_getnonblock_fd`. `setnonblock_op`
can probably be set to `pcap_setnonblock_fd`. `stats_op` would be set to a routine
that reports statistics. `close_op` can probably be set to `pcap_close_common`.

If there’s more than one +{dlt-glob}+ value, you definitely want a `set_datalink`
routine so that the user can select the appropriate link-layer type.

For Wireshark, you’d add support for those +{dlt-glob}+ values to
_wiretap/libpcap.c_, which might mean adding one or more _WTAP_ENCAP_ types to
_wtap.h_ and to the `encap_table[]` table in _wiretap/wtap.c_. You’d then
have to write a dissector or dissectors for the link-layer protocols or
protocols and have them register themselves with the `wtap_encap` dissector
table, with the appropriate _WTAP_ENCAP_ values by calling
`dissector_add_uint()`.

[[ChCaptureExtcap]]

=== Extcap: Developer Guide

The extcap interface is a versatile plugin interface that allows external binaries
to act as capture interfaces directly in Wireshark. It is used in scenarios, where
the source of the capture is not a traditional capture model (live capture from an
interface, from a pipe, from a file, etc). The typical example is connecting esoteric
hardware of some kind to the main Wireshark app.

Without extcap, a capture can always be achieved by directly writing to a capture file:

.Bash example for traditional capture with a capture file.
[source,bash]
----
$ the-esoteric-binary --the-strange-flag --interface=stream1 --file dumpfile.pcap &
$ wireshark dumpfile.pcap
----

but the extcap interface allows for such a connection to be easily established and
configured using the Wireshark GUI.

The extcap subsystem is made of multiple extcap binaries that are automatically
called by the GUI in a row. In the following chapters we will refer to them as
"the extcaps".

Extcaps may be any binary or script within the extcap directory. Please note, that
scripts need to be executable without prefacing a script interpreter before the call.

IMPORTANT: *Windows Users* Because of restrictions directly calling the script may not always work.
In such a case, a batch file may be provided, which then in turn executes the script.
Please refer to <<ChCaptureExtcapWindowsShell>> for more information.

[[ChCaptureExtcapProcess]]

==== Extcap command line interface

The actual capture is run after a setup process that can be made manually by the
user or automatically by the GUI. All the steps performed are done for every extcap.

Let's go through those steps.

===== Query for available interfaces

In the first step the extcap is queried for its interfaces.

[source,bash]
----
$ extcapbin --extcap-interfaces
----

This call must print the existing interfaces for this extcap and must return 0.
The output must conform to the grammar specified for extcap, and it is specified
in the doc/extcap.4 generated man page (in the build dir).

Since Wireshark 2.9 this call is extended with `--extcap-version x.x`, which will
allways represent the calling Wireshark's version information. This can be used
to change behavior depending on the Wireshark version in question.

.Example call for interface query
[source,bash]
----
$ extcap_examply.py --extcap-interfaces --extcap-version 3.0
extcap {version=1.0}{help=Some help url}
interface {value=example1}{display=Example interface 1 for extcap}
interface {value=example2}{display=Example interface 2 for extcap}
----

The *version* for the extcap sentence (which may exist as many times as is needed, but only
the last one will be used) will be used for displaying the version information of
the extcap interface in the about dialog of Wireshark.

The value for each interface will be used in subsequent calls as the interface name
IFACE.

Using the help argument, an interface may provide a generic help URL for the extcap
utility.

===== Ask for DLT's to each interface

The extcap binary is queried for all valid DLTs for all the interfaces returned by step 1.

[source,bash]
----
$ extcap_examply.py --extcap-dlts --extcap-interface IFACE
----

This call must print the valid DLTs for the interface specified. This call is
made for all the interfaces and must return 0.

.Example for the DLT query
[source,bash]
----
$ extcap_examply.py --extcap-interface IFACE --extcap-dlts
dlt {number=147}{name=USER1}{display=Demo Implementation for Extcap}
----

A binary or script, which neither provides an interface list or a DLT list will
not show up in the extcap interfaces list.


===== The extcap configuration interface

The extcap binary is asked for the configuration of a specific interface

[source,bash]
----
$ extcap_examply.py --extcap-interface IFACE --extcap-config
----

Each interface can have custom options that are valid for this interface only.
Those config options are specified on the command line when running the actual
capture. To allow an end-user to specify certain options, such options may be
provided using the extcap config argument.

To share which options are available for an interface, the extcap responds to
the command `--extcap-config`, that shows all the available options (aka additional command
line options).

Those options are automatically presented via a dialog to the user for the individual
interface.

.Example for interface options
[source,bash]
----
$ extcap_examply.py --extcap-interface IFACE --extcap-config
arg {number=0}{call=--delay}{display=Time delay}{tooltip=Time delay between packages}{type=integer}{range=1,15}{required=true}
arg {number=1}{call=--message}{display=Message}{tooltip=Package message content}{placeholder=Please enter a message here ...}{type=string}
arg {number=2}{call=--verify}{display=Verify}{tooltip=Verify package content}{type=boolflag}
arg {number=3}{call=--remote}{display=Remote Channel}{tooltip=Remote Channel Selector}{type=selector}
arg {number=4}{call=--server}{display=IP address for log server}{type=string}{validation=\\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\b}
value {arg=3}{value=if1}{display=Remote1}{default=true}
value {arg=3}{value=if2}{display=Remote2}{default=false}
----

Now the user can click on the options and change them. They are sent to the
extcap when the capture is launched.

There are three kind of options available:

[horizontal]
Flag:: boolflag for instance expects the option to be present resulting in the corresponding entry set to true, false otherwise
Value:: are value based options and each expect a single value via the command-line call
Selection:: are selections and can be presented multiple times in the command line. Both expect subsequent "value" items in the config list, with the corresponding argument selected via arg


===== The capture process

Once the interfaces are listed and configuration is customized by the user the capture is started.

[source,bash]
----
$ extcap_examply.py --extcap-interface IFACE [params] --capture [--extcap-capture-filter CFILTER]
  --fifo FIFO
----

To run the capture, the extcap must implement the `--capture`, `--extcap-capture-filter`
and `--fifo` option.

They are automatically added by Wireshark that opens the fifo for reading. All
the other options are automatically added to run the capture. The extcap interface
is used like all other interfaces (meaning that capture on multiple interfaces, as
well as stopping and restarting the capture is supported).

[[ChCaptureExtcapWindowsShell]]

====== Execute a script-based extcap on Windows

To use scripts on Windows, please generate an <scriptname>.bat inside
the extcap folder, with the following content (in this case for a Python-based extcap utility):

[source,batch]
----
@echo off
<Path to python interpreter> <Path to script file> %*
----

Windows is not able to execute most scripts directly (Powershell being an exception), which also goes for all other script-based formats besides VBScript and PowerShell



==== Extcap Arguments

The extcap interface provides the possibility for generating a GUI dialog to
set and adapt settings for the extcap binary.

All options must provide a number, by which they are identified. No `NUMBER` may be
provided twice. Also all options must present the elements `CALL` and `DISPLAY`, where
call provides the arguments name on the command-line and display the name in the GUI.

Additionally `TOOLTIP` and PLACEHOLDER may be provided, which will give the user an
explanation within the GUI, about what to enter into this field.

These options do have types, for which the following types are being supported:

[horizontal]
INTEGER, UNSIGNED, LONG, DOUBLE:: This provides a field for entering a numeric value of the given data type. A DEFAULT value may be provided, as well as a RANGE
+
[source,python]
----
arg {number=0}{call=--delay}{display=Time delay}{tooltip=Time delay between packages}{type=integer}{range=1,15}{default=0}
----

STRING:: Let the user provide a string to the capture
+
[source,python]
----
arg {number=1}{call=--server}{display=IP Address}{tooltip=IP Address for log server}{type=string}{validation=\\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\b}
----
+
`validation` allows to provide a regular expression string, which is used to check the user input for validity beyond normal data type or range checks. Back-slashes must be escaped (as in \\b for \b)

PASSWORD:: Let the user provide a masked string to the capture. Password strings are not saved, when the extcap configuration is being saved
+
[source,python]
----
arg {number=0}{call=--password}{display=The user password}{tooltip=The password for the connection}{type=password}
----

BOOLEAN, BOOLFLAG:: This provides the possibility to set a true/false value. BOOLFLAG values will only appear in the command-line if set to true, otherwise they will not be added to the command-line call for the extcap interface
+
[source,python]
----
arg {number=2}{call=--verify}{display=Verify}{tooltip=Verify package content}{type=boolflag}
----

 FILESELECT:: Let the user provide a filepath to the capture. If MUSTEXIST is being provided, the GUI checks if the file exists.
+
[source,python]
----
arg {number=3}{call=--logfile}{display=Logfile}{tooltip=A file for log messages}{type=fileselect}{mustexist=false}
----

SELECTOR, RADIO, MULTICHECK:: Optionfields, where the user may choose one or more options from. If PARENT is provided for the value items, the option fields for MULTICHECK and SELECTOR are being presented in a tree-like structure. SELECTOR and RADIO values must present a default value, which will be the value provided to the extcap binary for this argument
+
[source,python]
----
arg {number=3}{call=--remote}{display=Remote Channel}{tooltip=Remote Channel Selector}{type=selector}
value {arg=3}{value=if1}{display=Remote1}{default=true}
value {arg=3}{value=if2}{display=Remote2}{default=false}
----

===== Reload a selector
A selector may be reloaded from the configuration dialog of the extcap application within Wireshark. With the reload argument (defaults to false), the entry can be marked as reloadable.

[source,python]
----
arg {number=3}{call=--remote}{display=Remote Channel}{tooltip=Remote Channel Selector}{type=selector}{reload=true}{placeholder=Load interfaces...}
----

After this has been defined, the user will get a button displayed in the configuration dialog for this extcap application, with the text "Load interfaces..." in this case, and a generic "Reload" text if no text has been provided.

The extcap utility is then called again with all filled out arguments and the additional parameter `--extcap-reload-option <option_name>`. It is expected to return a value section for this option, as it would during normal configuration. The provided option list is then presented as the selection, a previous selected option will be reselected if applicable.

===== Validation of arguments

Arguments may be set with `{required=true}` which enforces a value being provided, before
a capture can be started using the extcap options dialog. This is not being checked, if
the extcap is started via a simple double-click. The necessary fields are marked for the
customer, to ensure a visibility for the end customer of the required argument.

Additionally text and number arguments may also be checked using a regular expression,
which is provided using the validation attribute (see example above). The syntax for
such a check is the same as for Qt RegExp classes. This feature is only active in the
Qt version of Wireshark.


==== Toolbar Controls

An extcap utility can provide configuration for controls to use in an interface toolbar.
These controls are bidirectional and can be used to control the extcap utility while
capturing.

This is useful in scenarios where configuration can be done based on findings in the
capture process, setting temporary values or give other inputs without restarting the
current capture.

.Example of interface definition with toolbar controls
[source,bash]
----
$ extcap_example.py --extcap-interfaces
extcap {version=1.0}{display=Example extcap interface}
interface {value=example1}{display=Example interface 1 for extcap}
interface {value=example2}{display=Example interface 2 for extcap}
control {number=0}{type=string}{display=Message}{tooltip=Package message content. Must start with a capital letter.}{validation=[A-Z]+}{required=true}
control {number=1}{type=selector}{display=Time delay}{tooltip=Time delay between packages}
control {number=2}{type=boolean}{display=Verify}{default=true}{tooltip=Verify package content}
control {number=3}{type=button}{display=Turn on}{tooltip=Turn on or off}
control {number=4}{type=button}{role=logger}{display=Log}{tooltip=Show capture log}
value {control=1}{value=1}{display=1 sec}
value {control=1}{value=2}{display=2 sec}{default=true}
----

All controls will be presented as GUI elements in a toolbar specific to the extcap
utility.  The extcap must not rely on using those controls (they are optional) because
of other capturing tools not using GUI (e.g. tshark, tfshark).


===== Controls
The controls are similar to the ARGUMENTS, but without the CALL element.  All controls
may be given a default value at startup and most can be changed during capture, both
by the extcap and the user (depending on the type of control).

All controls must provide a NUMBER, by which they are identified. No NUMBER may be
provided twice. Also all options must present the elements TYPE and DISPLAY, where
TYPE provides the type of control to add to the toolbar and DISPLAY the name in the GUI.

Additionally TOOLTIP and PLACEHOLDER may be provided, which will give the user an
explanation within the GUI.

All controls, except from the logger, help and restore buttons, may be disabled
(and enabled) in GUI by the extcap during capture. This can be because of set-once
operations, or operations which takes some time to complete.

All control values which are changed by the user (not equal to the default value) will
be sent to the extcap utility when starting a capture.  The extcap utility may choose
to discard initial values and set new values, depending on implementation.

These TYPEs are defined as controls:

[horizontal]
BOOLEAN:: This provides a checkbox with the possibility to set a true/false value. 
+
The extcap utility can set a default value at startup, and can change (set) and receive value changes while capturing. When starting a capture the GUI will send the value if different from the default value.
+
The payload is one byte with binary value 0 or 1.
+
Valid Commands: Set value, Enable, Disable.

BUTTON:: This provides a button with different ROLEs:

CONTROL:::: This button will send a signal when pressed. This is the default if no role is configured. The button is only enabled when capturing.
+
The extcap utility can set the button text at startup, and can change (set) the button text and receive button press signals while capturing.  The button is disabled and the button text is restored to the default text when not capturing.
+
The payload is either the button text or empty (signal).
+
Valid Commands: Set value, Enable, Disable.

LOGGER:::: This provides a logger mechanism where the extcap utility can send log entries to be presented in a log window. This communication is unidirectional.
+
The payload is the log entry, and should be ended with a newline. Maximum length is 65535 bytes.
+
Valid Commands: Set log entry, Add log entry.
+
The Set command will clear the log before adding the entry.
+
HELP:::: This button opens the help page, if configured. This role has no controls and will not be used in communication.
+
Valid Commands: NONE.

RESTORE:::: This button will restore all control values to default. This role has no controls and will not be used in communication. The button is only enabled when not capturing.
+
Valid Commands: NONE.

SELECTOR:: This provides a combo box with fixed values which can be selected.
+
The extcap utility can set default values at startup, and add and remove values and receive change in value selection while capturing. When starting a capture the GUI will send the value if different from the default value.
+
The payload is a string with the value, and optionally a string with a display value if this is different from the value. This two string values are separated by a null character.
+
Valid Commands: Set selected value, Add value, Remove value, Enable, Disable.
+
If value is empty the Remove command will remove all entries.

STRING:: This provides a text edit line with the possibility to set a string or any value which can be represented in a string (integer, float, date, etc.).
+
The extcap utility can set a default string value at startup, and can change (set) and receive value changes while capturing. When starting a capture the GUI will send the value if different from the default value.
+
The payload is a string with the value. Maximum length is 32767 bytes.
+
Valid Commands for control: Set value, Enable, Disable.
+
The element VALIDATION allows to provide a regular expression string, which is used to check the user input for validity beyond normal data type or range checks. Back-slashes must be escaped (as in \\b for \b).


===== Messages
In addition to the controls it's possible to send a single message from the extcap
utility to the user.  This message can be put in the status bar or displayed in a
information, warning or error dialog which must be accepted by the user.  This message
does not use the NUMBER argument so this can have any value.


====== Control Protocol

The protocol used to communicate over the control pipes has a fixed size header of
6 bytes and a payload with 0 - 65535 bytes.

.Control packet:
[cols="^m", width="50%"]
|===
|Sync Pipe Indication (1 byte)
|Message Length +
 (3 bytes network order)
|Control Number (1 byte)
|Command (1 byte)
|Payload +
 (0 - 65535 bytes)
|===

.Sync Pipe Indication:
  The common sync pipe indication. This protocol uses the value 'T'.

.Message Length:
  Payload length + 2 bytes for control number and command.

.Control Number:
   Unique number to identify the control. This number also gives the order of the controls in the interface toolbar.

.Commands and application for controls
[cols="1,2,3"]
|===
|Command Byte|Command Name|Control type

|0 |Initialized           |none
|1 |Set                   |boolean / button / logger / selector / string
|2 |Add                   |logger / selector
|3 |Remove                |selector
|4 |Enable                |boolean / button / selector / string
|5 |Disable               |boolean / button / selector / string
|6 |Statusbar message     |none
|7 |Information message   |none
|8 |Warning message       |none
|9 |Error message         |none
|===

 Payload Length:
   The length of the following payload.  Maximum length is 65535 bytes.

The Initialized command will be sent from the GUI to the extcap utility when all
user changed control values are sent after starting a capture. This is an indication
that the GUI is ready to receive control values.

The GUI will only send Initialized and Set commands.  The extcap utility shall not
send the Initialized command.

Messages with unknown control number or command will be silently ignored.


// End of WSDG Chapter Capture