This program generates complete pcap files containing the proposed U-SIG
radiotap TLVs along with enough else to make it readable. You cannot currently
read such packets with tshark or wireshark until I add U-SIG handling to
Wireshark.
We keep our various packaging assets in the "packaging" directory. Move
the Debian assets there. dpkg-buildpackage doesn't seem appear to have a
"debian directory path" option, but symlinking worked in my test
container.
macos-setup.sh:
- Fix filename of libtiff in existence test from "libtiff" to "tiff"
- Added fallback URL for libtiff when the downloaded file is not a valid gzip
archive. The host rotates older versions of libtiff into an "old"
subdirectory, so curl downloads a 404 Web page and exits without error. Then
the call to gzcat fails with an invalid gzip archive error. Maybe libtiff
version should be updated instead?
When checking is_dissector_file(), only match against files that
end in ".c" and not, e.g. ".c.swp" ".c~" or other such temporary
files that might be binary files (as with vim .swp files).
Prevents errors like "UnicodeDecodeError: 'utf-8' codec can't decode
byte 0xe4 in position 18: invalid continuation byte" with Python 3
when a dissector file is open in vim.
Repeated words were found with:
egrep "(\b[a-zA-Z]+) +\1\b" . -Ir
and then manually reviewed.
Non-displayed strings (e.g., in comments)
were also corrected, to ease future review.
The Enhanced Trading Interface (ETI) protocol and the Enhanced
Order Book Interface (EOBI) protocol are used by a few European
exchanges such as Eurex, Xetra and Börse Frankfurt.
Basically, a trader uses ETI to communicate with a matching
engine (over TCP), e.g. to add a new order, modify an existing
one, etc. while the matching engine also publicizes the current
state of the order book via EOBI over multicast UDP feeds.
ETI actually consists of two variants, i.e. ETI for derivatives
markets (such as Eurex) and ETI for cash markets (such as Xetra).
A common convention is to abbreviate them as ETI (for
derivatives) and XTI (for cash).
These protocols share the same encoding, i.e. messages start with
a length and a tag field and most messages and fields are fixed
size. See also
https://github.com/gsauthof/python-eti#protocol-introduction for
some more details.
The protocol specifications are openly available (cf.
https://github.com/gsauthof/python-eti#protocol-descriptions for
direct links) in human and machine-readable (XML) formats.
The Wireshark ETI/XTI/EOBI dissectors are code-generated by
`eti2wireshark.py`
(https://github.com/gsauthof/python-eti/blob/master/eti2wireshark.py)
which is GPL licensed. See also
https://github.com/gsauthof/python-eti#wireshark-protocol-dissectors
for usage examples and related work.
At least on Monterey, with Xcode 13.1, the linker whines that we weren't
granted the Sacred and Holy Right to link with the Python 2.7 framework.
As far as I know, we have no need to use that framework, so configure it
out.
Point it to fetch files from falcosecurity/libs repo.
Moreover, add support for blank spaces in param names.
Signed-off-by: Federico Di Pierro <nierro92@gmail.com>
Since we now support ISO 8601 Basic format, have asn2wrs.py
convert GeneralizedTime fields in BER to FT_ABSOLUTE_TIMEs and use
the new common code to convert them. This means that the fields
can be compared with other time fields in filters, etc.
C is notoriously difficult to bind from other languages
without additional metadata. The C ABI does not include
enums and macros that are an essential component of the
API.
To make Wireshark instrospectable and more binding friendly
include an introspection API to export enums and integer macros.
To avoid the tedious need to manually keep the code up to date
it uses the excellent pyclibrary python package to automatically
parse C headers and extract this data.
This is not a process that should be done automatically during
the build.
This could be used for example to replace most of the wslua
make-init-lua.pl perl script, which tries to do the same thing
using regular expressions.
Besides the downside of using Perl using regular expressions
is inferior to pyclibrary in 2 ways: 1) pyclibrary understands
most of C99 grammar so it is much more powerful; 2) pyclibrary
has a specific API to extract "values" (enums and constants)
automagically. We just need to take care to use only integer
values, for our purposes.
This was intentionally kept simple (matches the philosophy of Arch).
In particular I wasn't so concerned about what is a required build
dependency and what is an optional build dependency to compile the
programs. I don't know why one would ever wish to skip installation
of non-essential library dependencies. But others are very welcome
to extend this intentionally barebones effort.
The script also adds an "--install-all" flag to install everything
at once. I keep forgetting the name of the other options.
I used the build optional flag to install packages required to build
documentation and so on. Ancillary stuff.
PCRE2 is the future of PCRE. The only advantage of GRegex is that
it comes bundled with GLib, which is not an advantage at all.
PCRE2 is widely available, the GRegex abstractions layer are not a
good fit and abstract things that don't need abstracting or that we
could handle better ourselves, there are open bugs (#12997) and
maintenance is spotty at best.
GRegex comes with many of the problems of bundled code, aggravated by
the fact that it completely falls outside of our control.
The header ftypes-int.h should not be used outside of epan/ftypes
because it is a private header.
The functions fvalue_free() and fvalue_cleanup() need not and should
not be macros either.
Remove a duplicated argument to fix a warning:
Wrong number of arguments for string format.
Format ptvcursor_add(cursor, hf_skinny_%s, 6, ENC_NA);
takes 1, but 2 are provided.
Add tname as argument to Type eth_type_default_body() to fix a warning.
Call to method Type.eth_type_default_body with too many arguments;
should be no more than 1.
Add ID count sanity checks and make sure we don't update pci-ids.c
unless the checks pass. Fix a bunch of Pylint warnings. Strip leading
whitespace from our output strings.
I found that SkinnyProtocolOptimized.xml and packet-skinny.c.in are not in
sync with packet-skinny.c. Obviously packet-skinny.c file was modified
multiple times manually.
I made changes:
- synced all typos fixed in packet-skinny.c to SkinnyProtocolOptimized.xml
- improved parse_xml2skinny_dissector.py to be able to generate
additional information to flow sequence
- updated SkinnyProtocolOptimized.xml to mark where to generate
additional information
If fact the outcome is just refactoring of original code.
Asterix data format is a complex family of asterix categories,
where each individual category exists in multiple editions.
As a result of many variants, the epan/dissectors/packet-asterix.c
is one of the largest dissectors.
So far, the asterix dissector had been maintained manually, where the
generic decoding routines and category/edition specific definitions
were entangled in the same file (packet-asterix.c).
This commit preserves the overall dissector structure, but makes
it easy to update the dissector with new categories or editions as
they become available (via the update script from this commit).
See tools/asterix/README.md file for dissector update procedure.
This commit includes:
- tools/asterix/packet-asterix-template.c
Extraction of generic asterix decoding routines and
common data structures.
- tools/asterix/update-specs.py
Update script, to render the template with up-to-date asterix
specs files. The asterix specs files themselves are maintained in
a separate repository.
- epan/dissectors/packet-asterix.c
Automatically generated dissector for asterix data format.
Although generated, this file needs to remain in the repository,
to be able to build the project in a reproducible way.
The generated asterix dissector was additionally tested with:
- ./tools/check_typed_item_calls.py --mask
- ./tools/fuzz-test.sh
Sync with asterix-specs #cef694825c
Current errors are:
Error: epan/dissectors/packet-asterix.c filter= asterix.021_161_TN 0x0fff with len is 4 but type FT_UINT8 indicates max of 2 and extra digits are non-zero (0f)
Error: epan/dissectors/packet-capwap.c filter= capwap.control.message_element.ieee80211_station_session_key.flags_a 0x2000 with len is 4 but type FT_BOOLEAN indicates max of 1 and extra digits are non-zero (200)
Error: epan/dissectors/packet-capwap.c filter= capwap.control.message_element.ieee80211_station_session_key.flags_c 0x1000 with len is 4 but type FT_BOOLEAN indicates max of 1 and extra digits are non-zero (100)
Error: epan/dissectors/packet-cfdp.c filter= cfdp.trans_stat_2_b 0x6000 with len is 4 but type FT_UINT8 indicates max of 2 and extra digits are non-zero (60)
Error: epan/dissectors/packet-cfdp.c filter= cfdp.suspension_ind_b 0x8000 with len is 4 but type FT_UINT8 indicates max of 2 and extra digits are non-zero (80)
Error: epan/dissectors/packet-ixveriwave.c filter= ixveriwave.tx.factorydebug 0x7f80 with len is 4 but type FT_UINT8 indicates max of 2 and extra digits are non-zero (7f)
Error: epan/dissectors/packet-opa-snc.c filter= opa.snc.rhf.eccerr 0x200000000 with len is 9 but type FT_BOOLEAN indicates max of 8 and extra digits are non-zero (2)
Error: plugins/epan/ethercat/packet-ethercat-datagram.c filter= ecat.reg.ctrlstat.rdacc 0x0100 with len is 4 but type FT_BOOLEAN indicates max of 2 and extra digits are non-zero (01)
Error: plugins/epan/ethercat/packet-ethercat-datagram.c filter= ecat.reg.ctrlstat.wracc 0x0200 with len is 4 but type FT_BOOLEAN indicates max of 2 and extra digits are non-zero (02)
Error: plugins/epan/ethercat/packet-ethercat-datagram.c filter= ecat.reg.ctrlstat.reloadacc 0x0400 with len is 4 but type FT_BOOLEAN indicates max of 2 and extra digits are non-zero (04)
Error: plugins/epan/ethercat/packet-ethercat-datagram.c filter= ecat.reg.ctrlstat.crcerr 0x0800 with len is 4 but type FT_BOOLEAN indicates max of 2 and extra digits are non-zero (08)
Error: plugins/epan/ethercat/packet-ethercat-datagram.c filter= ecat.reg.ctrlstat.lderr 0x1000 with len is 4 but type FT_BOOLEAN indicates max of 2 and extra digits are non-zero (10)
Error: plugins/epan/ethercat/packet-ethercat-datagram.c filter= ecat.reg.ctrlstat.cmderr 0x2000 with len is 4 but type FT_BOOLEAN indicates max of 2 and extra digits are non-zero (20)
Error: plugins/epan/ethercat/packet-ethercat-datagram.c filter= ecat.reg.ctrlstat.wrerr 0x4000 with len is 4 but type FT_BOOLEAN indicates max of 2 and extra digits are non-zero (40)
Error: plugins/epan/ethercat/packet-ethercat-datagram.c filter= ecat.reg.ctrlstat.busy 0x8000 with len is 4 but type FT_BOOLEAN indicates max of 2 and extra digits are non-zero (80)
Asciidoctor is now required for packaging. Try to make sure it's
installed on CentOS 8 and openSUSE 15.2. Note that CentOS 8 doesn't have
an Asciidoctor package, which complicates our SPEC.
Convert doc/*.pod to Asciidoctor. This:
* Means we use the same markup for our man pages, the guides, and
release notes.
* Lets us add versions to our man pages.
* Gives us more formatting options, e.g. AsciiDoc supports `commands`,
nested lists and makes it easy to include version information. The
manpage backend doesn't seem to support tables very well,
unfortunately.
Convert our CMake configuration to produce *roff and html man pages
using Asciidoctor. Add a "manarg" block macro which makes our synopses
wrap correctly.
Similar to the release notes, guides, and FAQ, if Asciidoctor isn't
found the man pages won't be generated or installed.
Move Asciidoctor to the list of package build dependencies in various
places.
This commit includes the conversion script (pod2adoc.py), which will be
removed later.
Line count sanity check:
Man page .pod .adoc
androiddump 260 280
asn2deb 93 105
capinfos 401 471
captype 54 55
ciscodump 241 269
dftest 42 42
dpauxmon 153 169
dumpcap 464 534
editcap 528 583
etwdump 136 156
extcap 157 181
idl2deb 91 103
idl2wrs 120 100
mergecap 206 207
mmdbresolve 75 75
randpkt 107 111
randpktdump 158 184
rawshark 558 610
reordercap 76 78
sdjournal 145 157
sshdump 272 302
text2pcap 274 312
tshark 2135 2360
udpdump 133 151
wireshark-filter 486 479
wireshark 2967 3420
Implement little endian support for tvb_get_bits family of functions.
The big/little endian refers to bit numbering within an octet. In big
endian, the most significant bit is considered bit 0, while in little
endian the least significant bit is considered bit 0.
Add encoding parameters to proto tree bits format family functions.
Specify ENC_BIG_ENDIAN in all dissectors using these functions except in
USB HID that requires ENC_LITTLE_ENDIAN to work correctly.
When formatting bits values, always display most significant bit on the
leftmost position regardless of the encoding. This results in no gaps
between octets and makes the displayed value comprehensible.
Close#4478Fix#17014
Besides the obvious limitation of being unavailable on Windows,
the standard is vague about getopt() and getopt_long() has many
non-portable pitfalls and buggy implementations, that increase
the maintainance cost a lot. Also the GNU libc code currently
in the tree is not suited for embedding and is unmaintainable.
Own maintainership for getopt_long() and use the musl implementation
everywhere. This way we don't need to worry if optreset is available,
or if the $OPERATING_SYSTEM version behaves in subtly different ways.
The API is under the Wireshark namespace to avoid conflicts with
system headers.
Side-note, the Mingw-w64 9.0 getopt_long() implementation is buggy
with opterr and known to crash. In my experience it's a headache to
use the embedded getopt implementation if the system provides one.
Migrate compress-pngs from a Bash script that ran Make to a Python
script, which should be usable on more platforms.
Add Efficient Compression Tool (ect) to the list of compressors.
Add the compressors to the various *-setup.sh scripts, but comment them
out for now.
The Ubuntu build commented on some spelling errors in executable code
files. Fix the errors that don't come from external files containing
the spelling errors (USB product and vendor IDs, PCI IDs, ASN.1
specifications), and fix some errors that don't show up in the
executable code files (e.g., in comments and variable names).
Some checks intended for dissectors don't work well on dissector
*generators*, as they see stuff such as "value_string %s[]" in a format
string used to generate dissector code and get upset because the
purported value_string doesn't end with {0, NULL} (the generator *does*
put a {0, NULL} at the end, but the checker isn't clever enough to
figure that out).
A few of them just needed scratch memory, so allocate and free it
manually after doing any exception-raising checks.
A few others were returning memory, and needed conversion to accept a
wmem scope argument.
When cross-compiling wireshark the lemon tool should be built
using the host machine compiler to be run on the host. Before
cmake this was done via autotools CC_FOR_BUILD but cmake only
supports one compiler toolchain per build and requires some
workarounds like running cmake twice using separately defined
toolchains.
This gets ugly and complicated fast when considering multiple
toolchains, especially for a simple tool like lemon, so just
allow builds to override the C compiler and wipe the cflags.
This way systems like Gentoo/ChromeOS/Yocto with a properly
setup cross-compile environment can just point to the native
BUILD_CC or similar while minimizing complexity.
To quote the comment in plugins/epan/gryphon/packet-gryphon.c, "Note:
using external tfs strings doesn't work in a plugin", so, for plugins,
don't check to make sure that the file doesn't define a
true_false_string that's the same as one defined by tfs.c.
This is a new check added to check_typed_item_calls.py --label
Ignoring cases where item type is FT_NONE, as fpr tjpse
text was appended that otherwise would lack a colon.
For CI, will now return error codes only for those issues
that are definitely bugs that will require fixing. i.e.
- if the type is not compatible with the call
- if a TFS is (case-sensitively) identical to a tfs.c entrywq
We add them when configuring with autotools, so that we build GLib
appropriately for the OS versions we're targeting; do the same when
configuring with Meson.
If this version of macOS comes with a version of libffi, generate a .pc
file for it and install it in /usr/local/lib/pkgconfig, so that
pkg-config finds that version of libffi, and the GLib configuration
process - whether it's done with autotools or Meson - doesn't decide
that there is no libffi and fail or install its own libffi or whatever.
If we're running an external Python 3 package, pip3 will install scripts
in some directory under /Library; set MESON to point to the location
where Meson will be installed, and use that.
Have a meson-done file to indicate that Meson's been installed by us,
and uninstall it only if that's present.
We want to check whether *Apple* provides Python 3, not whether there's
a Python 3 installed; if there is no Apple-provided Python 3, but
there's somebody else's Python 3 installed, leave it alone, don't
uninstall it.
Newer versions of GLib require Meson (they don't support autotools) and
Ninja (they use Ninja rather than Make). Install Meson and, based on
the GLib version, use autotools+make or Meson+Ninja to build GLib.
Move up the installation of Python 3 so that it's available when we
install Meson, as Meson requires Python 3 and is installed with pip3.
Only install an external Python 3 if /usr/bin/python3 doesn't work; on
at least some versions of macOS, /usr/bin/python3 is a wrapper to run
Python 3 from Xcode, and at least some versions of Xcode provide Python
3.
1.10.2 is the latest version, and is the first version to ship as a fat
x86-64/ARM64 binary, so that we have native binaries for both platforms
supported by macOS.
Automated find/replace of wmem_packet_scope() with pinfo->pool in all
files where it didn't cause a build failure.
I also tweaked a few of the docs which got caught up.
Upgrade our vcpkg bundle to one that includes GLib 2.66.4 and libxml2
2.9.10.
Avoid running pkgconfig on Windows so that we don't find Strawberry
Perl's headers.
As of Debian bullseye and Ubuntu 21.04, `qt5-default` is no longer
available. This patch removes it and adds its dependencies instead
as suggested in <https://askubuntu.com/a/1335187/580576>.
Add a -t option to tools/fuzz-test.sh which lets you specify a maximum
fuzz time.
Add an initial "fuzz-test" job which fuzzes test/captures/* for 5
minutes. To do: Fuzz longer using our capture menagerie and report
failures.
Not all shells support [[ ]] compound commands; it's not in the most
recent Single UNIX Specification I could see, and the
ubuntu-clang-other-tests job is reporting
tools/validate-clang-check.sh: 18: [[: not found
Don't use [[ ]].
In addition, if you change extcap/etl.c, it tries to run clang-check on
it, but that file builds, and is only built, on Windows, so clang-check
fails dismally on UN*Xes. Omit it for now.
Update CMake (3.19.7), Qt (5.2.10), and Python (3.9.3) to later bugfix
versions of the current packages. CMake and Python have made tweaks in
the names of the binary packages that support different macOS versions.
Fixes downloading Python 3.9.2+ on macOS 11 after the package suffix
changed from -macos11.0.pkg to -macos11.pkg
Warn about the lack of Qt offline installers for version 5.15 and
greater.
The existing stuff doesn't appear to work (I tried it on 32-bit Ubuntu
18.04, and it did *not* add any flags to the compilation, as it appeared
not to conclude that they were necessary, even though they were).
Pull in the stuff from libpcap, which *does* appear to work. (it does
so in my 32-bit Ubuntu testing).
This should fix#17301.
While we're at it, fix cppcheck.sh so that it doesn't attempt to run
cppcheck on files that have been deleted.
It's not a valid field type, it's only a hack to support regular
expression matching in packet-matching expressions.
Instead, in the packet-matching code, have a separate syntax tree type
for Perl-compatible regular expressions, and a separate instruction to
load one into a register, and have the "matching" operator for field
types take a GRegex * as the second argument.
Jaap Keuter