Restores support for filters such as "mac-lte", that was broken
in 330d408328.
This means we are not able to support arithmetic expressions with binary
minus without spaces.
$ dftest 'tcp.port == 1-2'
dftest: "1-2" is not a valid number.
To allow an arithmetic expressions without spaces, such as "1+2",
we cannot match the expression in other lexical rules using "+". Because
of longest match this becomes the token LITERAL or UNPARSED with semantic value
"1+2". The same goes for all the other arithmetic operators.
So we need to remove [+-*/%] from "word chars" and add very specific
patterns (that won't mistakenly match an arithmetic expression) for
those literal or unparsed tokens we want to support using these characters.
The plus was not a problem but right slash is used for CIDR, minus for
mac address separator, etc.
There are still some corner case. 11-22-33-44-55-66 is a mac
address and not the arithmetic expression with six terms "eleven
minus twenty two minus etc." (if we ever support more than two terms
in the grammar, which we don't currently).
We lift some patterns from the flex manual to match on IPv4 and
IPv6 (ugly) and add MAC address.
Other hypothetical literal lexical values using [+-*/%] are already
supported enclosed in angle brackets but the cases of MAC/IPv4/IPv6 are
are very common and moreover we need to do the utmost to not break backward
compatibily here.
Before:
$ dftest "_ws.ftypes.int32 == 1+2"
dftest: "1+2" is not a valid number.
After:
$ dftest "_ws.ftypes.int32 == 1+2"
Filter: _ws.ftypes.int32 == 1+2
Instructions:
00000 READ_TREE _ws.ftypes.int32 -> reg#0
00001 IF_FALSE_GOTO 4
00002 ADD 1 <FT_INT32> + 2 <FT_INT32> -> reg#1
00003 ANY_EQ reg#0 == reg#1
00004 RETURN
This replaces the current macro reference system with
a completely different implementation. Instead of a macro a reference
is a syntax element. A reference is a constant that can be filled
in the dfilter code after compilation from an existing protocol tree.
It is best understood as a field value that can be read from a fixed
tree that is not the frame being filtered. Usually this fixed tree
is the currently selected frame when the filter is applied. This
allows comparing fields in the filtered frame with fields in the
selected frame.
Because the field reference syntax uses the same sigil notation
as a macro we have to use a heuristic to distinguish them:
if the name has a dot it is a field reference, otherwise
it is a macro name.
The reference is synctatically validated at compile time.
There are two main advantages to this implementation (and a couple of
minor ones):
The protocol tree for each selected frame is only walked if we have a
display filter and if the display filter uses references. Also only the
actual reference values are copied, intead of loading the entire tree
into a hash table (in textual form even).
The other advantage is that the reference is tested like a protocol
field against all the values in the selected frame (if there is more
than one).
Currently the reference fields are not "primed" during dissection, so
the entire tree is walked to find a particular reference (this is
similar to the previous implementation).
If the display filter contains a valid reference and the reference is
not loaded at the time the filter is run the result is the same as a
non existing field for a regular READ_TREE instruction.
Fixes#17599.
This change implements a unary minus operator.
Filter: tcp.window_size_scalefactor == -tcp.dstport
Instructions:
00000 READ_TREE tcp.window_size_scalefactor -> reg#0
00001 IF_FALSE_GOTO 6
00002 READ_TREE tcp.dstport -> reg#1
00003 IF_FALSE_GOTO 6
00004 MK_MINUS -reg#1 -> reg#2
00005 ANY_EQ reg#0 == reg#2
00006 RETURN
It is supported for integer types, floats and relative time values.
The unsigned integer types are promoted to a 32 bit signed integer.
Unary plus is implemented as a no-op. The plus sign is simply ignored.
Constant arithmetic expressions are computed during compilation.
Overflow with constants is a compile time error. Overflow with
variables is a run time error and silently ignored. Only a debug
message will be printed to the console.
Related to #15504.
Add support for masking of bits. Before the bitwise operator
could only test bits, it did not support clearing bits.
This allows testing if any combination of bits are set/unset
more naturally with a single test. Previously this was only
possible by combining several bitwise predicates.
Bitwise is implemented as a test node, even though it is not.
Maybe the test node should be renamed to something else.
Fixes#17246.
The representation "~= has been superseded by "!==" with the same
meaning, making it superfluous and somewhat confusing. Deprecate
"~=" and recommend "!==" instead.
The syntax for protocols and some literals like numbers
and bytes/addresses can be ambiguous. Some protocols can
be parsed as a literal, for example the protocol "fc"
(Fibre Channel) can be parsed as 0xFC.
If a numeric protocol is registered that will also take
precedence over any literal, according to the current
rules, thereby breaking numerical comparisons to that
number. The same for an hypothetical protocol named "true",
etc.
To allow the user to disambiguate this meaning introduce
new syntax.
Any value prefixed with ':' or enclosed in <,> will be treated
as a literal value only. The value :fc or <fc> will always
mean 0xFC, under any context. Never a protocol whose filter
name is "fc".
Likewise any value prefixed with a dot will always be parsed
as an identifier (protocol or protocol field) in the language.
Never any literal value parsed from the token "fc".
This allows the user to be explicit about the meaning,
and between the two explicit methods plus the ambiguous one
it doesn't completely break any one meaning.
The difference can be seen in the following two programs:
Filter: frame == fc
Constants:
Instructions:
00000 READ_TREE frame -> reg#0
00001 IF-FALSE-GOTO 5
00002 READ_TREE fc -> reg#1
00003 IF-FALSE-GOTO 5
00004 ANY_EQ reg#0 == reg#1
00005 RETURN
--------
Filter: frame == :fc
Constants:
00000 PUT_FVALUE fc <FT_PROTOCOL> -> reg#1
Instructions:
00000 READ_TREE frame -> reg#0
00001 IF-FALSE-GOTO 3
00002 ANY_EQ reg#0 == reg#1
00003 RETURN
The filter "frame == fc" is the same as "filter == .fc",
according to the current heuristic, except the first form
will try to parse it as a literal if the name does not
correspond to any registered protocol.
By treating a leading dot as a name in the language we
necessarily disallow writing floats with a leading dot. We
will also disallow writing with an ending dot when using
unparsed values. This is a backward incompatibility but has
the happy side effect of making the expression {1...2}
unambiguous.
This could either mean "1 .. .2" or "1. .. 2". If we require
a leading and ending digit then the meaning is clear:
1.0..0.2 -> 1.0 .. 0.2
Fixes#17731.
Before:
Filter: http.user_agent == açaí
dftest: "�" was unexpected in this context.
After:
Filter: http.user_agent == açaí
dftest: Non-printable ASCII characters may only appear inside double-quotes.
Related with #17770.
To complete the set of equality operators add an "all equal"
operator that matches a frame if all fields match the condition.
The symbol chosen for "all_eq" is "===".
Invalid character constants should be handled in the lexical scanner.
Todo: See if some code could be shared to parse double quoted strings.
It also fixes some unintuitive type coercions to string. Character
constants should be treated as characters, or maybe integers, or
maybe even throw an invalid comparison error, but coverting to a
literal string or byte array is surprising and not particularly
useful:
'\xFF' -> "'\xFF'" (equals)
'\xFF' -> "FF" (contains)
Before:
Filter: http.request.method contains "\x63"
Constants:
00000 PUT_FVALUE "c" <FT_STRING> -> reg#1
(...)
Filter: http.request.method contains '\x63'
Constants:
00000 PUT_FVALUE "63" <FT_STRING> -> reg#1
(...)
Filter: http.request.method == "\x63"
Constants:
00000 PUT_FVALUE "c" <FT_STRING> -> reg#1
(...)
Filter: http.request.method == '\x63'
Constants:
00000 PUT_FVALUE "'\\x63'" <FT_STRING> -> reg#1
(...)
After:
Filter: http.request.method contains '\x63'
Constants:
00000 PUT_FVALUE "c" <FT_STRING> -> reg#1
(...)
Filter: http.request.method == '\x63'
Constants:
00000 PUT_FVALUE "c" <FT_STRING> -> reg#1
(...)
For double quoted strings. This is consistent with single quote
character constants and the C standard. It also avoids common
mistakes where the superfluous backslash is silently suppressed.
Revert to the original design of having a single pattern to catch
everything as unparsed and also try to be less hackish and fragile
parsing "..".
Strings like "80..90" are tricky because it can be parsed as
INTEGER DOTDOT INTEGER or FLOAT FLOAT.
Deprecate the usage of significant whitespace to separate set elements
(or anywhere else for that matter). This will make the implementation
simpler and cleaner and the language more expressive and user-friendly.
Fixup comment to reflect that unparsed is the only token
type for the pattern, there is no "match" or "no match".
Tighten the CIDR patterns and make sure it cannot match a
rogue "..". This is ambiguous with 80..90 and must be treated
extra carefully to support floating point formats without
leading or terminating digits before or after the point.
Reject the default match as a syntax error. This pattern is
just a pitfall to make testing and debugging harder. Matching a
single arbitrary character is never a valid unparsed string,
except by chance.
Replace the LVAL macros with a plain variable declaration.
The other macros and assorted compatibility code has been superseded
by the requirement of reentrant flex and the use of more modern flex
options, as far as I can tell.
Using a hand written tokenizer is simpler than using flex start
conditions. Do the validation in the drange node constructor.
Add validation for malformed ranges with different endpoint signs.
Wireshark defines the relation of equality A == B as
A any_eq B <=> An == Bn for at least one An, Bn.
More accurately I think this is (formally) an equivalence
relation, not true equality.
Whichever definition for "==" we choose we must keep the
definition of "!=" as !(A == B), otherwise it will
lead to logical contradictions like (A == B) AND (A != B)
being true.
Fix the '!=' relation to match the definition of equality:
A != B <=> !(A == B) <=> A all_ne B <=> An != Bn, for
every n.
This has been the recomended way to write "not equal" for a
long time in the documentation, even to the point where != was
deprecated, but it just wasn't implemented consistently in the
language, which has understandably been a persistent source
of confusion. Even a field that is normally well-behaved
with "!=" like "ip.src" or "ip.dst" will produce unexpected
results with encapsulations like IP-over-IP.
The opcode ALL_NE could have been implemented in the compiler
instead using NOT and ANY_EQ but I chose to implement it in
bytecode. It just seemed more elegant and efficient
but the difference was not very significant.
Keep around "~=" for any_ne relation, in case someone depends
on that, and because we don't have an operator for true equality:
A strict_equal B <=> A all_eq B <=> !(A any_ne B).
If there is only one value then any_ne and all_ne are the same
comparison operation.
Implementing this change did not require fixing any tests so it
is unlikely the relation "~=" (any_ne) will be very useful.
Note that the behaviour of the '<' (less than) comparison relation
is a separate, more subtle issue. In the general case the definition
of '<' that is used is only a partial order.
Always use the internal API to access "deprecated" and initialize
the data structure on demand. This fixes a null pointer dereference
introduced previously.
Use reference counting to share the array cleanly and avoid memory
leaks.
Keep the pointer in dfwork_t.
The lexical rules for fields and unparsed strings are ambiguous,
e.g. "fc" can be the protocol fibre channel or the byte 0xfc.
In general a name is determined to be a protocol field or not by
checking the registry.
Resolving the name in the parser gives more flexibility, for example
to use different semantic rules according to the relation between
LHS and RHS, and allows function names and protocol names to co-exist
without ambiguity.
Before:
Filter: tcp == 1
Constants:
00000 PUT_FVALUE 01 <FT_PROTOCOL> -> reg#1
Instructions:
00000 READ_TREE tcp -> reg#0
00001 IF-FALSE-GOTO 3
00002 ANY_EQ reg#0 == reg#1
00003 RETURN
Filter: tcp() == 1
dftest: Syntax error near "(".
After:
Filter: tcp == 1
Constants:
00000 PUT_FVALUE 01 <FT_PROTOCOL> -> reg#1
Instructions:
(same)
Filter: tcp() == 1
dftest: Function 'tcp' does not exist
It's also a goal to make it easier to modify the lexer rules.
Ping #12810.
Do the integer conversion for ranges in the parser. This is more
conventional, I think, and allows removing the unnecessary integer
syntax tree node type.
Try to minimize the number and complexity of lexical rules for
ranges. But it seems we need to keep different states for integer
and punctuation because of the need to disambiguate the ranges
[-n-n] and [-n--n].
A function is grammatically an identifier that is followed by '(' and ')'
according to some rules. We should avoid assuming a token is a function
just because it matches a registered function name.
Before:
Filter: foobar(http.user_agent) contains "UPDATE"
dftest: Syntax error near "(".
After:
Filter: foobar(http.user_agent) contains "UPDATE"
dftest: The function 'foobar' does not exist.
This has the problem that a function cannot have the same name
as a protocol but that limitation already existed before.
When parsing we save the token value to the syntax tree. This is
useful for better error reporting. Use it to report an invalid
entity for the slice operation. Before only the memory location
was reported, which is not a good error message.
Before:
% dftest '"01:02:03:04"[0:3] == foo'
Filter: ""01:02:03:04"[0:3] == foo"
dftest: Range is not supported for entity <0x7f6c84017740> of type STRING
After:
% dftest '"01:02:03:04"[0:3] == foo'
Filter: ""01:02:03:04"[0:3] == foo"
dftest: Range is not supported for entity 01:02:03:04 of type STRING
When creating a new node from an old one we need to copy the token
value. Simple tokens such as RBRACKET, COMMA and COLON are
not part of the AST and don't have an associated semantic value.
Pass the deprecated data struture to the scanner and insert the deprecated
tokens there. This avoids having to keep a dedicated syntax node field
for this.
Pass the deprecated argument in dfwork_t instead of in a separate
argument. This is less cumbersome than adding an extra argument
to every level of the semantic checker.
When byte escape sequences, that is hex \xhh or octal \0ddd,
are interpreted at the lexical level it is not possible to
use strings with embedded NUL bytes. The NUL byte is interpreted
as a C string terminator. As a consequence, for example, the
strings "AB" and "AB\x00CDE" compare as the same. This leads to
unexpected false matches and a poor user experience.
Disallow embedded NULs for regular strings (strings literals that
do not begin with 'r' or 'R') for this reason.
It is possible to use a raw string instead (eg: r"AB\x00C")
to match embedded NUL bytes, although that only works with regular
expressions. Normal escape rules would also work with regular
expressions (eg: "AB\\x00C"). This is the same string as the previous
one, written in an alternate form. What won't work is "AB\x00C", this
string is synctatically invalid.
So the expression: data matches r"AB\x00C"
will match the bytes {'A', 'B', '\0', '\C'}.
However the expression: data contains r"AB\x00C"
won't match the fvalue above. Because the "contains" operator
doesn't compile a regular expression it literally tries to
contains-match the bytes {'A', 'B', '\\', 'x', '0', '0', 'C'}.
Therefore raw strings are very convenient but it is still necessary
to be aware that the matches operator has an extra level of indirection
than other string operators (same as in Python).
Fixes#16156.
Add support for a literal string specification copied from Python
raw strings[1].
Raw string literals are enclosed with r"..." or R"...". Double quotes
can be include in the string but they must be escaped with backslash.
In escape sequences backslashes are preserved in the final result.
So for example the string "a\\\"b" is the same as r"a\"b".
r"\\\a" is the same as "\\\\\\a".
Raw strings should be used for convenience wherever a regular expression
is used in a display filter expression.
[1]https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
(It's llvm-gcc, and appears to be a version too old to support
_Pragma("gcc diagnostic ...").)
Change-Id: I220997890636d5a5c33889d2e6c640feee155deb
Reviewed-on: https://code.wireshark.org/review/29580
Reviewed-by: Guy Harris <guy@alum.mit.edu>
Add support for aliasing one protocol name to another and for filtering
using aliased fields. Mark aliased fields as deprecated.
Rename the BOOTP dissector to DHCP and alias "bootp" to "dhcp". This
lets you use both "dhcp.type" and "bootp.type" as display filter fields
without having to duplicate all 500+ DHCP/BOOTP fields.
To do:
- Add checks to proto.c:check_valid_filter_name_or_fail?
- Transition SSL to TLS.
- Rename packet-bootp.c to packet-dhcp.c?
Change-Id: I29977859995e8347d80b8e83f1618db441b10279
Ping-Bug: 14922
Reviewed-on: https://code.wireshark.org/review/29327
Reviewed-by: Gerald Combs <gerald@wireshark.org>
Petri-Dish: Gerald Combs <gerald@wireshark.org>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Previously a filter such as `http.request.method in {"GET"HEAD""}` would
be parsed as three strings (GET, HEAD and an empty string). As it seems
more likely that people make typos rather than intending to construct
such a filter, forbid this by always requiring a whitespace separator.
Change-Id: I77e531fd6be072f62dd06aac27f856106c8920c6
Reported-by: Stig Bjørlykke
Reviewed-on: https://code.wireshark.org/review/26989
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
For numeric values such as port numbers, "4430..4434" looks more
natural than "4430 .. 4434", so support that.
To make this possible, the display filter syntax needs to be restricted.
Assume that neither field names nor values can contain "..". The display
filter `data contains ..` will now be considered a syntax error and must
be written as `data contains ".."` instead. More generally, all values
that contain ".." must be quoted.
Other than the ".." restriction, the scanner deliberately accepts more
characters that can potentially form invalid input. This is to prevent
accidentally splitting input in multiple tokens. For example, "9.2." in
"frame.time_delta in {9.2.}" is currently parsed as one token and then
rejected because it cannot be parsed as time. If the scanner was made
stricter, it could treat it as two tokens (floats), "9." and "2." which
has different meaning for the set membership operator.
An unhandled edge case is "1....2" which is parsed as "1 .. .. 2" but
could have been parsed as "1. .. .2" instead. A float with trailing dots
followed by ".." seems sufficiently weird, so rejection is fine.
Ping-Bug: 14180
Change-Id: Ibad8e851b49346c9d470f09d5d6a54defa21bcb9
Reviewed-on: https://code.wireshark.org/review/26960
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Allow "tcp.srcport in {1662 1663 1664}" to be abbreviated to
"tcp.srcport in {1662 .. 1664}". The range operator is supported for any
field value which supports the "<=" and "=>" operators and thus works
for integers, IP addresses, etc.
The naive mapping "tcp.srcport >= 1662 and tcp.srcport <= 1664" is not
used because it does not have the intended effect with fields that have
multiple occurrences (e.g. tcp.port). Each condition could be satisfied
by an other value. Therefore a new DVFM instruction (ANY_IN_RANGE) is
added to test the range condition against each individual field value.
Bug: 14180
Change-Id: I53c2d0f9bc9d4f0ffaabde9a83442122965c95f7
Reviewed-on: https://code.wireshark.org/review/26945
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
João Valverde