The change is massive, mostly due to KIP-482. The flexible version coding affects every
string, bytes or array field. In order to keep the compatibility the old and new style field
codings must stay next to each other.
Plus:
* correlation-id request/response matching
* new fields (other than KIP-482)
* some fixes to the messages that were not tested sufficiently before
Bug: 16540
Bug: 16706
Bug: 16708
Change-Id: I39b1b6a230e393d3bee3e3d8625541add9c83e5d
Reviewed-on: https://code.wireshark.org/review/37886
Petri-Dish: Martin Mathieson <martin.r.mathieson@googlemail.com>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Fix the composite tvb handling for zstd decompression in the same way as
we already did for lz4 and snappy.
Allocate the composite tvb only if we are cetain that data will be added
to it. Do not free the composite tvb ourselves, leave this to epan cleanup.
Change-Id: Iac74ea6e6d220b05858a7eb267276ff983b1b2ab
Reviewed-on: https://code.wireshark.org/review/37900
Reviewed-by: Martin Kaiser <wireshark@kaiser.cx>
Petri-Dish: Martin Kaiser <wireshark@kaiser.cx>
Tested-by: Petri Dish Buildbot
Reviewed-by: Alexis La Goutte <alexis.lagoutte@gmail.com>
The snappy decompression routine has the same bug that was fixed for lz4 in
79576219c9 ("kafka: lz4: free the composite tvb only once").
Refactor the composite tvb handling for snappy as well. Allocate the
composite tvb only if we are cetain that data will be added to it.
Do not free the composite tvb ourselves, leave this to epan cleanup.
Change-Id: Ide3a88d1c02e525fe1aadd176068ce68c2330b98
Reviewed-on: https://code.wireshark.org/review/37838
Reviewed-by: Martin Kaiser <wireshark@kaiser.cx>
Petri-Dish: Martin Kaiser <wireshark@kaiser.cx>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Try to clean up the composite tvb handling during lz4 decompression.
If we detect an error straight away before doing any lz4 decompression, we
don't allocate a composite tvb at all. The comments in the tvb code say
explicitly that we must not call tvb_new_composite() without adding at
least one piece of data.
If we start decompressing and run into problems after creating the
composite tvb and linking it to the packet's main tvb, we must not free
the composite tvb manually. The epan library will do this for us when
dissection of the packet is finished.
While at it, make sure that we always finalize the composite tvb if we
allocated it and added data to it.
Bug: 16672
Change-Id: I3e3fb303a823640d7707277a109019fc3aad22f2
Reviewed-on: https://code.wireshark.org/review/37696
Petri-Dish: Alexis La Goutte <alexis.lagoutte@gmail.com>
Reviewed-by: Anders Broman <a.broman58@gmail.com>
There was a problem in FETCH response parsing when the server had more data than the requested
maximal return size. In such case the server checks if the first chunk of data fits into buffer. If it does not,
the first chunk is returned as a whole to the requestor. Otherwise it is assumed that the client is capable
of discarding invalid content and the server pushes maximum available block. It makes sense, because
the default block is 10MB and pushing it opaque leverages zero-copy IO from the file system to the network.
In the existing implementation it was assumed that the last batch is aligned with the end of the buffer.
Actually, if there is some data more, the last part is delivered truncated.
This patch:
* fixes the last part alignment handling
* adds opaque field for truncated content
* moves preferred replica field to the proper context
Bug: 16623
Change-Id: Iee6d513ce6711091e5561646a3fd563501eabdda
Reviewed-on: https://code.wireshark.org/review/37446
Petri-Dish: Alexis La Goutte <alexis.lagoutte@gmail.com>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Have dissect_kafka_string_new() set a flag if the length was negative.
If the length is negative, don't try to process what comes afterwards.
Make the length argument to decompression routines unsigned, and do
various checks. Don't try to decompress a zero-length block, and quit
if the decompressed block is zero-length.
Bug: 16082
Change-Id: I34c2ea99aa096b3f5724d9b113171b105bd6c60b
Reviewed-on: https://code.wireshark.org/review/34867
Petri-Dish: Guy Harris <guy@alum.mit.edu>
Tested-by: Petri Dish Buildbot
Reviewed-by: Guy Harris <guy@alum.mit.edu>
Fix dead store (Dead assignement/Dead increment) Warning found by Clang
Change-Id: I013c1bdc943033550f497b1be0dfc7979ca49517
Reviewed-on: https://code.wireshark.org/review/34484
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Fix dead store (Dead assignement/Dead increment) Warning found by Clang
Change-Id: I3ac2e2b6a1ed7621f65f1a98e8b7b3704e8b299d
Reviewed-on: https://code.wireshark.org/review/34481
Petri-Dish: Alexis La Goutte <alexis.lagoutte@gmail.com>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Add "native" support for the "zig-zag" version of a varint in proto.[ch] and
tvbuff.[ch]. Convert the use of varint in the KAFKA dissector to use the (new)
"native" API.
Ping-Bug: 15988
Change-Id: Ia83569203877df8c780f4f182916ed6327d0ec6c
Reviewed-on: https://code.wireshark.org/review/34386
Petri-Dish: Alexis La Goutte <alexis.lagoutte@gmail.com>
Tested-by: Petri Dish Buildbot
Reviewed-by: Alexis La Goutte <alexis.lagoutte@gmail.com>
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Bug: 16023
Change-Id: I78e1354ac5509707c818d7968c7067583fb469ba
Reviewed-on: https://code.wireshark.org/review/34379
Petri-Dish: Michael Mann <mmann78@netscape.net>
Tested-by: Petri Dish Buildbot
Reviewed-by: Michael Mann <mmann78@netscape.net>
Existing Apache Kafka support in Wireshark ends at version 0.10.
The version 0.11 (June 2017) brought significant changes to the message
format. This change makes the Wireshark Kafka dissector obsolete.
The recently released Kafka 2.3 has a lot of additions to the wire
protocol, which should be also addressed.
Major changes:
* Applied Kafka protocol changes since 0.10
* Zstd-packed message decompression (since Kafka 2.1)
* Added support for Kafka over TLS decryption
Bug: 15988
Change-Id: I2bba2cfefa884638b6d4d6f32ce7d016cbba0e28
Reviewed-on: https://code.wireshark.org/review/34224
Petri-Dish: Anders Broman <a.broman58@gmail.com>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Change all wireshark.org URLs to use https.
Fix some broken links while we're at it.
Change-Id: I161bf8eeca43b8027605acea666032da86f5ea1c
Reviewed-on: https://code.wireshark.org/review/34089
Reviewed-by: Guy Harris <guy@alum.mit.edu>
Handling of preferences is often done in the dissector handoff
registration. Therefore this function is often registered as
callback while registering preference handling for the module.
In this way the preferences are processed both when registering
the dissector and when changes happen.
Some dissectors opt to register a seperate callback function to
be called when preferences change. Now these have to be called
from the dissector handoff function explicitly, in order to have
the preferences processed during dissector registration.
This becomes explicitly apparent when the port registration comes
into play. With the migration to using dissector registration on
ports with preference this port (range) is often retrieved from
the preferences to match against the ports in a packet to determine
an incoming or outgoing packet of a server. In case the callback
function is not called from the dissector registration this
determination fails, until the preferences are applied/changed,
causing the preference handling callback to be called.
This change add the calling of the callback during dissector
registration, fixing some dissector port registrations in the
process.
Change-Id: Ieaea7f63f8f9062c56582a042a3a5a862e286406
Signed-off-by: Jaap Keuter <jaap.keuter@xs4all.nl>
Reviewed-on: https://code.wireshark.org/review/30848
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Avoid passing an uninitialized "raw" pointer to "proto_tree_add_item".
Change-Id: Ic7d3542b4aa5650f452aad7ac3b783769cb44eb7
Reviewed-on: https://code.wireshark.org/review/28922
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Tested-by: Petri Dish Buildbot
Reviewed-by: Peter Wu <peter@lekensteyn.nl>
This allows a better check of the required version.
Change-Id: I6c4aab67c73434aff4ad744caa2d0add9ec6225c
Reviewed-on: https://code.wireshark.org/review/26889
Petri-Dish: Alexis La Goutte <alexis.lagoutte@gmail.com>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Using GCC version 7.1.1.
Change-Id: I7447a48fc97efb1eb15a016a29165f69d37f40a6
Reviewed-on: https://code.wireshark.org/review/23399
Petri-Dish: João Valverde <j@v6e.pt>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: João Valverde <j@v6e.pt>
But the lz4 frame API was introduced in [r123](https://github.com/lz4/lz4/blob/r123/lz4frame.h).
Put the code behind a lz4 version check.
Change-Id: If8a7e5f7f2e7c9beac89eec334368a5b95a1bc3f
Reviewed-on: https://code.wireshark.org/review/21684
Petri-Dish: Pascal Quantin <pascal.quantin@gmail.com>
Reviewed-by: Pascal Quantin <pascal.quantin@gmail.com>
LZ4F_decompress accepts a NULL parameter to apply default parameters, do
not bother setting a parameter (this also avoids
-Wmissing-field-initializers warnings from clang).
Change-Id: Id2cd8d58cd9339dd5af395092a8104b559b6e194
Reviewed-on: https://code.wireshark.org/review/21675
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Reviewed-by: Guy Harris <guy@alum.mit.edu>
At least some compilers will warn about not initializing the reserved
values.
Change-Id: Id907468be757a1cd3137140d285803f3e9f1724a
Reviewed-on: https://code.wireshark.org/review/21674
Reviewed-by: Guy Harris <guy@alum.mit.edu>
This is mostly to address memory leaks in range preferences (the biggest
user of range functionality) on shutdown.
Now range preferences must use epan scoped memory when referencing
internal preference structures to keep consistency.
Change-Id: Idc644f59b5b42fa1d46891542b53ff13ea754157
Reviewed-on: https://code.wireshark.org/review/19387
Petri-Dish: Michael Mann <mmann78@netscape.net>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Peter Wu <peter@lekensteyn.nl>
Reviewed-by: Michael Mann <mmann78@netscape.net>
Resolves truncation warnings on the x86 clang build
Change-Id: I14ebbe39b8235bd1b909c488c0402b77deb6dde1
Reviewed-on: https://code.wireshark.org/review/19354
Petri-Dish: Michael Mann <mmann78@netscape.net>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Michael Mann <mmann78@netscape.net>
- support rest of api keys
- dissect kafka.required_acks with constants
- dissect kafka.message_timestamp_type
- add expert info about missing request
Change-Id: I3d18936adac6702a61f545385bdec1b75b564bd9
Reviewed-on: https://code.wireshark.org/review/18954
Petri-Dish: Alexis La Goutte <alexis.lagoutte@gmail.com>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Martin Mathieson <martin.r.mathieson@googlemail.com>
This allows dissectors to not need to know about the internal preference structure.
Change-Id: I1ae67248cd0b0132aefc225ea0a9befaf9afdde2
Reviewed-on: https://code.wireshark.org/review/18864
Petri-Dish: Michael Mann <mmann78@netscape.net>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Michael Mann <mmann78@netscape.net>
Details:
- update supported api keys
- add api key ApiVersions
- change api key names according to documentation
- add pcapng files for supported api keys
- add new documentation link
- add declaration of lz4 message codec
Change-Id: I943dc31144890dcd3dd333981a86754668c2bec4
Reviewed-on: https://code.wireshark.org/review/18861
Reviewed-by: Martin Mathieson <martin.r.mathieson@googlemail.com>
This patch introduces new APIs to allow dissectors to have a preference for
a (TCP) port, but the underlying data is actually part of Decode As functionality.
For now the APIs are intentionally separate from the regular APIs that register a
dissector within a dissector table. It may be possible to eventually combine the
two so that all dissectors that register with a dissector table have an opportunity
to "automatically" have a preference to adjust the "table value" through the
preferences dialog.
The tcp.port dissector table was used as the guinea pig. This will eventually be
expanded to other dissector tables as well (most notably UDP ports). Some
dissectors that "shared" a TCP/UDP port preference were also converted. It also
removed the need for some preference callback functions (mostly when the callback
function was the proto_reg_handoff function) so there is cleanup around that.
Dissectors that has a port preference whose default was 0 were switched to using
the dissector_add_for_decode_as_with_preference API rather than dissector_add_uint_with_preference
Also added comments for TCP ports used that aren't IANA registered.
Change-Id: I99604f95d426ad345f4b494598d94178b886eb67
Reviewed-on: https://code.wireshark.org/review/17724
Reviewed-by: Michael Mann <mmann78@netscape.net>
Gerald Combs