Add fragment_add_check_with_fallback() and use it in USBLL dissector
instead of fragment_add_check() to avoid last fragment retransmissions
from being treated as separate transfers. With this change, the last
fragment retransmissions are correctly grouped together with the rest
of the transfer.
Only skip single fragment reassembly if retransmission is not possible
at the protocol level, i.e. for SETUP DATA0 (when it is not merged with
OUT data) and for isochronous transfers. The reassembly must not be
skipped for other transfers (especially for full-speed bulk) because
otherwise it wouldn't be possible to group retransmissions together with
the first data packet.
Do not use DATA0/DATA1 tracking for isochronous transfers. Isochronous
data cannot be retransmitted because there are no handshakes (there is
no ACK nor NAK after isochronous data packets).
Store pointer to first gap to reduce number of full list traversals
needed when linking new fragments. When all captured fragments are in
order, the first gap is effectively pointing to list tail. The best case
scenario, where the list traversals are completely eliminated, happens
every time for protocols that always have the fragments ordered (most
notably USBLL Full-Speed capture containing Bulk OUT transfers with
a lot of retransmissions).
The memory usage is increased by a single pointer and 32-bit contiguous
length counter per fragment head. The additional CPU usage is constant
per insertion, i.e. does not increase with the number of fragments in
the list.
Fixes#17311
Separate fragment_head and fragment_item into two
different types of structs.
Remove "offset" from fragment_head, which was unused,
making fragment heads 4 bytes smaller.
Remove fragment_nr_offset, datalen, reassembled_in,
reas_in_layer_num, and error from fragment_item,
making them 24 bytes smaller.
Change a few dissectors which were using fragment_head
and fragment_item indistinguishably.
Ping #17311
Because completed reassemblies are hashed in the reassembled_table for
all the frame numbers that contributed fragments,
fragment_get_reassembled_id() works wherever fragment_get_reassembled()
does, and also works where the fragment id is not the frame number.
However, since the reassembled_table hash key only depends on the
fragment id and the frame number, it only allows a frame to have
one reassembly with a given fragment id. Some protocols can have
more than one reassembly with a given fragment id (that differ on
addresses or other keys), such as GSM SMS, and the wrong reassembly
is retrieved on the second pass in those cases.
For this reason, we might want to add additional key elements to
reassembled_table, such as layer number. fragment_get_reassembled_id
already takes packet_info as a parameter and can accommodate that
without further changes, but fragment_get_reassembled cannot, so
remove the latter in favor of the former.
When processing segments out of order in TCP, it is possible to
get new segments that fill a sequence gap and be able to dissect
at least one PDU but need more data for additional PDUs (that have
data from the contiguous stream bytes.) We can only determine this
after passing the reassembled segments to the subdissector first.
To keep dissection and layer numbers consistent between passes,
split the multisegment PDU, keeping the already dissect PDU(s) in
the current reassembly and creating a new MSP for the parts not yet
dissected.
Update the dissection test to enable the currently skipped test that
require MSP splitting and remove test_tcp_out_of_order_twopass_with_bug
Repeated words were found with:
egrep "(\b[a-zA-Z]+) +\1\b" . -Ir
and then manually reviewed.
Non-displayed strings (e.g., in comments)
were also corrected, to ease future review.
Currently out-of-order segments will result in cutting a stream into
two pieces while the out-of-order segment itself is ignored. For
example, a stream of segments "ABDCE" is interpreted as "AB", "DE" with
"C" ignored. This behavior breaks TLS decryption or prevent application
layer PDUs (such as HTTP requests/responses) from being reconstructed.
To fix this, buffer segments when a gap is detected.
The proposed approach extends the "multi-segment PDU" (MSP) mechanism
which is normally used for linking multiple, sequential TCP segments
into a single PDU. When a gap is detected between segments, it is
assumed that the segments within this gap are out-of-order and will be
received (or retransmitted) later.
The current implementation has a limitation though, if multiple gaps
exist, then the subdissector will only be called when all gaps are
filled (the subdissector will receive segments later than necessary).
For example with "ACEBD", "ABC" can already be processed after "B" is
received (with "E" still buffered), but due to how MSP are extended, it
must receive "D" too before it reassembles "ABCDE". In practice this
could mean that the request/response times between HTTP requests and
responses are slightly off, but at least the stream is correct now.
(These limitations are documented in the User's Guide.)
As the feature fails at least the 802.11 decryption test where packets
are missing (instead of OoO), hide this feature behind a preference.
Tested with captures containing out-of-order TCP segments from the
linked bug reports, comparing the effect of toggling the preference on
the summary output of tshark, the verbose output (-V) and the two-pass
output (-2 or -2V). Captures marked with "ok" just needed "simple"
out-of-order handling. Captures marked with "ok2" additionally required
the reassembly API change to set the correct reassembled length.
This change does "regress" on bug 10289 though when the preference is
enabled as retransmitted single-segment PDUs are now passed to
subdissectors. I added a TODO comment for this unrelated cosmetic issue.
Bug: 3389 # capture 2907 (HTTP) ok
Bug: 4727 # capture 4590 (HTTP) ok
Bug: 9461 # capture 12130 (TLS/HTTP/RPC-over-HTTP +key 12131) ok
Bug: 12006 # capture 14236 (HTTP) ok2; capture 15261 (HTTP) ok
Bug: 13517 # capture 15370 (HTTP) ok; capture 16059 (MQ) ok
Bug: 13754 # capture 15593 (MySQL) ok2
Bug: 14649 # capture 16305 (WebSocket) ok
Change-Id: If3938c5c1c96db8f7f50e39ea779f623ce657d56
Reviewed-on: https://code.wireshark.org/review/27943
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Tested-by: Petri Dish Buildbot
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Register all reassembly tables with a central unit, allowing the
central unit to have the callback that initializes and destroys
the reassembly tables, rather than have dissectors do it individually.
Change-Id: Ic92619c06fb5ba6f1c3012f613cae14982e101d4
Reviewed-on: https://code.wireshark.org/review/19834
Petri-Dish: Michael Mann <mmann78@netscape.net>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Michael Mann <mmann78@netscape.net>
Add support for defragmentation of fragments that use the defragmentation
scheme of PPP MP (RFC 1990). Instead of getting "sequence_number,
fragment_number, last" as in other protocols, PPP MP provides a single
sequence number that is effectively "seqnum + fragnum", though it provides
flags for both the first and last fragment of a reassembly.
See Appendix A of RFC 4623 (PWE3 Fragmentation and Reassembly) for a list
of protocols that use this style, including PPP MP (RFC 1990), PWE3 MPLS
(RFC 4385), L2TPv2 (RFC 2661), L2TPv3 (RFC 3931), ATM, and Frame Relay.
Also add support for the Multi-class Extension to Multilink PPP (RFC 2686),
which uses some of the previously reserved bits as classes that distinguish
otherwise identical sequence numbers.
Bug: 12548
Change-Id: Ic2ce3c50e61ab2eb50e4d92fd353ca4d2a48fe18
Reviewed-on: https://code.wireshark.org/review/16327
Reviewed-by: Michael Mann <mmann78@netscape.net>
Petri-Dish: Michael Mann <mmann78@netscape.net>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Anders Broman <a.broman58@gmail.com>
Do not try to recover from truncated tvbs for fragment_add_seq-like
functions:
- If it is the first block and the dissector requested frag_data_len
number of bytes, we should not lie and pretend that we are fully
reassembled.
- For other blocks, returning NULL as no reassembly was possible makes
sense. But other fragments in the list should not be cleared as there
may be partial fragments which were returned before.
It seems that this special behavior was introduced in
b2c11b5e13 (freeing fragments and
returning NULL as an optimization when fragments are deemed not needed
anymore) and faeb2c2ee1 (for returning
fd_head for the first fragment, "so the first fragment gets dissected as
fragmented packet").
Now in theory unused fragments could stick around, but that also
possible with the normal fragment_add functions.
Bug: 11799
Change-Id: I20829c54e1b2eee25a91fe4de51b19b1458c7789
Reviewed-on: https://code.wireshark.org/review/14082
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Peter Wu <peter@lekensteyn.nl>
Reviewed-by: Michael Mann <mmann78@netscape.net>
Documentation changes only (comments and docbook).
Update WSDG with the fragment_add_seq_check API that was introduced in
Wireshark 1.10.
Fix typos and clarify the many functions we have for adding reassembling
fragments.
Change-Id: I38715a8f58e9cf1fe3e34ee4b1a4ae339630282b
Reviewed-on: https://code.wireshark.org/review/14066
Petri-Dish: Peter Wu <peter@lekensteyn.nl>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Anders Broman <a.broman58@gmail.com>
frame
but in different SCTP DATA chunks, whitout the patch the message is
reassembled in both chunks leading to duplicated upper layer PDU:s in the
frame.
Change-Id: Ie31142c38c728018178947544b571622447d8e8f
Reviewed-on: https://code.wireshark.org/review/7716
Reviewed-by: Anders Broman <a.broman58@gmail.com>
(Using sed : sed -i '/^ \* \$Id\$/,+1 d')
Fix manually some typo (in export_object_dicom.c and crc16-plain.c)
Change-Id: I4c1ae68d1c4afeace8cb195b53c715cf9e1227a8
Reviewed-on: https://code.wireshark.org/review/497
Reviewed-by: Anders Broman <a.broman58@gmail.com>
revisit this reassembly (in a multi-pass program such as Wireshark, or
TShark with -2), we'll throw the same error.
In fragment_set_tot_len(), allow the length to be set to a value that's
before the offset of existing fragments; we'll catch that later when the
reassembly completes. This lets us handle some problems with DTLS less
confusingly.
When adding frames to an already-completed reassembly, check for
fragments that overlap existing fragments or go past the end of the
reassembly, and report errors.
When completing a reassembly, make the buffer for the reassembled data
big enough to contain the specified data length for the reassembly, even
if that's less than the offset + length of the last fragment. Flag all
fragments that go past that length as "too long", and only copy out what
part of them fits, if any. That lets us flag the correct fragment or
fragments as being "too long".
When adding fragments, do some additional checks, even if we're not
doing the first pass through the packets, so errors that show up in the
first pass also show up on subsequent passes.
svn path=/trunk/; revision=48909
be done on flows from one address to another; reassembly for protocols
running atop TCP should be done on flows from one TCP endpoint to
another.
We do this by:
adding "reassembly table" as a data structure;
associating hash tables for both in-progress reassemblies and
completed reassemblies with that data structure (currently, not
all reassemblies use the latter; they might keep completed
reassemblies in the first table);
having functions to create and destroy keys in that table;
offering standard routines for doing address-based and
address-and-port-based flow processing, so that dissectors not
needing their own specialized flow processing can just use them.
This fixes some mis-reassemblies of NIS YPSERV YPALL responses (where
the second YPALL response is processed as if it were a continuation of
a previous response between different endpoints, even though said
response is already reassembled), and also allows the DCE RPC-specific
stuff to be moved out of epan/reassembly.c into the DCE RPC dissector.
svn path=/trunk/; revision=48491
The reassembled fragments tree in the Packet Details view is awesome, but it
lacks one thing: a field that exposes the reassembled data.
tcp.data already exists for exposing a single TCP segment's payload as a byte
array. It would be handy to have something similar for a single application
layer PDU when TCP segment reassembly is involved. I propose
tcp.reassembled.data, named and placed after the already existing field
tcp.reassembled.length.
My primary use case for this feature is outputting tcp.reassembled.data with
tshark for further processing with a script.
The attached patch implements this very feature. Because the reassembled
fragment tree code is general purpose, i.e. not specific to just TCP, any
dissector that relies upon it can add a similar field very cheaply. In that
vein I've also implemented ip.reassembled.data and ipv6.reassembled.data, which
expose reassembled fragment data as a single byte stream for IPv4 and IPv6,
respectively. All other protocols that use the reassembly code have been left
alone, other than inserting NULL into their initializer lists for the newly
introduced struct field reassemble.h:fragment_items.hf_reassembled_data.
svn path=/trunk/; revision=44802
in README.devloper. Remove g_gnuc.h since it's no longer needed. Remove
tvbuff_init(), tvbuff_cleanup(), reassemble_init(), and
reassemble_cleanup() since they were only used for older GLib versions
which didn't support GSlices. Assume we always support the "matches"
operator.
svn path=/trunk/; revision=37978
compiling again.
fragment_add_seq_check(), fragment_add_seq_802_11(), and fragment_add_seq_next()
all call fragment_add_seq_check_work() so make their prototypes match each other
in const-ness. This fixes a warning when compiling reassemble_test.
svn path=/trunk/; revision=32933
https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=4422
From me: Fix a number of instances where the function prototype or
the function definition wasn't changed so there was a mismatch
thus causing Windows (but not gcc) compilation errors.
svn path=/trunk/; revision=32365
01_reassemble_test.patch
------------------------
I didn't want to do anything without some unit tests, so here they are.
This allows a standalone binary, epan/reassemble_test, to be built; this can be run from the commandline and should end up printing out "success"
if all goes well.
NOTE the changes to makefile.am NOT checked in currently.
Incidentally: is it possible to get the buildbot to run things like this, exntest and tvbtest?
02_reassemble_refactor.patch
----------------------------
fragment_add_seq, fragment_add_dcerpc_dg and fragment_add_seq_check_work were all pretty much carbon-copies of each other. This patch factors out the common parts of the routines into a new routine, fragment_add_seq_key().
03_reassemble_partial_reassembly.patch
---------------------------------------
This makes fragment_set_partial_reassembly() work for datagrams assembled with fragment_add_seq(). The patch itself is actually quite small, but it adds another unit test which is reasonably lengthy.
svn path=/trunk/; revision=20888
having been reassembled.
Fix the comments in reassembly.c and reassembly.h regarding what the reassembly
routines actually return in the 802.11 and no-sequence-number cases when they
are given the first and last packet (that is, a non-segmented packet): in
particular the routines return a pointer to a list containing just the one
fragment.
svn path=/trunk/; revision=20505
add a check to fragment_add_common() if the given tvb parameters are ok, otherwise throw a DissectorError
add some more symbols to libethereal.def
svn path=/trunk/; revision=17073
in addition, I had to implement fragment_get_reassembled() in addition to fragment_get(), which works with reassembled_table
svn path=/trunk/; revision=15762
as connection oriented (cn) and connectionless (dg) DCE/RPC uses different ways to handle defragmentation and this function is only used for dg
svn path=/trunk/; revision=15757
Tomasz Moń