This parameter was introduced as a safeguard for bugs
that generate an unbounded string but its utility for
that purpose is doubtful and the way it is being used
creates problems with invalid truncation of UTF-8
strings.
Rename wmem_strbuf_sized_new() with a better name.
Only dissectors are using this function and there is no use case,
as far as I know, that requires its use. Any limitation of length
is imposed transparently by the UI backend.
This function is problematic because it is not Unicode aware and
will truncate a string on an arbitrary byte boundary for multibyte
strings.
Replace its use with a normal strbuf without a length limite and
remove the function because it is not useful and the ITEM_LABEL_LENGTH
parameter does not belong in wmem anyway.
If a character is not a valid Unicode codepoint, i.e. one of
the code points reserved for surrogate pairs or a code point
above 0x10FFFF, don't add it to a wmem_strbuf when converting
from other encodings but add a replacement character instead, by
using a new wmem_strbuf_append_unichar_validated() function.
Now we produce valid UTF-8 in various situations where UCS-2 or UTF-32
can encode unpaired surrogate codepoints. Consolidate some related
checks that are now redundant.
Also add a replacement character to the end of invalid UCS-2 strings
with an odd number of bytes, as done with UTF-16 and UTF-32.
Fix#18508
Add UNICODE_REPLACEMENT_CHARACTER as a #define for the Unicode
REPLACEMENT CHARACTER code point (0x00FFFD), and use that instead of
0xfffd/0xFFFD/0x00FFFD in cases where that value refers to REPLACEMENT
CHARACTER.
Use length and size consistently. strbuf->len does not
include the terminating nul. strbuf->alloc_len includes
the terminating nul.
Use consistent language and use "length" to mean size without
nul byte and "size" to mean size with all bytes, including nul.
Some older dissectors that predate Unicode and parse text protocols
are prone to generate invalid UTF-8 strings. This is a bug and can have
safety implications.
For example passing invalid UTF-8 to proto_tree_add_string() is a
common bug. There are safeguards in format_text() but this should
not be relied on as a general solution to the problem.
For one, as the name implies, it is only used with representation of a
field value, which is not the same as the value itself of an FT_STRING field.
Issue #18317 shows another reason why.
For now this compile flag only enables extra checks for string ftypes,
which covers a subset of proto.h APIs including
proto_tree_append_string(). Later is should be extended to other
interfaces.
This is also not expected to be disabled for release builds because
there are still many dissectors that do not correctly handle strings.
More work is needed to 1) identify them and 2) fix them.
Ping #18317
We have two format_size()s, with and without wmem scoped memory.
Move the wmem version to wsutil and add a convenience macro to
use g_malloc()ed memory.
This allows wmem to be used from other libraries, namely wsutil.
It is often the case that a funtion exists in wsutil and cannot
be used with a wmem scope, requiring some code duplication or
extra memory allocations, or vice-versa, code in epan cannot be
moved to wsutil because it has a wmem dependency.
To this end wmem is moved to wsutil. Scope management remains part
of epan because those scope semantics are specific to dissection.
João Valverde