We want to do more sophisticated processing of UTF-8 in wmem and
for that we want to use the unicode utility functions in wsutil.
We also want to use wmem scoped memory in wsutil unicode utility
functions.
This introduces a circular dependency. Fix that by making both
the same library and removing the sanitary cordon separating
them.
We still need to be mindful of public header depencies of wmem on
wsutil because wmem.h is included in wireshark.h and we want to
be parsimonious with the use of global includes.
Develpment headers are a sizeable part of the binary installation
and most users won't ever require them. It's recommended to package
them separately in a devel package or SDK.
Create a CMake installation component for development headers
and add the EXCLUDE_FROM_ALL property.
Headers can be installed using the invocation:
cmake --install <dir> --component Development
Originally WS_DISABLE_DEBUG was chosen to be
similar to G_DISABLE_ASSERT and NDEBUG.
However generator expressions are essential for modern CMake
but the syntax is weird and having to use negations makes it
ten-fold worse.
Remove the negation. Instead of changing the CMake variable
reverse the macro definition for WS_DISABLE_DEBUG.
The $<CONFIG:cgs> generator expression with multiple config arguments
requires CMake >= 3.19 so we can't use that yet for a further
syntactical simplification.
This parameter was introduced as a safeguard for bugs
that generate an unbounded string but its utility for
that purpose is doubtful and the way it is being used
creates problems with invalid truncation of UTF-8
strings.
Rename wmem_strbuf_sized_new() with a better name.
Only dissectors are using this function and there is no use case,
as far as I know, that requires its use. Any limitation of length
is imposed transparently by the UI backend.
This function is problematic because it is not Unicode aware and
will truncate a string on an arbitrary byte boundary for multibyte
strings.
Replace its use with a normal strbuf without a length limite and
remove the function because it is not useful and the ITEM_LABEL_LENGTH
parameter does not belong in wmem anyway.
If a character is not a valid Unicode codepoint, i.e. one of
the code points reserved for surrogate pairs or a code point
above 0x10FFFF, don't add it to a wmem_strbuf when converting
from other encodings but add a replacement character instead, by
using a new wmem_strbuf_append_unichar_validated() function.
Now we produce valid UTF-8 in various situations where UCS-2 or UTF-32
can encode unpaired surrogate codepoints. Consolidate some related
checks that are now redundant.
Also add a replacement character to the end of invalid UCS-2 strings
with an odd number of bytes, as done with UTF-16 and UTF-32.
Fix#18508
Add UNICODE_REPLACEMENT_CHARACTER as a #define for the Unicode
REPLACEMENT CHARACTER code point (0x00FFFD), and use that instead of
0xfffd/0xFFFD/0x00FFFD in cases where that value refers to REPLACEMENT
CHARACTER.
wmem_strbuf_grow should set the correct size with regard to max_size,
if set. In any case passing the actual free "raw" size to g_strlcpy is
always the correct thing to do.
Use length and size consistently. strbuf->len does not
include the terminating nul. strbuf->alloc_len includes
the terminating nul.
Use consistent language and use "length" to mean size without
nul byte and "size" to mean size with all bytes, including nul.
Some older dissectors that predate Unicode and parse text protocols
are prone to generate invalid UTF-8 strings. This is a bug and can have
safety implications.
For example passing invalid UTF-8 to proto_tree_add_string() is a
common bug. There are safeguards in format_text() but this should
not be relied on as a general solution to the problem.
For one, as the name implies, it is only used with representation of a
field value, which is not the same as the value itself of an FT_STRING field.
Issue #18317 shows another reason why.
For now this compile flag only enables extra checks for string ftypes,
which covers a subset of proto.h APIs including
proto_tree_append_string(). Later is should be extended to other
interfaces.
This is also not expected to be disabled for release builds because
there are still many dissectors that do not correctly handle strings.
More work is needed to 1) identify them and 2) fix them.
Ping #18317
Like wmem_map_remove(), this frees the key/value pair item
in the map but not the key or the value itself (which may
in fact be the same object.) Not generally a problem, as
they'll get freed by the pool. (If someone wants to manage
memory themselves, they should probably be using a GHashTable.)
Add some C99 stdio.h numbers to compare with GLib on platforms
(such as Windows) where they use different implementations.
Add a wmem string test with NULL allocator, to compare wmem and GLib
performance with roughly the same memory allocation.
Use the block allocator as being more representative of normal
wmem performance, instead of using strict, that is normally
used for wmem debugging.
These are not pass/fail tests, so the automation cannot
validate them. They just slow down the CI builds. To
enable pass -m perf.
I think the --verbose comment is wrong, I did not detect
any difference in output with or without --verbose.
Because we already have the length of the output string after
calling vsnprintf(), we should avoid calling wmem_strdup(), which
will ignore that and recompute the length.
Increase the buffer size to a value that seems reasonable to
minimize the chance of a second call to vsnprintf().
Add @file markers for remaining non-dissector
files that contain functions exported with
WS_DLL_PUBLIC so that Doxygen will
generate documentation for them.
A number of protocols have IDs that can be reused that are used as
lookup keys. In most cases the frame number should be used as well
to differentiate repeat appearances of an ID. For response/request
matching, it is frequently useful to find the most recent frame number
(greatest value less than or equal to the current one) that contained
an ID.
We can achieve that by using a multimap that stores values with a given
ID in a tree keyed with the frame number. This works better than using
a map or a tree alone:
1) A map isn't ordered, so doesn't allow for less than or equal comparison.
2) Using a tree requires an ordering on all the ID components, and then
having to test all the components other than the frame number separately
for equality after retrieval.
Currently the multimap does not support inserting items without specifying
the tree key (and having the multimap generate a key), because the total
capacity of trees (including deleted nodes) is not tracked. If other use
cases are needed, this could be added later along with more generic
multimap support.
Use a multimap in ANSI MAP, ANSI TCAP, and GSM SMS, all of which need to
match lookup IDs that can be reused. Fix#7653.
Change our developer.gnome.org/glib URLs to
developer-old.gnome.org/glib. The official documentation for GLib
appears to be at https://docs.gtk.org/glib/, but it has a different
layout than the gnome.org content (and is surprisingly resistant to
exploration IMHO). We can switch to developer-old.gnome.org using a
simple substitution and it still seems to be updated, so do that for
now.
This should fix the cppcheck warning "The unsigned expression
'sizeof(struct _PKT_INFO)' will never be negative so it is either
pointless or an error to check if it is."
wmem_safe_mult() was only used to do an overflow-safe multiplication of
a type size and a count of elements of that type; replace it with
wmem_safe_mult_type_size(), which takes the type as the first argument,
and checks only whether the count of elements is <= 0.
We have two format_size()s, with and without wmem scoped memory.
Move the wmem version to wsutil and add a convenience macro to
use g_malloc()ed memory.
This allows wmem to be used from other libraries, namely wsutil.
It is often the case that a funtion exists in wsutil and cannot
be used with a wmem scope, requiring some code duplication or
extra memory allocations, or vice-versa, code in epan cannot be
moved to wsutil because it has a wmem dependency.
To this end wmem is moved to wsutil. Scope management remains part
of epan because those scope semantics are specific to dissection.
João Valverde