2012-10-30 01:24:16 +00:00
|
|
|
$Id$
|
|
|
|
|
|
|
|
1. Introduction
|
|
|
|
|
2012-12-08 21:57:05 +00:00
|
|
|
NB: Wmem still does not provide all of the functionality of emem
|
|
|
|
(see README.malloc), although it should provide most of it. New code
|
|
|
|
may still need to use emem for the time being.
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
The 'emem' memory manager (described in README.malloc) has been a part of
|
|
|
|
Wireshark since 2005 and has served us well, but is starting to show its age.
|
|
|
|
The framework has become increasingly difficult to maintain, and limitations
|
2012-11-10 14:43:27 +00:00
|
|
|
in the API have blocked progress on other long-term goals such as multi-
|
|
|
|
threading, and opening multiple files at once.
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
The 'wmem' memory manager is an attempt to write a new memory management
|
|
|
|
framework to replace emem. It provides a significantly updated API, a more
|
|
|
|
modular design, and it isn't all jammed into one 2500-line file.
|
|
|
|
|
|
|
|
Wmem was originally conceived in this email to the wireshark-dev mailing list:
|
|
|
|
https://www.wireshark.org/lists/wireshark-dev/201210/msg00178.html
|
|
|
|
|
|
|
|
The wmem code can now be found in epan/wmem/ in the Wireshark source tree.
|
|
|
|
|
|
|
|
2. Usage for Consumers
|
|
|
|
|
|
|
|
If you're writing a dissector, or other "userspace" code, then using wmem
|
|
|
|
should be very similar to using emem. All you need to do is include the header
|
|
|
|
(epan/wmem/wmem.h) and get a handle to a memory pool (if you want to *create*
|
|
|
|
a memory pool, see the section "3. Usage for Producers" below).
|
|
|
|
|
|
|
|
A memory pool is an opaque pointer to an object of type wmem_allocator_t, and
|
2012-11-10 14:43:27 +00:00
|
|
|
it is the very first parameter passed to almost every call you make to wmem.
|
|
|
|
Other than that parameter (and the fact that functions are prefixed wmem_
|
|
|
|
instead of ep_ or se_) usage is exactly like that of emem. For example:
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
wmem_alloc(myPool, 20);
|
|
|
|
|
|
|
|
allocates 20 bytes in the pool pointed to by myPool.
|
|
|
|
|
2012-11-03 17:58:07 +00:00
|
|
|
2.1 Available Pools
|
2012-10-30 01:24:16 +00:00
|
|
|
|
2012-12-02 17:16:09 +00:00
|
|
|
2.1.1 (Sort Of) Global Pools
|
|
|
|
|
2012-11-03 17:58:07 +00:00
|
|
|
Dissectors that include the wmem header file will have three pools available
|
2012-12-02 17:16:09 +00:00
|
|
|
to them automatically: wmem_packet_scope(), wmem_file_scope() and
|
|
|
|
wmem_epan_scope();
|
2012-11-03 17:58:07 +00:00
|
|
|
|
2012-12-02 17:16:09 +00:00
|
|
|
The packet pool is scoped to the dissection of each packet, replacing
|
2012-11-03 17:58:07 +00:00
|
|
|
emem's ep_ allocators. The file pool is scoped to the dissection of each file,
|
|
|
|
replacing emem's se_ allocators. For example:
|
2012-10-30 01:24:16 +00:00
|
|
|
|
2012-11-10 14:43:27 +00:00
|
|
|
ep_malloc(32);
|
|
|
|
se_malloc(sizeof(guint));
|
2012-10-30 01:24:16 +00:00
|
|
|
|
2012-11-03 17:58:07 +00:00
|
|
|
could be replaced with
|
|
|
|
|
2012-11-10 14:43:27 +00:00
|
|
|
wmem_alloc(wmem_packet_scope(), 32);
|
|
|
|
wmem_alloc(wmem_file_scope(), sizeof(guint));
|
2012-11-03 17:58:07 +00:00
|
|
|
|
2012-11-09 00:10:21 +00:00
|
|
|
NB: Using these pools outside of the appropriate scope (eg using the packet
|
2012-11-03 17:58:07 +00:00
|
|
|
pool when there isn't a packet being dissected) will throw an assertion.
|
|
|
|
See the comment in epan/wmem/wmem_scopes.c for details.
|
|
|
|
|
2012-12-02 17:16:09 +00:00
|
|
|
The epan pool is scoped to the library's lifetime - memory allocated in it is
|
|
|
|
not freed until epan_cleanup() is called, which is typically at the end of the
|
|
|
|
program.
|
|
|
|
|
|
|
|
2.1.2 Pinfo Pool
|
|
|
|
|
|
|
|
Certain places (such as AT_STRINGZ address allocations) need their memory to
|
|
|
|
stay around a little longer than the usual packet scope - basically until the
|
|
|
|
next packet is dissected. This is effectively the scope of Wireshark's pinfo
|
|
|
|
structure, so the pinfo struct has a 'pool' member which is a wmem pool scoped
|
|
|
|
to the lifetime of the pinfo struct.
|
|
|
|
|
2012-11-03 17:58:07 +00:00
|
|
|
2.2 Core API
|
|
|
|
|
|
|
|
- wmem_alloc
|
|
|
|
- wmem_alloc0
|
2013-01-04 22:11:13 +00:00
|
|
|
- wmem_new
|
|
|
|
- wmem_new0
|
2012-11-03 17:58:07 +00:00
|
|
|
|
|
|
|
2.3 String Utilities
|
|
|
|
|
|
|
|
- wmem_strdup
|
|
|
|
- wmem_strndup
|
2012-11-25 14:16:50 +00:00
|
|
|
- wmem_strdup_printf
|
|
|
|
- wmem_strdup_vprintf
|
2012-11-03 17:58:07 +00:00
|
|
|
|
|
|
|
2.4 Stack
|
|
|
|
|
2012-12-18 23:25:11 +00:00
|
|
|
- wmem_stack_new
|
2012-11-03 17:58:07 +00:00
|
|
|
- wmem_stack_push
|
|
|
|
- wmem_stack_pop
|
|
|
|
- wmem_stack_peek
|
|
|
|
- wmem_stack_count
|
|
|
|
|
2012-12-19 01:37:28 +00:00
|
|
|
2.5 Singly-Linked List
|
|
|
|
|
|
|
|
- wmem_slist_new
|
|
|
|
- wmem_slist_prepend
|
|
|
|
- wmem_slist_remove
|
|
|
|
- wmem_slist_front
|
|
|
|
- wmem_slist_frame_next
|
|
|
|
- wmem_slist_frame_data
|
|
|
|
- wmem_slist_count
|
|
|
|
|
|
|
|
2.6 Slab
|
2012-10-30 01:24:16 +00:00
|
|
|
|
2012-12-18 23:25:11 +00:00
|
|
|
- wmem_slab_new
|
2012-11-03 17:58:07 +00:00
|
|
|
- wmem_slab_alloc
|
|
|
|
- wmem_slab_free
|
2012-10-30 01:24:16 +00:00
|
|
|
|
2012-12-19 01:37:28 +00:00
|
|
|
2.7 String-Buffers
|
|
|
|
|
|
|
|
- wmem_strbuf_new
|
|
|
|
- wmem_strbuf_sized_new
|
2012-12-19 23:12:58 +00:00
|
|
|
- wmem_strbuf_append
|
2012-12-29 20:41:12 +00:00
|
|
|
- wmem_strbuf_append_printf
|
2012-12-19 01:37:28 +00:00
|
|
|
- wmem_strbuf_get_str
|
|
|
|
- wmem_strbuf_get_len
|
|
|
|
|
2012-10-30 01:24:16 +00:00
|
|
|
3. Usage for Producers
|
|
|
|
|
|
|
|
NB: If you're just writing a dissector, you probably don't need to read
|
|
|
|
this section.
|
|
|
|
|
|
|
|
One of the problems with the old emem framework was that there were basically
|
|
|
|
two allocator backends (glib and mmap) that were all mixed together in a mess
|
2012-11-10 14:43:27 +00:00
|
|
|
of if statements, environment variables and #ifdefs. In wmem the different
|
|
|
|
allocator backends are cleanly separated out, and it's up to the owner of the
|
|
|
|
pool to pick one.
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
3.1 Available Allocator Back-Ends
|
|
|
|
|
2012-12-19 00:43:36 +00:00
|
|
|
Each available allocator type has a corresponding entry in the
|
|
|
|
wmem_allocator_type_t enumeration defined in wmem_core.h.
|
|
|
|
|
2012-10-30 01:24:16 +00:00
|
|
|
The currently available allocators are:
|
2012-12-19 00:43:36 +00:00
|
|
|
- WMEM_ALLOCATOR_SIMPLE (wmem_allocator_simple.*)
|
|
|
|
A trivial allocator that g_allocs requested memory and tracks
|
|
|
|
allocations via a simple linked list.
|
|
|
|
- WMEM_ALLOCATOR_BLOCK (wmem_allocator_block.*)
|
|
|
|
A block allocator that grabs large chunks of memory at a time
|
2012-11-25 13:58:06 +00:00
|
|
|
(8 MB currently) and serves allocations out of those chunks.
|
2012-12-28 17:09:07 +00:00
|
|
|
- WMEM_ALLOCATOR_STRICT (wmem_allocator_strict.*)
|
|
|
|
An allocator that does its best to find invalid memory usage via
|
|
|
|
things like canaries and scrubbing freed memory. Valgrind is the
|
|
|
|
better choice on platforms that support it.
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
3.2 Creating a Pool
|
|
|
|
|
2012-12-19 00:43:36 +00:00
|
|
|
To create a pool, include the regular wmem header and call the
|
|
|
|
wmem_allocator_new() function with the appropriate type value.
|
|
|
|
For example:
|
2012-10-30 01:24:16 +00:00
|
|
|
|
2012-12-19 00:43:36 +00:00
|
|
|
#include "wmem/wmem.h"
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
wmem_allocator_t *myPool;
|
2012-12-19 00:43:36 +00:00
|
|
|
myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
From here on in, you don't need to remember which type of allocator you used
|
|
|
|
(although allocator authors are welcome to expose additional allocator-specific
|
|
|
|
helper functions in their headers). The "myPool" variable can be passed around
|
|
|
|
and used as normal in allocation requests as described in section 2 of this
|
|
|
|
document.
|
|
|
|
|
2012-12-27 22:57:05 +00:00
|
|
|
Note that the type of pool you request won't always be the type you get - the
|
|
|
|
WIRESHARK_DEBUG_WMEM_OVERRIDE environment variable (if set) can be used to
|
|
|
|
force all calls to wmem_allocator_new() to returns a specific type for debugging
|
|
|
|
purposes. It will always be safe to call allocator-specific helpers though, they
|
|
|
|
will simply be no-ops if the type doesn't match.
|
|
|
|
|
2012-10-30 01:24:16 +00:00
|
|
|
3.3 Destroying a Pool
|
|
|
|
|
|
|
|
Regardless of which allocator you used to create a pool, it can be destroyed
|
|
|
|
with a call to the function wmem_destroy_allocator(). For example:
|
|
|
|
|
|
|
|
#include "wmem/wmem.h"
|
|
|
|
|
|
|
|
wmem_allocator_t *myPool;
|
|
|
|
|
2012-12-19 00:43:36 +00:00
|
|
|
myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
/* Allocate some memory in myPool ... */
|
|
|
|
|
|
|
|
wmem_destroy_allocator(myPool);
|
|
|
|
|
|
|
|
Destroying a pool will free all the memory allocated in it.
|
|
|
|
|
|
|
|
3.4 Reusing a Pool
|
|
|
|
|
|
|
|
It is possible to free all the memory in a pool without destroying it,
|
2012-11-10 14:43:27 +00:00
|
|
|
allowing it to be reused later. Depending on the type of allocator, doing this
|
|
|
|
(by calling wmem_free_all()) can be significantly cheaper than fully destroying
|
|
|
|
and recreating the pool. This method is therefore recommended, especially when
|
|
|
|
the pool would otherwise be scoped to a single iteration of a loop. For example:
|
2012-10-30 01:24:16 +00:00
|
|
|
|
|
|
|
#include "wmem/wmem.h"
|
|
|
|
|
|
|
|
wmem_allocator_t *myPool;
|
|
|
|
|
2012-12-19 00:43:36 +00:00
|
|
|
myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
|
2012-10-30 01:24:16 +00:00
|
|
|
for (...) {
|
|
|
|
|
|
|
|
/* Allocate some memory in myPool ... */
|
|
|
|
|
2012-11-10 14:43:27 +00:00
|
|
|
/* Free the memory, faster than destroying and recreating
|
|
|
|
the pool each time through the loop. */
|
2012-10-30 01:24:16 +00:00
|
|
|
wmem_free_all(myPool);
|
|
|
|
}
|
|
|
|
wmem_destroy_allocator(myPool);
|
|
|
|
|
|
|
|
4. Internal Design
|
|
|
|
|
2012-11-10 14:43:27 +00:00
|
|
|
Despite being written in Wireshark's standard C90, wmem follows a fairly
|
|
|
|
object-oriented design pattern. Although efficiency is always a concern, the
|
|
|
|
primary goals in writing wmem were maintainability, and preventing memory
|
|
|
|
leaks.
|
|
|
|
|
|
|
|
4.1 struct _wmem_allocator_t
|
|
|
|
|
|
|
|
The heart of wmem is the _wmem_allocator_t structure defined in the
|
|
|
|
wmem_allocator.h header file. This structure uses C function pointers to
|
2013-01-15 18:29:27 +00:00
|
|
|
implement a common object-oriented design pattern known as an interface (also
|
|
|
|
known as an abstract class to those who are more familiar with C++).
|
2012-11-10 14:43:27 +00:00
|
|
|
|
|
|
|
Different allocator implementations can provide exactly the same interface by
|
2013-01-15 18:29:27 +00:00
|
|
|
assigning their own functions to the members of an instance of the structure.
|
|
|
|
The structure currently has five members:
|
2012-11-10 14:43:27 +00:00
|
|
|
|
|
|
|
- alloc()
|
|
|
|
- free_all()
|
|
|
|
- destroy()
|
|
|
|
- private_data
|
2012-12-27 22:38:48 +00:00
|
|
|
- type
|
2012-11-10 14:43:27 +00:00
|
|
|
|
|
|
|
The private_data pointer is a void pointer that the implementation can use to
|
2012-12-27 22:38:48 +00:00
|
|
|
store whatever internal structures it needs. The type field is an enumeration of
|
|
|
|
type wmem_allocator_type_t (see section 3.1) set by the wmem_allocator_new()
|
2013-01-15 18:29:27 +00:00
|
|
|
function. This field must NEVER be written to by the allocator implementation,
|
|
|
|
although it may be read. The three function pointers should be fairly obvious;
|
|
|
|
alloc and free_all are called with pointers to the private_data field, and
|
|
|
|
destroy is called with a pointer to the allocator itself.
|
2012-11-10 14:43:27 +00:00
|
|
|
|
|
|
|
4.2 Pool-Agnostic API
|
|
|
|
|
|
|
|
One of the issues with emem was that the API (including the public data
|
|
|
|
structures) required wrapper functions for each scope implemented. Even
|
|
|
|
if there was a stack implementation in emem, it wasn't necessarily available
|
|
|
|
for use with file-scope memory unless someone took the time to write se_stack_
|
|
|
|
wrapper functions for the interface.
|
|
|
|
|
|
|
|
In wmem, all public APIs take the pool as the first argument, so that they can
|
|
|
|
be written once and used with any available memory pool. Data structures like
|
|
|
|
wmem's stack implementation only take the pool when created - the provided
|
|
|
|
pointer is stored internally with the data structure, and subsequent calls
|
|
|
|
(like push and pop) will take the stack itself instead of the pool.
|
2012-10-30 01:24:16 +00:00
|
|
|
|
2012-12-02 17:16:09 +00:00
|
|
|
5. TODO List
|
|
|
|
|
2012-12-27 22:38:48 +00:00
|
|
|
The following is a list of things that wmem provides but are incomplete
|
|
|
|
(i.e. missing common operations):
|
|
|
|
|
|
|
|
- string buffers
|
|
|
|
- singly-linked list
|
|
|
|
|
2012-12-02 17:16:09 +00:00
|
|
|
The following is an incomplete list of things that emem provides but wmem has
|
|
|
|
not yet implemented:
|
|
|
|
|
|
|
|
- red-black tree
|
|
|
|
- tvb_memdup
|
|
|
|
|
|
|
|
The following is a list of things that emem doesn't provide but that it might
|
|
|
|
be nice if wmem did provide them:
|
|
|
|
|
|
|
|
- radix tree
|
2012-12-19 00:43:36 +00:00
|
|
|
- dynamic array
|
2013-01-15 18:29:27 +00:00
|
|
|
- hash table
|
2012-12-19 00:43:36 +00:00
|
|
|
- realloc
|
2012-12-02 17:16:09 +00:00
|
|
|
|
2012-10-30 01:24:16 +00:00
|
|
|
/*
|
|
|
|
* Editor modelines - http://www.wireshark.org/tools/modelines.html
|
|
|
|
*
|
|
|
|
* Local variables:
|
|
|
|
* c-basic-offset: 4
|
|
|
|
* tab-width: 8
|
|
|
|
* indent-tabs-mode: nil
|
|
|
|
* End:
|
|
|
|
*
|
|
|
|
* vi: set shiftwidth=4 tabstop=8 expandtab:
|
|
|
|
* :indentSize=4:tabSize=8:noTabs=true:
|
|
|
|
*/
|