Using mbedtls commit f9c599cd8ac9d00c484d4f5b027e18c6af4f9fdf before
they re-licensed to Apache 2.0, so we have a GPL-v2-or-later bsae64
implementation and avoid having code under a different license in the
tree.
This code is the unmodified import, so we can record any local changes
compared to the original version.
Change-Id: I39a9d3ab98257d21b9439b00528c744efa372c14
There also is an osmo_stat_item_desc, so the name 'desc' makes it hard
to read the code / the upcoming refactoring patches. It is an
osmo_stat_item_group_desc, so call it group_desc.
Related: SYS#5542
Change-Id: I07bc011450549a44ebf043e7d8a70718ddfd900e
Add "missing" API for looking up a stat_item_group by its index-name.
A subsequent patch, which adds stat_items to the CTRL interface, will
use this to look up stat item groups by object name.
In stat item groups, there are group names, having a number of indexes
denoting different objects. An object can have, besides the index, also
a name that is equivalent to the index.
Apologies for the weird function name, it's still the best one I
could come up with: "group_by_name" refers to the group name, and
"idxname" refers to the name that the object index is associated with.
We already have osmo_stat_item_get_group_by_name_idx().
Other contestants for name of this new function were:
- osmo_stat_item_get_group_by_name_name()
because there is a "name" instead of "idx", but I find it confusing.
- osmo_stat_item_get_group_by_name_idx_name()
but I find that the last "name" should be closer to the "idx".
Related: SYS#5542
Change-Id: Ia1a77a1e4657ba624dd4f4bf7ad274e7751d0141
Properly converting a string to an integer while validating against all
possible errors is not trivial. It is a recurring theme in code review,
and there are places in osmo code that do it wrong.
End this by providing a simple API, if for nothing else then as an
example of how to use strol() / strtoul() / strtoll() / strtoull()
in an airtight way.
A subsequent patch, adding stat items to the CTRL interface, uses this
to properly validate indexes in CTRL variables and convert them to int.
Related: SYS#5542
Change-Id: I4dac826aab00bc1780a5258b6b55d34ce7d50c60
Background:
* Individual values can be added to osmo_stat_item.values at any time.
* Stats are reported at a fixed interval (see vty 'stats interval'),
e.g. every 10 seconds.
* In order to report a new stat value, we use the maximum of all
osmo_stat_item.values added since the last report.
* By default, we do not send new stat values if they did not change
(see vty 'config-stats' -> 'flush-period' default of 0).
Fix the following bug:
* If 'flush-period' is 0, and no new osmo_stat_item.values are coming
in, the last value that gets reported is not necessarily the last
entry in osmo_stat_item.values.
* For attached reporters (statsd), it could then be that the given stat
stays at the wrong value for a long stretch of time (think of several
hours/days/forever).
Explanation of how the test shows that it is fixed:
* stats get reported (value is irrelevant)
* osmo_stat_item gets a new value: 20
* osmo_stat_item gets a new value: 10
* stats get reported (value: 20, the maximum of both new values)
* osmo_stat_item gets no new values
* stats get reported (value: 10, this is new because of the bug fix,
the real last value in osmo_stat_item, different from the 20 sent
earlier, without the fix it would not send anything here and the last
sent value would be 20)
* osmo_stat_item gets no new values
* stats get reported (nothing gets sent, since the real last value was
already sent and 'flush-period' is 0)
Fixes: OS#5215
Change-Id: Ibeefd0e3d1dbe4be454ff05a21df4848b2abfabe
It's really a false positive since _sb_l is compared and granted to be
psotivie by the time we compare, so we don't really care, but c++ is not
happy about it.
"""
/osmocom/core/utils.h:227:40: error: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
227 | if (_sb_l < 0 || _sb_l > _sb_remain) \
| ~~~~~~^~~~~~~~~~~~
"""
Change-Id: I90e7374aa959468670f1c0ea65a427398d423ddb
It is not be obvious on the first look that ->prev actually points to the last
element of a list, lets add a macro for that to make the API easier to use
Change-Id: Icf455bf6ba9d60bd311af17c9e80febaa42cacc9
Related: SYS#4971
Allow telling osmo_select_main* to only service pending writes (shutdown
mode). Introduce API fuctions to indicate a shutdown request, and find
out whether shutdown is complete.
Some osmo programs have a curious sleep of few seconds upon receiving
SIGTERM. The idea presumably was to finish off pending writes before
halting the program. But a sleep() on program exit is annoying,
especially when there usually are no pending writes, and when osmo-bsc
is launched numerous times for tests.
Change-Id: Ib94d4316924103459577087c2214188679db2227
Underlaying APIs (msgb_alloc) use a uint16_t as a type, which means
until now passing a value > 2^16 would succeed providing a msgb with
less space than requested.
Since those are static inline, there's no symbols used by apps, so we
should be safe enough changing the type to be uint16_t, since change
would only be applied at re-compile time.
Change-Id: I83c8222484e4856c68134a1a9d8cf96eb91af1b8
This patch adds a new field "name" to the rate_ctr and osmo_stat_item_group
structs, together with an API to set it. This new field allows for easy
identification of specific group instances when several of them exists,
rather than using a sometimes random/increasing index value.
If set, this name (string) is used instead of the index by the stats
reporter.
The name, if set, is also printed during "show stats" VTY commands.
It's up to the user or application to set up unique or meaningful names
to fullfill one's needs.
WARNING: this commit breaks ABI and possibly creates unexpected behavior
when run with non-rebuilt apps which use the modified structs directly
to get the coutners, or if use the static inline API rate_ctr_inc2().
Existing users of these structs should migrate to use new APIs
introduced in follow-up commits instead of accessing the field directly.
Related: SYS#5456
Change-Id: I0dc510783dd9ae8436dae8005a7b3330e80d36f3
Having this API and forcing apps to use it will allow easily adding new
members to the group structure without having so much impact in users of
this struct.
Related: SYS#5456
Change-Id: Iebbf401f11e36645f8964d389460918eb9e0910e
The api doc indicates the possibility to pass -1, and calling
osmo_tdef_get() actually casts the arg to a signed long. To end the
confusion, change default_timeout from unsigned long to long.
Change-Id: I51b9172603984839448346c9836e43c8c802fcf8
Every socket function that can be passed a 'flags' argument now
supports the following two additional macros that can be or-ed in
with the flags:
* OSMO_SOCK_F_DSCP(x) -- specify the IP DSCP of the socket
* OSMO_SOCK_F_PRIO(x) -- specify the priority of the socket
The existing osmo_sock_set_{dscp,priority}() functions are useful,
but you cannot call them in between the socket creation and the
connect() operation when using our socket helpers. This means that
the first packet sent will have the default DSCP/priority, and only
later packets would have the desired values.
When using the functionality introduced by this patch, we can ensure
that even the very first packet of e.g. a TCP or SCTP connect()
will have the correct DSCP/priority applied.
Change-Id: If22988735fe05e51226c6b091a5348dcf1208cdf
Related: SYS#5427
In some situations we want to set the SO_PRIORITY socket option
to determine the in-kernel priority of packets generated by this
socket.
Change-Id: I89abffcd125e6d073338a5c6437b9433220e1823
Related: SYS#5427
At least on Linux, sockets have a IP_TOS socket option that can be
configured to set the TOS. However, TOS (of RFC791) was replaced
by the DSCP (of RFC2474) in 1998.
As the DCSP bits are only the upper 6 bits of the TOS bits, let's
introduce a helper to get, mask and set the DSCP values in the TOS
bits.
Related: OS#5136, SYS#5427
Change-Id: Ia4ba389a5b7e3e9d5f17a742a900d6fd68c08e40
Fix counting of values missed because of FIFO overflow in
osmo_stat_item_get_next(), by assigning a new item value id effectively
as item->value[n + 1].id = item->value[n].id + 1, instead of increasing
a global_value_id that is shared between all items and groups. With
global_value_id, the count of values missed was wrong for one item, as
soon as a new value was added to another item.
This partially reverts b27b352e ("stats: Use a global index for stat
item values") from 2015, right after stats was added to libosmocore. It
was supposed to make multiple readers (reporters) possible, which could
read independently from stat_item (and later added comments explain it
like that). But this remained unused, stats has implemented multiple
reporters by reading all stat_items once and sending the same data to
all enabled reporters. The patch caused last_value_index in struct
osmo_stat_item to always remain at -1.
Replace this unused last_value_index with stats_next_id, so stats can
store the item-specific next_id in the struct again. It appears that
stats is the only direct user of osmo_stat_item, but if there are
others, they can bring their own item-specific next_id: functions in
stat_item.c still accept a next_id argument.
Related: OS#5088
Change-Id: Ie65dcdf52c8fc3d916e20d7f0455f6223be6b64f
Let osmo_stat_item_get_next, osmo_stat_item_discard,
osmo_stat_item_discard_all consistently refer to their next_id arg as
such (and not idx or next_idx). It refers to an ID (item->values[i].id),
not an index (item->values[i]), and it is always the next one, never the
current one.
Do the same change for _index/_idx variables in stats.c, which are used
as arguments to these functions. Replace rd_ with next_id_ in
stats_test.c, too.
Related: OS#5088
Change-Id: I5dd566b08dff7174d1790f49abd2d6ac020e120e
Introduce 2 new logging sub systems for signal and unit data.
Unify log messages so all log messages look similiar.
Log also Rx PDUs. Ensure dropped Tx packets (BLOCK/RESET on SNS)
contain *Tx*.
Change-Id: I34b8fde2955ecc010d1dcd9512e1bba9211e2c0d
Let's flag the API as deprecated so that people start using
log_set_print_filename2() API instead, which has less ackward
behavior implications like changing the print status of category-hex.
Related: OS#5034
Change-Id: If9b6b322989536a12094e6105c3aabc84d8be24a
This API wraps conventional gettid() linux-specific API, which even in
Linux itself is sometimes not properly supported/announced.
This API also allows future porting to other platforms if needed, and so
far falls back to getpid() if no gettid(9 can be found.
Code ported from osmo-trx.git, see commit 7a07de1efd4eb7cc11c33d3ad25cb2df70aa1ef1.
Related: OS#5027
Change-Id: Id7534beeb22fcd50813dab76dd68818e2ff87ec2
This adds an inter-thread queue "it_q" to libosmocore. With it_q,
one can perform thread-safe enqueing of messages to another thread,
who will receive the related messages triggered via an eventfd
handled in the usual libosmocore select loop abstraction.
Change-Id: Ie7d0c5fec715a2a577fae014b0b8a0e9c38418ef
In 'struct gsmtap_hdr' field 'snr_db' is defined as a signed integer,
however all functions that fill this structure accept an unsigned
integer. This is wrong, because SNR can be negative.
Let's use 'int8_t' instead of 'uint8_t'. Changing from unsigned
to signed should be relatively safe compared to the opposite.
Most of the callers I am aware of always do pass 0 anyway.
Change-Id: I9f432be5c346d563bf518111c14ff04d4a63f592
Related: SYS#5073
This function is called automatically on the main thread, but needs to
ba called explicitly in order to run the select loop on another thread.
Make it available for applications through talloc.h
Change-Id: Ie710ca9ad01d3fadb9f4ff344a55d6c01004727b
This is a helper function to broadcast an event to all of the
siblings of a specified FSM instance.
Change-Id: I2ce398741a8672d7b7c4058d056f46e2fe7353c1
Historically, BSSGP uses a non-constant, user-configurable integer
varieable for the logging sub-system. Let's replace this with a
statically-allocated library logging constant.
This is required if we want to use the subsystem number in e.g.
static initialized for osmo_fsm.log_subsys.
Change-Id: I506190aae9217c0956e4b5764d1a0c0772268e93
When importing the hashtable code in I8ef73a62fe9846ce45058eb21cf999dd3eed5741
I didn't import actual implementations of the fls() and fls64()
implementations, as at least gcc-10 was smart enough to detect
we only use it on constant types and hence the computation can happen
at build time via const_ilog2()
However, in our jenkins build verification' this doesn't appear to
happen, as we get below errors:
/build/deps/install/stow/libosmocore/include/osmocom/core/log2.h: In function ‘__ilog2_u32’:
/build/deps/install/stow/libosmocore/include/osmocom/core/log2.h:20:9: error: implicit declaration of function ‘fls’ [-Werror=implicit-function-declaration]
return fls(n) - 1;
^~~
/build/deps/install/stow/libosmocore/include/osmocom/core/log2.h: In function ‘__ilog2_u64’:
/build/deps/install/stow/libosmocore/include/osmocom/core/log2.h:28:9: error: implicit declaration of function ‘fls64’ [-Werror=implicit-function-declaration]
return fls64(n) - 1;
^~~~~
Let's provide some generic implementations for this case. If needed
one could also introduce architecture-specific assembly implementations
like in the Linux kernel, but so far we managed to keep libosmocore free
of any assembly tweaks.
Change-Id: Ifa4898eb66c8d949618edd47961b7a0330ed35b5
'new' is a reserved keyword in C++, so including this header from
a C++ project (like osmo-pcu) breaks compilation. Let's rename
it in the same way as it's already done in this file: add '_'.
Change-Id: I7f7d9143edca75ce932601386a8766b0a62c0e24
Fixes: I8ef73a62fe9846ce45058eb21cf999dd3eed5741
For more than a decade we've used the linuxlist.h for double-linked
lists. Let's also add the hlist (double-linked lists with single
pointer sized head, and the hashtable that builds on top of it.
This reflects the versions included in Linux 5.8 with some modifications
to make them build in userspace (remove RCU versions, adjust for
userspace include files and types, convert to doxygen).
Change-Id: I8ef73a62fe9846ce45058eb21cf999dd3eed5741
NSVC filtering was only implemented on sending messages, this also adds
log_set_context() calls to ns2_recv_vc()
Filtering by NSE is implemented similar to NSVC.
Change-Id: I63c0e85f82f5d08c5a6f535da94b8648498439d2
Related: SYS#5232
This adds an easy way to listen to netlink events form the Linux kernel
from within libosmocore applications.
The new dependency can be disabled via the "--disable-lbimnl" configure flag.
Change-Id: I4f787ee68f0d6d04f0a5655eb57d55b3b326a42f
This allows usual integer parsing at app level and calling this function
to make sure correct values will be passed to
osmo_serial_set_baudrate().
Change-Id: I41415c99d26128b33a8bf5ef7b38948bd1fe5d50
Some applications may need submillisecond timers, such as those
interacting with modbus serial lines (RS-485, RTU), which require
timers of values around 1.5 char-time (T1.5), where a data char is
composed of 11 bits sent on the line: 1 start bit, 8 data bits,
1 stop bit, and and parity bit (or 2nd stop bits if no parity).
For instance, for a baudrate of 9600:
1.5 * 11 / 9600 = 1.718 ms = 1718 us
So having a granularity of MS is not enough here.
Change-Id: I71848d7c1ee0649929ce07680ee7320bb2a42f0e
select is an ancient interface with weird restrictions, such as
the fact that it cannot be used for file descriptor values > 1024.
This may have been sufficient 40 years ago, but certainly is not in
2020. I wanted to migrate to epoll(), but unfortunately it doesn't
work well with the fact that existing programs simply set osmo_fd.flags
without making any API calls at the time they change those flags.
So let's do the migration to poll() as a first step, and then consider
epoll() as a second step further down the road, after introducing new
APIs and porting applications over.
The poll() code introduced in this patch is not extremely efficient,
as it needs to do extensive linked list iterations after poll() returns
in order to find the osmo_fd from the fd. Optimization is possible,
but let's postpone that to a follow-up patch.
At compile time, a new --enable-force-io-select argument can be given
to configure, forcing the use of the old select() backend instead of the
new poll() based backend.
Change-Id: I9e80da68a144b36926066610d0d3df06abe09bca
This change implements 'systemd-journal' logging target, that is
similar to the existing 'syslog' target. The key difference is
that 'systemd-journal' allows us to offload rendering of the meta
information, such as location (file name, line number), subsystem,
and logging level, to systemd. Moreover, we can attach arbitrary,
user-specific fields [1] to the logging messages, so they can be
used for advanced log filtering (e.g. by IMSI/TMSI/TLLI):
$ journalctl OSMO_SUBSYS=DMSC -f
Since we don't want to make libsystemd a required dependency, this
feature is optional, and needs to be enabled at build-time:
$ ./configure --enable-systemd-logging
The new logging target can be configured in the same way as any
other one - via the VTY interface, or using the configuration file:
log systemd-journal [raw]
logging level set-all notice
logging filter all 1
Two logging handlers are available: generic and raw. The first one
behaves similarly to both 'syslog' and 'stderr', i.e. all the meta
information is rendered by libosmocore itself, and then passed to
systemd together with the logging message. The later is more like
the 'gsmtap' target, so all available meta information is handed
over to systemd in form of fields [1]:
- CODE_FILE / CODE_LINE - location info,
- PRIORITY - syslog-compatible logging level,
- OSMO_SUBSYS - Osmocom-specific sub-system (e.g. DMSC),
- OSMO_SUBSYS_HEX - same as OSMO_SUBSYS, but encoded in hex,
- MESSAGE - the logging message itself,
and then can be rendered in any supported format (e.g. JSON).
More details about the API can be found in [2].
[1] https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html
[2] https://www.freedesktop.org/software/systemd/man/sd-journal.html
Change-Id: I609f5cf438e6ad9038d8fc95f00add6aac29fb23
If we watn to migrate to something like epoll(), user application
code must call a function of the libosmocore API whenever it changes
its read/write interest in a file descriptor.
Let's introduce API so applications can be ported to this API,
before making direct 'ofd->when' manipulations illegal as a second step.
Change-Id: Idb89ba7bc7c129a6304a76900d17f47daf54d17d
This will be useful to handle latitude and longitude numbers for GAD, which is
the location estimate representation used for LCS (Location Services).
The OsmoSMLC VTY user interface will provide floating-point strings like
"23.456" while GAD stores them as micro-degress 23456000. The osmo_gad_to_str*
will also convert latitude and longitude to floating-point string.
There was code review concerns against adding this API, upon which I tried to
use floating point string formats. But I encountered various problems with
accuracy and trailing zeros. For global positioning data (latitude and
longitude), even inaccuracy on the sixth significant decimal digit causes
noticeable positional shift. To achieve sufficient accuracy on the least
significant end, I need to use double instead of float. To remove trailing
zeros, the idea was to use '%.6g' format, but that can cause rounding. '%.6f'
on a double looks ok, but always includes trailing zeros. A test program shows:
%.6g of ((double)(int32_t)23230100)/1e6 = "23.2301" <-- good
%.6g of ((double)(int32_t)42419993)/1e6 = "42.42" <-- bad rounding
%.6g of ((double)(int32_t)23230199)/1e6 = "23.2302" <-- bad rounding
%.6f of ((double)(int32_t)23230100)/1e6 = "23.230100" <-- trailing zeros
%.6f of ((double)(int32_t)42419993)/1e6 = "42.419993" <-- good
%.6f of ((double)(int32_t)23230199)/1e6 = "23.230199" <-- good
It looks like when accepting that there will be trailing zeros, using double
with '%.6f' would work out, but in the end I am not certain enough that there
aren't more hidden rounding / precision glitches. Hence I decided to reinforce
the need to add this API: it is glitch free in sufficient precision for
latitude and longitude data, because it is based on integer arithmetic.
The need for this precision is particular to the (new) OsmoSMLC vty
configuration, where reading and writing back user config must not modify the
values the user entered. Considering to add these functions to osmo-smlc.git,
we might as well add them here to libosmocore utils, and also use them in
osmo_gad_to_str_*() functions.
Change-Id: Ib9aee749cd331712a4dcdadfb6a2dfa4c26da957
So far there is only osmo_use_count_name_buf(). Also provide a use count to
string using a talloc context, allowing to use OTC_SELECT.
- instead of foo_name(), rather use foo_to_str().
- osmo_use_count_name_buf() returns the buf and not the chars_needed. So add
osmo_use_count_to_str_buf() with a signature that is usable by
OSMO_NAME_C_IMPL().
- provide osmo_use_count_to_str_c() using OSMO_NAME_C_IMPL().
Change-Id: I1d2e7ee979f8c316ef99f7c65675b36d092ddfca
As shown in the recently added bitgen_test.c, using osmo_loadXXbe_ext() with a
smaller n produces results aligned on the most significant bytes, which is
cumbersome, since it does not return a previously stored value. This problem
exists only for the big-endian functions, the little-endian osmo_loadXXle_ext()
properly return values adjusted on the least significant octets.
Add osmo_loadXXbe_ext_2() variants that properly right-adjust the returned
value. Prominently highlight this behavior in API doc. Test the new functions
in bitgen_test.c.
For example, this eases handling of 24bit integers (e.g. loaded from buffer to
uint32_t, and stored into buffer from uint32_t). Also explicitly show this 24
bit case in bitgen_test.c
Change-Id: I2806df6f0f7bf1ad705d52fa386d4525b892b928
Add OSMO_ASSERT()s to ensure bounds checking.
For example, for osmo_store32le_ext(), passing n > 5 would read past the end of
the uint32_t. Similarly, osmo_load32le_ext() for n > 4 would write past the
uint32_t's end.
Change-Id: I2dc21582cd8a679b6624cefbc0c1678b093a3d08
Harald Welte
Neels Hofmeyr
Oliver Smith
Pau Espin
Philipp Maier
Neels Hofmeyr
Vadim Yanitskiy
Daniel Willmann