*Request for comments:*
In previous patches, I have fixed some more ue_context leak situations.
But since we don't really use ue_context for anything, we could also
just drop this completely.
On HNBAP UE Register, we collect the ue_contexts in a ue_list. But we
never do anything with this list, at all. Furthermore, users are
reporting the list of ue_context growing indefinitely.
Simply drop the ue_context listing. Simply acknowledge all HNBAP UE
Register and DeRegister requests without storing any context IDs.
Change-Id: Ida7eadf36abcf465ae40003725c49e8e321a28c9
During tests without an SGSN, I noticed SCCP connections failing, and I
also noticed that I couldn't tell if they were CS or PS related. Add a
"CS"/"PS" indicator to FSM instance IDs for RUA and SCCP.
Change-Id: I5b8242196af3b08eaf64ca5ac1c257a97a5d02cb
When receiving SCTP_RESTART for a given HNB, directly clear the UE
Contexts.
(The HNB typically connects via HNBAP shortly after this causing a UE
Context clearing too, but UE state should always be cleared on
SCTP_RESTART, no matter what.)
Change-Id: I583922193ba73e17ab85152005535188c2762b85
Whenever a HNB reconnects, we want to discard all previous UE state and
any active connections.
In one HNB reconnect scenario, we receive SCTP_RESTART. This sets
hnb_registered == false, which in turn skipped the UE cleanup step in
hnbgw_rx_hnb_register_req(), leaking UE Contexts.
Remove all weird conditions like that, and simply clear out all UE state
whenever a HNB registers, period.
Change-Id: I370966d2d76fd263714e727918fcc1ea2f2315fa
When waiting for CC expires, we should tell the SCCP-SCOC that we've
stopped waiting: send N-DISCONNECT, instead of nothing.
Change-Id: Ie94fcee4e2507a55449050aab96307199aed99a2
the function handle_rab_release() is difficult to read and it is not
immediately clear what happens and why. Lets split this up and add some
comments to improve this.
Related: OS#5916
Change-Id: I3595502b98ea5febbde7f2fab3999e2533766b48
Add braces around context part.
In the HNBGW_Tests.ttcn output, I see this:
DRUA DEBUG TTCN3 HNodeB transmitting RUA DirectTransfer
which reads like the hNodeB would transmit a RUA to us. Instead, this is
us sending RUA to the hNodeB, which is much clearer like this:
DRUA DEBUG (TTCN3 HNodeB) transmitting RUA DirectTransfer
This matches the way we typically show context info in osmo logging.
Change-Id: If6f0c3ae81c737b7488fa93c435179dcf27a5c94
Fix PS RAB Assignment operation:
- add #include "config.h"
- fix uncompilable code in ENABLE_PFCP sections.
A series of oversights made me completely break PFCP / UPF operation for
PS RAB Assignments in this commit:
'context map: introduce RUA and SCCP FSMs to fix leaks'
ed424d5be4
I6ff7e36532ff57c6f2d3e7e419dd22ef27dafd19
- In my HNBGW_Tests.ttcn, I used osmo-hnbgw-with-pfcp.cfg, but failed to
set mp_enable_pfcp_tests := true in HNBGW_Tests.cfg.
- Hence I was under the impression that I was testing PFCP via UPF when
really I wasn't. So I did not notice that:
- in the new files context_map_rua.c and context_map_sccp.c, the
ENABLE_PFCP macro was always unset because I forgot to
#include "config.h".
- Hence I did not notice that the moved PFCP RAB code was never once
compiled or run.
My bad, that was really dumb.
Related: SYS#6297
Change-Id: I5df5073f0092ecdfd73d5d08207ca4042154eab1
Silence the error log about "unknown" PCSTATE prim.
Todo / coming up: instead of ignoring, detect a Destination Unavailable
from prim->u.pcstate and disconnect all conns with that particular CN
link.
Depends: libosmo-sccp If381f537ab91af1feef7f0e51921217f27e18e6a
Change-Id: I547387a5cc14ccb506be04ac785e6807fc4e6a96
Refactor the entire RUA <-> SCCP connection-oriented message forwarding:
- conquer confusion about hnbgw_context_map release behavior, and
- eradicate SCCP connection leaks.
Finer points:
== Context map state ==
So far, we had a single context map state and some flags to keep track
of both the RUA and the SCCP connections. It was easy to miss connection
cleanup steps, especially on the SCCP side.
Instead, the two FSMs clearly define the RUA and SCCP conn states
separately, and each side takes care of its own release needs for all
possible scenarios.
- When both RUA and SCCP are released, the context map is discarded.
- A context map can stay around to wait for proper SCCP release, even if
the RUA side has lost the HNB connection.
- Completely drop the async "context mapper garbage collection", because
the FSMs clarify the release and free steps, synchronously.
- We still keep a (simplified) enum for global context map state, but
this is only used so that VTY reporting remains mostly unchanged.
== Context map cleanup confusion ==
The function context_map_hnb_released() was the general cleanup function
for a context map. Instead, add separate context_map_free().
== Free context maps separately from HNB ==
When a HNB releases, talloc_steal() the context maps out of the HNB
specific hnb_ctx, so that they are not freed along with the HNB state,
possibly leaving SCCP connections afloat.
(It is still nice to normally keep context maps as talloc children of
their respective hnb_ctx, so talloc reports show which belongs to
which.)
So far, context map handling found the global hnb_gw pointer via
map->hnb_ctx->gw. But in fact, a HNB may disappear at any point in time.
Instead, use a separate hnb_gw pointer in map->gw.
== RUA procedure codes vs. SCCP prims ==
So far, the RUA rx side composed SCCP prims to pass on:
RUA rx ---SCCP-prim--> RANAP handling ---SCCP-prim--> SCCP tx
That is a source of confusion: a RUA procedure code should not translate
1:1 to SCCP prims, especially for RUA id-Disconnect (see release charts
below).
Instead, move SCCP prim composition over to the SCCP side, using FSM
events to forward:
RUA rx --event--> RUA FSM --event--> SCCP FSM --SCCP-prim--> SCCP tx
+RANAP +RANAP +RANAP
RUA tx <--RUA---- RUA FSM <--event-- SCCP FSM <--event-- SCCP rx
+RANAP +RANAP +RANAP
Hence choose the correct prim according to the SCCP FSM state.
- in hnbgw_rua.c, use RUA procedure codes, not prim types.
- via the new FSM events' data args, pass msgb containing RANAP PDUs.
== Fix SCCP Release behavior ==
So far, the normal conn release behavior was
HNB HNBGW CN
| --id-Disconnect--> | ---SCCP-Released--> | Iu-ReleaseComplete
| | <--SCCP-RLC-------- | (no data)
Instead, the SCCP release is now in accordance with 3GPP TS 48.006 9.2
'Connection release':
The MSC sends a SCCP released message. This message shall not contain
any user data field.
i.e.:
HNB HNBGW CN
| --id-Disconnect--> | ---Data-Form-1(!)--> | Iu-ReleaseComplete
| | <--SCCP-Released---- | (no data)
| | ---SCCP-RLC--------> | (no data)
(Side note, the final SCCP Release Confirm step is taken care of
implicitly by libosmo-sigtran's sccp_scoc.c FSM.)
If the CN fails to respond with SCCP-Released, on new X31 timeout,
osmo-hnbgw will send an SCCP Released to the CN as fallback.
== Memory model for message dispatch ==
So far, an osmo_scu_prim aka "oph" was passed between RUA and SCCP
handling code, and the final dispatch freed it. Every error path had to
take care not to leak any oph.
Instead, use a much easier and much more leakage proof memory model,
inspired by fixeria:
- on rx, dispatch RANAP msgb that live in OTC_SELECT.
- no code path needs to msgb_free() -- the msgb is discarded via
OTC_SELECT when handling is done, error or no error.
- any code path may also choose to store the msgb for async dispatch,
using talloc_steal(). The user plane mapping via MGW and UPF do that.
- if any code path does msgb_free(), that would be no problem either
(but none do so now, for simplicity).
== Layer separation ==
Dispatch *all* connection-oriented RUA tx via the RUA FSM and SCCP tx
via the SCCP FSM, do not call rua_tx_dt() or osmo_sccp_user_sap_down()
directly.
== Memory model for decoded ranap_message IEs ==
Use a talloc destructor to make sure that the ranap_message IEs are
always implicitly freed upon talloc_free(), so that no code path can
possibly forget to do so.
== Implicit cleanup by talloc ==
Use talloc scoping to remove a bunch of explicit cleanup code. For
example, make a chached message a talloc child of its handler:
talloc_steal(mgw_fsm_priv, message);
mgw_fsm_priv->ranap_rab_ass_req_message = message;
and later implicitly free 'message' by only freeing the handler:
talloc_free(mgw_fsm_priv)
Related: SYS#6297
Change-Id: I6ff7e36532ff57c6f2d3e7e419dd22ef27dafd19
Planning new connection-oriented RUA and SCCP FSMs to
- conquer confusion about hnbgw_context_map release behavior, and
- eradicate SCCP connection leaks.
Related: SYS#6297
Change-Id: I661bf65d79972a732c52732934095e8bfcd99694
Change the order in the struct definition so that the items for RUA and
for SCCP are grouped together. Tweak comments.
Change-Id: I84424617996d8e1a6814aa122e63454c0666b724
Instead of osmo_scu_prim_type, use RUA_ProcedureCode to tell
rua_to_scu() the RUA message type being handled.
Cosmetically prepare for upcoming patch that completely moves
osmo_scu_prim composition from rua_to_scu() to a new context map SCCP
FSM.
Related: I6ff7e36532ff57c6f2d3e7e419dd22ef27dafd19
Change-Id: I28735945844f732e7bd7f09db1f453c608415b84
From rua_to_scu(), drop dead code for connection-less messages. In fact,
rua_to_scu() is only used for connection-oriented messages. Clarify
that.
Change-Id: Id03c412cde066a6b2bbc26cb6b87053aa2408cba
Rename context_map_deactivate() to context_map_hnb_released().
This function is called only when the HNB is released / lost.
Change-Id: I6dcb26c94558fff28faf8f823e490967a9baf2ec
The SCCP CR payload length limit is now implemented in libosmo-sigtran
v1.7.0. The limit no longer needs to be enforced in osmo-hnbgw.
This reverts commit 2c91bd66a1,
except for keeping the cfg option, marked deprecated, and not doing
anything.
Fixes: OS#5906
Related: SYS#5968
Related: OS#5579
Depends: I174b2ce06a31daa5a129c8a39099fe8962092df8 (osmo-ttcn3-hacks)
Change-Id: I18dece84b33bbefce8617fbb0b2d79a7e5adb263
User reports show leaked UE contexts over time, in a scenario where HNB
regularly disconnect and reconnect.
So far, when we receive a HNB REGISTER REQ, we log as "duplicated" and
continue to use the hnb_context unchanged. But it seems obvious that a
HNB that registers does not expect any UE state to remain. It will
obviously not tear down UE contexts (HNBAP or RUA) that have been
established before the HNB REGISTER REQUEST. These hence remain in
osmo-hnbgw indefinitely.
When receiving a HNB REGISTER REQUEST, release all its previous UE
state. Allow the HNB to register with a clean slate.
The aim is to alleviate the observed build-up of apparently orphaned UE
contexts, in order to avoid gradual memory exhaustion.
Related: SYS#6297
Change-Id: I7fa8a04cc3b2dfba263bda5b410961c77fbed332
'map' is already guaranteed to be not NULL for that message type by the
preceding switch().
Related: CID#307434
Change-Id: Id30f459616391187aa2f8ad400c316e2144154c6
Clean up SCCP connections when a HNB disconnects.
When a HNB disconnects, we clean up all RUA <-> SCCP connection state
for that HNB. In that cleanup, discarding the SCCP connection is so far
missing.
Add a flag indicating true between SCCP CC and DISCONNECT. Hence we can
tell during context_map_deactivate() whether the cleanup is graceful
(DISCONNECT already sent) or non-graceful (need to DISCONNECT).
Change-Id: Icc2db9f6c0b2d0a814ff1110ffbe5e8f7f629222
Deallocate a recently allocated UE context if the UE REGISTER procedure
fails internally, in two places.
The UE REGISTER error is rather unlikely to happen in practice: it only
occurs when encoding the HNBAP response fails, which only gets checked
input and thus is unlikely to fail.
The same code paths also possibly leak asn1c allocations -- leave those
for another patch.
Related: SYS#6297
Change-Id: Icf4b82f9a904d56332c567f7ccfb24231ee66270
Fix leaked msgb introduced by the MGW support recently added, and from
there copied to the UPF support added after that.
Fixes leaked "RANAP Tx" msgb, one per RAB Assignment that involves an
MGW or UPF proxying of user data.
Related: SYS#6297
Change-Id: Ie30e880301346ffca72f98f8c467e56d622fb03f
bisect shows that the segfault was introduced by using the MGCP client
pool:
e62af4d46a is the first bad commit
Author: Pau Espin Pedrol <pespin@sysmocom.de>
Introduce support for libosmo-mgcp-client MGW pooling
Change-Id I371dc773b58788ee21037dc25d77f556c89c6b61
The segfault:
20230117224550365 DLMGCP DEBUG MGCP_CONN(to-HNB)[0x612000003ca0]{ST_CRCX_RESP}: Timeout of T1 (fsm.c:317)
[...]
20230117224550366 DLMGCP DEBUG mgw-endp(mgw-fsm-14429752-0)[0x612000003b20]{WAIT_MGW_RESPONSE}: Deallocated (fsm.c:568)
20230117224550366 DMGW DEBUG mgw(mgw-fsm-14429752-0)[0x612000003820]{MGW_ST_CRCX_HNB}: Received Event MGW_EV_MGCP_TERM (mgcp_client_endpoint_fsm.c:869)
=================================================================
==255699==ERROR: AddressSanitizer: heap-use-after-free on address 0x62b000000260 at pc 0x7f282a6ee143 bp 0x7fff0d9bcae0 sp 0x7fff0d9bcad8
READ of size 8 at 0x62b000000260 thread T0
#0 0x7f282a6ee142 in osmo_mgcpc_ep_client ../../../../src/osmo-mgw/src/libosmo-mgcp-client/mgcp_client_endpoint_fsm.c:223
#1 0x55e2a84f1889 in mgw_fsm_allstate_action ../../../../src/osmo-hnbgw/src/osmo-hnbgw/mgw_fsm.c:504
#2 0x7f2829d50c56 in _osmo_fsm_inst_dispatch ../../../src/libosmocore/src/fsm.c:863
#3 0x7f2829d55a08 in _osmo_fsm_inst_term ../../../src/libosmocore/src/fsm.c:962
#4 0x7f282a72679a in osmo_mgcpc_ep_fsm_check_state_chg_after_response ../../../../src/osmo-mgw/src/libosmo-mgcp-client/mgcp_client_endpoint_fsm.c:869
#5 0x7f282a6f1869 in on_failure ../../../../src/osmo-mgw/src/libosmo-mgcp-client/mgcp_client_endpoint_fsm.c:414
#6 0x7f282a727ac6 in osmo_mgcpc_ep_fsm_handle_ci_events ../../../../src/osmo-mgw/src/libosmo-mgcp-client/mgcp_client_endpoint_fsm.c:935
[...]
When a CRCX times out, MGCP_CONN fsm terminates (libosmo-mgcp-client).
In turn the parent mgw-endp fsm terminates (libosmo-mgcp-client).
This generates an MGW_EV_MGCP_TERM event to the mgw_fsm (osmo-ttcn3-hacks).
This attempts to retrieve a pointer from mgw_fsm state:
mgw_fsm_priv->mgcpc_ep->mgcp_client
where the middle one, mgcpc_ep, is the 'mgw-endp' that already deallocated above.
To fix, add to /osmo-hnbgw/mgw_fsm.c a separate pointer to the
mgcp_client, to call mgcp_client_pool_put() on it. Do not use mgcpc_ep
to get the mgcp_client, because mgcpc_ep deallocates independently.
Related: OS#5862
Change-Id: I460d7249f4fc7edcfd94f6084fc8f933b491520c
Looking at a customer's log, these error logs got my attention. There
seems to be no point in logging this at all.
Change-Id: I89dd4fb6913bfb84b6667b159e09968734e2102a
This fixes bug in use of M3UA/SCCP after 2**24 connection IDs have been
allocated.
Related: SYS#6211
Change-Id: I03bad960f65fbff6e467def5bba60fefb328f962
This way we document the recently gained support for MGW pooling.
Related: SYS#5987
Depends: osmo-gsm-manuals.git Change-Id Ieda0d4bfe6fc90da6e19c791d8ec2da89427ba3b
Change-Id: I3dc8a4b50f13ad50390ba82e64fe4ebe0b97d4e5
Large RAN installations may benefit from distributing the RTP voice
stream load over multiple media gateways.
libosmo-mgcp-client supports MGW pooling since version 1.8.0 (more than
one year ago). OsmoBSC has already been making use of it since then (see
osmo-bsc.git 8d22e6870637ed6d392a8a77aeaebc51b23a8a50); lets use this
feature in osmo-hngw too.
This commit is also part of a series of patches cleaning up
libosmo-mgcp-client and slowly getting rid of the old non-mgw-pooled VTY
configuration, in order to keep only 1 way to configure
libosmo-mgcp-client through VTY.
Related: SYS#5091
Related: SYS#5987
Change-Id: I371dc773b58788ee21037dc25d77f556c89c6b61
Otherwise the libosmo-netif stream API may continue accessing the conn
after returning if the socket has the WRITE flag active in the same main
loop iteration.
Change-Id: I628c59a88d94d299f432f405b37fbe602381d47e
It was seen on a real pcap trace (sctp & gsmtap_log) that the kernel
stack may decide to kill the connection (sending an ABORT) if it fails
to transmit some data after a while:
ABORT Cause code: "Protocol violation (0x000d)",
Cause Information: "Association exceeded its max_retrans count".
When this occurs, the kernel sends the
MSG_NOTIFICATION,SCTP_ASSOC_CHANGE,SCTP_COMM_LOST notification when
reading from the socket with sctp_recvmsg(). This basically signals that
the socket conn is dead, and subsequent writes to it will result in
send() failures (and receive SCTP_SEND_FAILED notification upon follow
up reads).
It's important to notice that after those events, there's no other sort
of different event like SHUTDOWN coming in, so that's the time at which
we must tell the user to close the socket.
Hence, let's signal the caller that the socket is dead by returning 0,
to comply with usual recv() API.
Related: SYS#6113
Change-Id: If35efd404405f926a4a6cc45862eeadd1b04e08c
Neels Hofmeyr
Philipp Maier
Pau Espin
arehbein