From d18c6e8db90d918c9fbc095c64feb2b10f404048 Mon Sep 17 00:00:00 2001 From: Harald Welte Date: Tue, 2 Jul 2019 14:06:40 +0800 Subject: [PATCH] osmdevcon 2019 nextepc --- .../osmodevcon2019-osmocom_nextepc.adoc | 217 + .../osmodevcon2019-osmocom_nextepc.html | 4644 +++++++++++++++++ 2 files changed, 4861 insertions(+) create mode 100644 2019/osmodevcon2019-osmocom_nextepc/osmodevcon2019-osmocom_nextepc.adoc create mode 100644 2019/osmodevcon2019-osmocom_nextepc/osmodevcon2019-osmocom_nextepc.html diff --git a/2019/osmodevcon2019-osmocom_nextepc/osmodevcon2019-osmocom_nextepc.adoc b/2019/osmodevcon2019-osmocom_nextepc/osmodevcon2019-osmocom_nextepc.adoc new file mode 100644 index 0000000..95dabe6 --- /dev/null +++ b/2019/osmodevcon2019-osmocom_nextepc/osmodevcon2019-osmocom_nextepc.adoc @@ -0,0 +1,217 @@ +nextepc from Osmocom point of view +================================== +:author: Harald Welte +:copyright: 2019 by Harald Welte (License: CC-BY-SA) +:backend: slidy +:max-width: 45em + +== nestepc intro + +Please see presentation of nextepc developer Sukchan Kim at OsmoDevCon 2019 + +This talk is about a "behind the scenes" look at the nextepc codebase through the eyes of an Osmocom +developer. + +Goal is to understand the high-level software architecture, and whether there is any chance for sharing +code/infrastructure between the two projects + +== linked lists + +* `lib/core/include/core_list.h` +* double-linked lists like our `linuxlist.h` +* not thread safe +* `list_insert_sorted()` has no libosmo* equivalent +** only user is timer.c, which appears not used? + +=> looks very much compatible to libosmocore logging + +== logging + +* static/global number of _log targets_ +** stdout, console, syslog, network, file +** each target has independent log level +* message types (defines formatting) +** RAW, TRACE, LOG, ASSERT +* non-RAW logging happens via snprintf to 8k sized stack buffer + +=> looks very much compatible to libosmocore logging + +== FSM + +* nextepc FSM abstraction in `lib/core/src/fsm.c` +* `fsm_init()`, `fsm_dispatch()` and `fsm_final()` are only API functions +* states are expressed by switching to a different function pointer for the event handler function +* rather simplistic when compared to osmo_fsm +** no onenter/onleave +** no constraints on permitted events / state transitions +** no integrated logging +** no introspection (like VTY or CTRL) +** no concept of FSM classes / instances +** no FSM hierarchy +** no extensive logging by FSM infrastructure + +== FSM usage + +* The only point where many instances of FSMs are used in parallel is inside the MME context.c (EMM and ESM). +* There don't appear to be multiple threads in the EMM and ESM FSMs, AFAICT. +* overall surprisingly low number of FSMs + +=> migration to `osmo_fsm` seems feasible + +== Events + +* `lib/core/src/event.c` +** asynchronous event delivery based on message queue +** `event_{create,delete,send,recv,timedrecv}()` +* timed events use dynamically-allocated timer to send event via queue at timer expiration +** `event_timer_create()`, `timer_create()`, `periodic_timer_create()` +* SGW and PGW combine event queue with FSM, where main thread receives events from queue to dispatch them to FSM + +=> nice idea. libosmo* signals and FSM input events are entirely synchronous. This ensures that all related +data structures exist while the event is being processed. Having queued events would require very careful +code design and possibly refcounting for pretty much all objects :/ + +== packet buffers + +* nextepc packet buffers in `lib/core/src/unix/pkbuf.c` +* _management_ structure is kept separate from _packet payload_ +** similar to Linux `sk_buff`, where you can `skb_clone()` whcih just duplicates the `struct sk_buff` but not the actual packet data +* single `payload` pointer to distinguish headers from payload +* pools of different buffer sizes (127/256/512/1024/2048/8192) +** index into pool is used as TEID :/ +* allocations are thread-safe +* buffers have reference count (to avoid deep copy?) +** only one user currently: S1AP paging message copying + +== memory allocator + +* `lib/core/src/unix/malloc.c` +* `core_{malloc,free,calloc,realloc}()` +* internally uses `pkbuf` as backing storage (not other way around) + +== message queue + +* `lib/core/include/core_msgq.h` +* `msgbq_{init,final,create,delete,send,recv,timedrecv}()` +* contains mutexes, condition variable +* blocking and non-blocking receive semantics + +=> may be candidate for `osmo_it_msgq` which si currently WIP. Signaling happens via osmo-select compatible eventfd, not condition variable + +== string utilities + +* `core_strdup`, `core_strndup` +* dynamically allocate their result using `core_malloc` + +== timers + +* `lib/core/include/core_timer.h` +* maintains separate list of active and idle timers +** no rbtree, linear list iteration +* supports both one-shot and periodic timers (libosmocore only one-shot) +* six (!) arguments for timer expiration function call-back function +* doesn't seem to have any direct users, only indirectly via `lib/core/src/event.c` + +== TLV + +* `lib/core/include/core_tlv.h` +* hierarchical TLV parser using dynamically-allocated objects for each TLV +* can express repeated tags +* can express nested tags +* supports only a sub-set of the TLV types (TLV, TL16V, T16L16) +* used heavily throughout [generated] GTP + +== freeDiameter + +* C-language DIAMETER protocol library +* development seems mostly discontinued during past 5 years +** 0 commits during past 12 months +** 11 commits during past 24 months +* projects typically use "fork" of freeDiameter copied into their repo +* internally uses plenty of threads +* applications register callback functions whenever related DIAMETER message is received +** callbacks executed in context of whichever freeDiameter thread + + +== MME + +* has the following threads: +** `sm_thread` / `sm_main()` +*** inbound event queue, dispatched into mme_sm FSM +** `net_thread` / `net_main()` +*** endless `sock_select_loop()` +** whatever threads freeDiameter creates + +== SGW + +* has the following threads: +** `sgw_thread` / `sgw_main()` +*** inbound event queue, dispatched into `sgw_sm` FSM +*** handles all of the SGW functionality (S1U to eNB, S11 to MME, S8 to PGW) + +== PGW + +* has the following threads: +** `pgw_thread` / `pgw_main()` +** inbound event queue, dispatched into pgw_sm FSM +** whatever threads freeDiameter creates + + +== HSS + +* FIXME +* web UI using node.js +** usees tons of dependencies (npm nightmare) +** doesn't rely on older/distibution-packaged versions +** should IMHO be an optional part, not mandatory + +== GTPv2C code generation + +* use 3GPP TS 29.274 word documen +* convert .doc to .docx (Office 2007+) +* use python script to parse tables in .docx +* generate C source code for encode/decoder from python script +* https://github.com/acetcom/nextepc/blob/master/lib/gtp/support/gtp_tlv.py[lib/gtp/support/gtp_tlv.py] + +=> I love it :) + +* Same approach also used for generating NAS encoder/decoder +** https://github.com/acetcom/nextepc/blob/master/lib/nas/support/nas_message.py[/lib/nas/support/nas_message.py] + +== S1-AP / ASN.1 PER / asn1c + +== Tests + +* C-language testsuite sending/receiving messages onvarious interfaces +* resembles the kind of tests we usually do in TTCN-3, but without TTCN-3 + +== Conclusions + +* nextepc uses threads, but is not heavily multithreaded +** this actually would make libosmocore integration more feasible than originally expected +* nextepc seems a much more heavyweight heap user +** e.g. every event or every timer is dynamically allocated vs. osmocom 'struct embedding', heavy stack use and synchronous event delivery +* there are some interesting ideas in nextepc, such as queued event, and timers that generate them +* most difficult problems probably around refcounting of packet buffers +* GTPv2 code generation could be adopted for OsmoGGSN GTPv2 support + +== Conclusions + +* I would love to bring some of our powerful features to nextepc +** talloc with related ability for memory leak debugging +** osmo_fsm with all of its power +** VTY for state introspection and runtime config changes + +== If I had a dream... + +... I would +* osmo-ify nextepc MME (logging, FSMs, VTY) +* extend OsmoGGSN with GTPv2 support via the nextepc code generation approach +* implement a simple DIAMENTER->GSUP translator to use OsmoHLR with nextepc +** use the kernel-side GTP-U user plane to focus on control plane only in GGSN/G-GW +* keep S-GW as-is for now, but think about kernel user plane there, too. +* convert test suite TTCN-3 + +== EOF + +End of File diff --git a/2019/osmodevcon2019-osmocom_nextepc/osmodevcon2019-osmocom_nextepc.html b/2019/osmodevcon2019-osmocom_nextepc/osmodevcon2019-osmocom_nextepc.html new file mode 100644 index 0000000..c75210c --- /dev/null +++ b/2019/osmodevcon2019-osmocom_nextepc/osmodevcon2019-osmocom_nextepc.html @@ -0,0 +1,4644 @@ + + + + +nextepc from Osmocom point of view + + + + + + + + +
+

nestepc intro

+
+

Please see presentation of nextepc developer Sukchan Kim at OsmoDevCon 2019

+

This talk is about a "behind the scenes" look at the nextepc codebase through the eyes of an Osmocom +developer.

+

Goal is to understand the high-level software architecture, and whether there is any chance for sharing +code/infrastructure between the two projects

+
+
+
+

linked lists

+
+
    +
  • + +lib/core/include/core_list.h + +
    • +
  • + +double-linked lists like our linuxlist.h + +
    • +
  • + +not thread safe + +
    • +
  • + +list_insert_sorted() has no libosmo* equivalent + +
      +
    • + +only user is timer.c, which appears not used? + +
      • +
    +
    • +
+

⇒ looks very much compatible to libosmocore logging

+
+
+
+

logging

+
+
    +
  • + +static/global number of log targets + +
      +
    • + +stdout, console, syslog, network, file + +
      • +
    • + +each target has independent log level + +
      • +
    +
    • +
  • + +message types (defines formatting) + +
      +
    • + +RAW, TRACE, LOG, ASSERT + +
      • +
    +
    • +
  • + +non-RAW logging happens via snprintf to 8k sized stack buffer + +
    • +
+

⇒ looks very much compatible to libosmocore logging

+
+
+
+

FSM

+
+
    +
  • + +nextepc FSM abstraction in lib/core/src/fsm.c + +
    • +
  • + +fsm_init(), fsm_dispatch() and fsm_final() are only API functions + +
    • +
  • + +states are expressed by switching to a different function pointer for the event handler function + +
    • +
  • + +rather simplistic when compared to osmo_fsm + +
      +
    • + +no onenter/onleave + +
      • +
    • + +no constraints on permitted events / state transitions + +
      • +
    • + +no integrated logging + +
      • +
    • + +no introspection (like VTY or CTRL) + +
      • +
    • + +no concept of FSM classes / instances + +
      • +
    • + +no FSM hierarchy + +
      • +
    • + +no extensive logging by FSM infrastructure + +
      • +
    +
    • +
+
+
+
+

FSM usage

+
+
    +
  • + +The only point where many instances of FSMs are used in parallel is inside the MME context.c (EMM and ESM). + +
    • +
  • + +There don’t appear to be multiple threads in the EMM and ESM FSMs, AFAICT. + +
    • +
  • + +overall surprisingly low number of FSMs + +
    • +
+

⇒ migration to osmo_fsm seems feasible

+
+
+
+

Events

+
+
    +
  • + +lib/core/src/event.c + +
      +
    • + +asynchronous event delivery based on message queue + +
      • +
    • + +event_{create,delete,send,recv,timedrecv}() + +
      • +
    +
    • +
  • + +timed events use dynamically-allocated timer to send event via queue at timer expiration + +
      +
    • + +event_timer_create(), timer_create(), periodic_timer_create() + +
      • +
    +
    • +
  • + +SGW and PGW combine event queue with FSM, where main thread receives events from queue to dispatch them to FSM + +
    • +
+

⇒ nice idea. libosmo* signals and FSM input events are entirely synchronous. This ensures that all related +data structures exist while the event is being processed. Having queued events would require very careful +code design and possibly refcounting for pretty much all objects :/

+
+
+
+

packet buffers

+
+
    +
  • + +nextepc packet buffers in lib/core/src/unix/pkbuf.c + +
    • +
  • + +management structure is kept separate from packet payload + +
      +
    • + +similar to Linux sk_buff, where you can skb_clone() whcih just duplicates the struct sk_buff but not the actual packet data + +
      • +
    +
    • +
  • + +single payload pointer to distinguish headers from payload + +
    • +
  • + +pools of different buffer sizes (127/256/512/1024/2048/8192) + +
      +
    • + +index into pool is used as TEID :/ + +
      • +
    +
    • +
  • + +allocations are thread-safe + +
    • +
  • + +buffers have reference count (to avoid deep copy?) + +
      +
    • + +only one user currently: S1AP paging message copying + +
      • +
    +
    • +
+
+
+
+

memory allocator

+
+
    +
  • + +lib/core/src/unix/malloc.c + +
    • +
  • + +core_{malloc,free,calloc,realloc}() + +
    • +
  • + +internally uses pkbuf as backing storage (not other way around) + +
    • +
+
+
+
+

message queue

+
+
    +
  • + +lib/core/include/core_msgq.h + +
    • +
  • + +msgbq_{init,final,create,delete,send,recv,timedrecv}() + +
    • +
  • + +contains mutexes, condition variable + +
    • +
  • + +blocking and non-blocking receive semantics + +
    • +
+

⇒ may be candidate for osmo_it_msgq which si currently WIP. Signaling happens via osmo-select compatible eventfd, not condition variable

+
+
+
+

string utilities

+
+
    +
  • + +core_strdup, core_strndup + +
    • +
  • + +dynamically allocate their result using core_malloc + +
    • +
+
+
+
+

timers

+
+
    +
  • + +lib/core/include/core_timer.h + +
    • +
  • + +maintains separate list of active and idle timers + +
      +
    • + +no rbtree, linear list iteration + +
      • +
    +
    • +
  • + +supports both one-shot and periodic timers (libosmocore only one-shot) + +
    • +
  • + +six (!) arguments for timer expiration function call-back function + +
    • +
  • + +doesn’t seem to have any direct users, only indirectly via lib/core/src/event.c + +
    • +
+
+
+
+

TLV

+
+
    +
  • + +lib/core/include/core_tlv.h + +
    • +
  • + +hierarchical TLV parser using dynamically-allocated objects for each TLV + +
    • +
  • + +can express repeated tags + +
    • +
  • + +can express nested tags + +
    • +
  • + +supports only a sub-set of the TLV types (TLV, TL16V, T16L16) + +
    • +
  • + +used heavily throughout [generated] GTP + +
    • +
+
+
+
+

freeDiameter

+
+
    +
  • + +C-language DIAMETER protocol library + +
    • +
  • + +development seems mostly discontinued during past 5 years + +
      +
    • + +0 commits during past 12 months + +
      • +
    • + +11 commits during past 24 months + +
      • +
    +
    • +
  • + +projects typically use "fork" of freeDiameter copied into their repo + +
    • +
  • + +internally uses plenty of threads + +
    • +
  • + +applications register callback functions whenever related DIAMETER message is received + +
      +
    • + +callbacks executed in context of whichever freeDiameter thread + +
      • +
    +
    • +
+
+
+
+

MME

+
+
    +
  • + +has the following threads: + +
      +
    • + +sm_thread / sm_main() + +
        +
      • + +inbound event queue, dispatched into mme_sm FSM + +
        • +
      +
      • +
    • + +net_thread / net_main() + +
        +
      • + +endless sock_select_loop() + +
        • +
      +
      • +
    • + +whatever threads freeDiameter creates + +
      • +
    +
    • +
+
+
+
+

SGW

+
+
    +
  • + +has the following threads: + +
      +
    • + +sgw_thread / sgw_main() + +
        +
      • + +inbound event queue, dispatched into sgw_sm FSM + +
        • +
      • + +handles all of the SGW functionality (S1U to eNB, S11 to MME, S8 to PGW) + +
        • +
      +
      • +
    +
    • +
+
+
+
+

PGW

+
+
    +
  • + +has the following threads: + +
      +
    • + +pgw_thread / pgw_main() + +
      • +
    • + +inbound event queue, dispatched into pgw_sm FSM + +
      • +
    • + +whatever threads freeDiameter creates + +
      • +
    +
    • +
+
+
+
+

HSS

+
+
    +
  • + +FIXME + +
    • +
  • + +web UI using node.js + +
      +
    • + +usees tons of dependencies (npm nightmare) + +
      • +
    • + +doesn’t rely on older/distibution-packaged versions + +
      • +
    • + +should IMHO be an optional part, not mandatory + +
      • +
    +
    • +
+
+
+
+

GTPv2C code generation

+
+
    +
  • + +use 3GPP TS 29.274 word documen + +
    • +
  • + +convert .doc to .docx (Office 2007+) + +
    • +
  • + +use python script to parse tables in .docx + +
    • +
  • + +generate C source code for encode/decoder from python script + +
    • +
  • + +lib/gtp/support/gtp_tlv.py + +
    • +
+

⇒ I love it :)

+ +
+
+
+

S1-AP / ASN.1 PER / asn1c

+
+
+
+
+

Tests

+
+
    +
  • + +C-language testsuite sending/receiving messages onvarious interfaces + +
    • +
  • + +resembles the kind of tests we usually do in TTCN-3, but without TTCN-3 + +
    • +
+
+
+
+

Conclusions

+
+
    +
  • + +nextepc uses threads, but is not heavily multithreaded + +
      +
    • + +this actually would make libosmocore integration more feasible than originally expected + +
      • +
    +
    • +
  • + +nextepc seems a much more heavyweight heap user + +
      +
    • + +e.g. every event or every timer is dynamically allocated vs. osmocom struct embedding, heavy stack use and synchronous event delivery + +
      • +
    +
    • +
  • + +there are some interesting ideas in nextepc, such as queued event, and timers that generate them + +
    • +
  • + +most difficult problems probably around refcounting of packet buffers + +
    • +
  • + +GTPv2 code generation could be adopted for OsmoGGSN GTPv2 support + +
    • +
+
+
+
+

Conclusions

+
+
    +
  • + +I would love to bring some of our powerful features to nextepc + +
      +
    • + +talloc with related ability for memory leak debugging + +
      • +
    • + +osmo_fsm with all of its power + +
      • +
    • + +VTY for state introspection and runtime config changes + +
      • +
    +
    • +
+
+
+
+

If I had a dream…

+
+
    +
  1. + +I would + +
      +
    • + +osmo-ify nextepc MME (logging, FSMs, VTY) + +
      • +
    • + +extend OsmoGGSN with GTPv2 support via the nextepc code generation approach + +
      • +
    • + +implement a simple DIAMENTER→GSUP translator to use OsmoHLR with nextepc + +
        +
      • + +use the kernel-side GTP-U user plane to focus on control plane only in GGSN/G-GW + +
        • +
      +
      • +
    • + +keep S-GW as-is for now, but think about kernel user plane there, too. + +
      • +
    • + +convert test suite TTCN-3 + +
      • +
    +
    2. +
+
+
+
+

EOF

+
+

End of File

+
+
+ +