OCTOI - Osmocom Community TDM over IP ===================================== :author: Harald Welte :copyright: 2022 by Harald Welte (License: CC-BY-SA) :backend: slidy :max-width: 45em == Overview * WTF? Why? * History: Telephony / TDM networks * existing TDMoIP technology * OCTOI Protocol * OCTOI Software * Current OCTOI Network * WIP / Future Plans == Introduction Some people collect + operate old computers (_retrocomputing_) ... But what's a 1980s or 1990s computer without it's contemporary networking technology? == Zyxel V.32bis Modem image::zyxel.jpg[width=1000,align="center"] == ELSA MicroLink ISDN TA image::ELSA_MicroLink_ISDN_2ab_front.jpg[width=800,align="center"] == TELES S0/16.3 passiver ISDN-Adapter image::teles_s0_16_3.jpg[width=1000,align="center"] == T-DisplayTel ISDN-BTX-Telefon image::displaytel.jpg[width=1000,align="center"] == T-View 100 ISDN-Videotelefon image::tview100.jpg[width=800,align="center"] == WTF is this all about? * enable people to run legacy WAN equipment ** Modems ** ISDN-Adapters ** Phones, PBXs ** Telefax ** Frame Relay / HDLC / X.25 / routers * in a distributed fashion, over the internet * allow to run retronetworking tech from mid-1980s to about 2010 at a time where the related transport services are no longer available from public telephone operators / carriers == Migration paths for ISDN equipment * SIP ATA ** provides analog phone jacks on phone side, speaks SIP to operator ** works fine for voice telephony ** can occasionally work for Fax ** doesn't work well for Modems == Migration paths for ISDN equipment * Fritzbox with ISDN BRI (S0) port ** terminates Q.921/Q.931 ISDN protocol stack internally ** speaks SIP to operator ** works fine for voice telephony ** can work for Fax ** doesn't work well for Modems ** doesn't work at all for X.25/X.31 == Migration paths for ISDN equipment * _enterprise grade_ SIP/ISDN gateways ** available for BRI (S0) and PRI (E1) ** typically to help companies avoid replacing their ISDN PBX ** terminates Q.921/Q.931 ISDN protocol stack internally ** speaks SIP to operator ** works fine for voice telephony ** can work for Fax ** doesn't work well for Modems ** doesn't work at all for X.25/X.31 == VoIP: Why does only voice work well * well, it's called _voice_ over IP, not _modem_ over ip ;) * one problem is audio codecs ** most VoIP uses lossy compression (think of MP3) ** lossy codecs work for voice, but nothing else ** can only be solved if end-to-end VoIP path can be forced to G711 or CLEARMODE ** depending on situation (involved SIP operators, ...) this may or may not be possible * other problem is clocking ** impossible to solve / work around == Clocking in synchronous networks vs VoIP * classic analog networks had analog connection end-to-end. There's constant phase / delay * ISDN networks used PDH and later SDH for transmission ** tight synchronization between the clocks of all central offices ** clocks means both the sample clock of ADCs/DACs as well as bit clock of ISDN PRI/BRI * In VoIP networks everyone uses their own clock source ** typically a normal XTAL in your ATA / FritzBox with as much as 20 ppm tolerance * if clocks between two partners drift, there will be underruns or overruns ** if this happens, you drop samples or insert samples ** this creates phase discontinuity ** modem standards were *not* designed for this == Modem/ISDN in the VoIP world * VoIP really is designed for voice only ** codec issue can be worked around if you can control codecs end-to-end ** clock issue requires hardware modding your ATA/FritzBox/Gateway * you still have no transparency ** what about ISDN signaling subtleties like HLC/LLC in SETUP? ** what about X.25 over ISDN D-Channel (X.31) What you really want a *transparent* way of transporting ISDN/PSTN over IP, not some gateway that's way too high up the protocol stack. // *********************************************************************** // History: TDM networks // *********************************************************************** == History: Analog Telephony Networks * analog telephony exited (in Germany) since 1877 * analog electrical circuit from end to end * switching initially manually * automatic (electromechanical) switching introduced 1908-1966 == History: Digitization inside the network * analog pairs of copper lines * carrier frequency systems (TF-Systeme) over coaxial wire * first digital PCM (pulse code modulation) links between exchanges * digital switching with analog switch matrix (EWS) * digital switching with digital switch matrix (EWSD, S12) * 1979: decision to migrate entire German PSTN to digital switching ** estimated completion date in 2018 ** actual completion of all 6200 exchanges in 1997 == History: TDM networks * at some point (1970s?), digital PCM technology was introduced between exchanges ** initially PDH, later SDH (which can transport PDH) ** see https://media.ccc.de/v/osmodevcall-20211112-laforge-tdm[OsmoDevCall on TDM/PDH/SDH] * external interfaces, particularly to subscriber, still analog until mid-1980s * ISDN changed that: Introduced digital subscriber lines ** Basic Rate ISDN (BRI): 2x B, 1x D-Channel *** German: Basisanschluss (Anlagenanschluss, Mehrgeraeteanschluss) ** Primary Rate ISDN (PRI): 30x B, 1x D-Channel *** German: Primaermultiplexer / S2M == Present: TDM networks mostly dead * Telcos have meanwhile mostly migrated to All-IP telephony * PDH/SDH networks mostly shut down, occasionally still in operation for legacy customers or some internal legacy services. Not actively sold anymore. * You cannot get a real PRI line anymore to connect your ISDN PBX or your Cisco router * Idea: Create community network of people who want to play with ISDN / TDM stuff * Naive approach: Use existing, off-the-shelf TDMoIP devices == Wanted: TDMoIP network [graphviz] ---- graph G { hub [shape=box, label="TDMoIP hub\ncross-connect\nin public internet"]; subgraph cluster_1 { label="Hobbyist A"; ad1 [label="Access Device\n"]; pbx1 [label="PBX"]; phone1a [label="Phone"]; modem1b [label="Modem"]; ta1c [label="ISDN TA"]; pbx1 -- ad1 [label="E1"]; phone1a -- pbx1 [label="POTS"]; modem1b -- pbx1 [label="POTS"]; ta1c -- pbx1 [label="ISDN-BRI"]; } subgraph cluster_2 { label="Hobbyist B"; ad2 [label="Access Device\n"]; pbx2 [label="PBX"]; phone2a [label="Phone"]; modem2b [label="Modem"]; ta2c [label="ISDN TA"]; pbx2 -- ad2 [label="E1"]; phone2a -- pbx2 [label="POTS"]; modem2b -- pbx2 [label="POTS"]; ta2c -- pbx2 [label="ISDN-BRI"]; } subgraph cluster_3 { label="Hobbyist C"; ad3 [label="Access Device\n"]; pbx3 [label="PBX"]; phone3a [label="Phone"]; modem3b [label="Modem"]; ta3c [label="ISDN TA"]; pbx3 -- ad3 [label="E1"]; phone3a -- pbx3 [label="POTS"]; modem3b -- pbx3 [label="POTS"]; ta3c -- pbx3 [label="ISDN-BRI"]; } ad1 -- hub [label="TDMoIP\nInternet"]; ad2 -- hub [label="TDMoIP\nInternet"]; ad3 -- hub [label="TDMoIP\nInternet"]; } ---- // *********************************************************************** // Existing TDMoIP technology // *********************************************************************** == Existing TDMoIP: SAToP / CESoPSN So we just get some of these and all is good? image::RAD_IPmux-1_front.jpg[width=1000,align="center"] == Existing TDMoIP: SAToP / CESoPSN Examples for _transparent_ TDMoIP technologies * SAToP (Structure Agnostic TDM over IP) ** RFC 4553 * CESoPSN (Circuit Emulation Service over Packet Switched Network) ** RFC 5086 * transport of raw E1 frames over MPLS or UDP * designed for use in LAN or carrier backbone networks * waste a lot of bandwidth, even if TDM circuits are completely idle ** > 2Mbps plus header/packet overhead, bi-directional * typically no support for dynamic IP addresses == Existing TDMoIP: SIGTRAN IUA, EL2TPD Examples for _non-transparent_ TDMoIP technologies * SIGTRAN IUA (ISDN User Adaptation) ** RFC 3057 ** works only for ISDN BRI/PRI with Q.921 as Layer 2 on signaling channel ** uses SCTP as transport (might not pass all middleboxes in public internet) ** no specification how the B-channels are handled, pure signaling solution * Ericsson L2TP / PacketAbis over IP ** proprietary, but FOSS implementation in @osmo-el2tpd@ ** makes specific assumption of use of E1 as Ericsson Abis (GSM BTS back-haul) // *********************************************************************** // OCTOI TDMoIP protocol // *********************************************************************** == The case for a new TDMoIP protocol There's room for a new protocol based on the following goals: * *transparent*. Should work for ISDN and other use cases (Frame Relay, Abis, SS7, ATM, ...) * *efficient*. Should not waste a lot of bandwidth on an otherwise idle/unused link * *dynamic IP*. End-user internet access normally has dynamic IP addresses * *nat friendly*. Should work through any number of NATs and CG-NAT without special ALG/helper * *IPv6 support*. Should support IPv6 just like IPv4 * *authentication*. Should have built-in mutual authentication == OCTOI Protocol: Bandwidth * Timeslot Suppression ** Transmitter has history of 1 TDM frame ** Before Tx, to-be-transmitted frame is compared with last frame ** only those timeslots with value != that of last frame are transmitted * Batching ** Batch 32 (up to 40) E1 frames per UDP packet *** 8000 frames/s / 32 => 250 packets/s Result: About 70 kbit/s internet bandwidth use (including UDP + IPv4 overhead) for an idle E1 link == OCTOI Protocol: dynamic IP / NAT-friendly * Classic client/server approach * Server currently requires fixed IP (no STUN/ICE/...) * Clients can have dynamic IPs * All messages (control plane + TDM user plan) in one UDP flow // *********************************************************************** // OCTOI Software // *********************************************************************** == First supported OCTOI hardware: icE1usb * osmocom icE1usb is an open source hardware/gateware/firmware/driver E1 interface * originally developed for talking to GSM base stations * contains a GPS disciplined oscillator for high-precision bit clock timing * further reading ** https://osmocom.org/projects/e1-t1-adapter/wiki/IcE1usb ** https://media.ccc.de/v/osmodevcon2019-97-software-defined-e1 image::icE1usb-usb_side.jpg[width=550,align="center"] == E1oIP: osmo-e1d + icE1usb * osmo-e1d was the first (libusb, userspace) driver for the icE1usb hardware * traditionally, it sits between icE1usb hardware and applications using E1 like osmo-bsc * instead of a local application, it can now interface icE1usb to OCTOI ** `octoi-server` mode ** `octoi-client` mode First 5 OCTOI users were connected to the hub that way [graphviz] ---- graph { rankdir=LR; subgraph cluster_L { label="Site L"; PBX_L [label="PBX"]; icE1usb_L [label="icE1usb"]; GW_L [label="GW L"]; PBX_L -- icE1usb_L [label="E1"]; icE1usb_L -- GW_L [label="USB"]; } subgraph cluster_R { label="Site R"; PBX_R [label="PBX"]; icE1usb_R [label="icE1usb"]; GW_R [label="GW R"]; PBX_R -- icE1usb_R [label="E1"]; icE1usb_R -- GW_R [label="USB"]; } Internet; GW_L -- Internet [label="IP"]; GW_R -- Internet [label="IP"]; } ---- == E1oIP: osmo-e1d + dahdi-trunkdev * having one icE1usb per peer/user doesn't scale at the hub * this triggered the development of `dahdi-trunkdev` ** virtual E1 _span_ for DAHDI without any real hardware ** simply provides stream of E1 frames on a character-device ** can be used to implement any virtual TDM interface/protocol * osmo-e1d was extended to open dahdi-trunkdev instead of icE1usb This has become the only method of connecting to the OCTOI hub in September 2022 [graphviz] ---- graph { rankdir=LR; subgraph cluster_L { label="Site L"; PBX_L [label="PBX"]; icE1usb_L [label="icE1usb"]; GW_L [label="GW L"]; PBX_L -- icE1usb_L [label="E1"]; icE1usb_L -- GW_L [label="USB"]; } subgraph cluster_R { label="Site R"; PBX_R [label="PBX\n(Virtual DAHDI)"]; } Internet; GW_L -- Internet [label="IP"]; PBX_R -- Internet [label="IP"]; } ---- // *********************************************************************** // Current OCTOI Network // *********************************************************************** == Current OCTOI network hub [graphviz] ---- graph G { subgraph cluster_colo { label = "Co-Located Hub"; divf [label="DIVF\nCentral Switch",shape="house"]; icE1usb; pm3 [label="Livingston Portmaster 3"]; as54 [label="Cisco AS5400"]; icE1usb -- divf [label="E1 (timing master)"]; pm3 -- divf [label="E1"]; as54 -- divf [label="7xE1"]; } divf -- laforge [label="TDMoIP"]; divf -- manawyrm [label="TDMoIP"]; divf -- gruetzkopf [label="TDMoIP"]; divf -- roox [label="TDMoIP"]; divf -- DrDeke [label="TDMoIP"]; etc [label="other users..."]; divf -- etc [label="TDMoIP"]; } ---- * DIVF: Debian 11 with `dahdi-trunkdev`, `osmo-e1d` and `yate` * 2x TE820 8-port E1 cards, attaching to ** icE1usb (clock source) ** Livingston PortMaster 3 RAS ** Cisco AS5400 RAS == Rack view image::octoi_hub-colocated.jpg[width=900] == 1U Enclosure with APU, icE1usb, OVP image::octoi_hub-ice1usb.jpg[width=1100] == GPS Antenna mounted on the roof image::octoi_hub-gpsant.jpg[width=600] this is an old antenna mount previously used for a satellite dish == GPS Antenna mounted on the roof image::octoi_hub-gps.jpg[width=1200] == Current OCTOI participants The hub currently has the following clients / participants: * using icE1usb at hub side ** laforge ** manawyrm ** gruetzkopf ** roox ** cquirin ** drdeke ** tmwawpl ** tom/sirtux ** tnt ** marrold == Current OCTOI services * E1 / TDM as "transport" layer * ISDN network as "routing" layer ** hub simulates the network / central office / switch side ** client devices implement the "user" side, just like when attaching to ISDN/PSTN * Services on top of ISDN ** Audio Telephony ** Video Telephony (T-View 100 / H.320) ** ISDN Data Calls ** Analog Modem Calls ** Telefax == OCTOI ISDN / Modem calls * hub currently has a https://osmocom.org/projects/retronetworking/wiki/Livingston_Portmaster_3[Livingston Portmaster 3] RAS ** up to 30 ISDN data or analog modem calls (up to V.90) * services can be created in RADIUS, identified by Called Party Number ** PPP dial-up locally terminated ** PPP dial-up with forwarding of data via L2TP to remote LAC ** telnet or TCP clear forwarding to remote BBSs * Cisco AS5400 with much higher capacity is waiting to be provisioned == Challenges: Clocking * TDM networks need a stable bitclock at all parts of the network * everyone must transmit and receive at the same rate of bits / frames * we use GPS disciplined oscillators (GPS-DO) to ensure same clock everywhere ** this avoids overruns / underruns resulting in cycle slips that would create phase discontinuity for any modem signals carried over the network == Challenges: Packet Re-ordering * It seems like particularly on DOCSIS there is quite a bit of UDP packet re-ordering * Initially, osmo-e1d used a FIFO and dropped all out-of-order packets * later, we introduced a RIFO (Random In, First Out) to support re-ordering without loss == Challenges: Packet Loss * There is quite a bit of packet loss on the public internet * People probably don't generally notice much, as most services use TCP or retransmit UDP * Surprisingly, I couldn't find any comprehensive studies / research papers on packet loss behaviour of consumer internet that are less than 10 years old? * Right now we just accept it exists * Some early thoughts and experiments on FEC (Forward Error Correction) == Challenges: Latency * intercontinental public IP can easily reach >= 200ms RTT * ISDN timers in Q.921 and Q.931 are luckily working just fine over that kind of RTT * some suspicion that the high latency might have negative impact on Fax (T.30 timers) and modems (buffer sizes). I'm personally not yet convinced it is really an issue. == Current Services on top of OCTOI ISDN * Participants can call each other for any ISDN call/bearer type * Central hub provides ** Dial-Up (Modem up to V.90 + ISDN V.120/X.75) access *** Tons of telnet-reachable BBSs all around the world *** BTX (an instance of Neu-Ulm courtesy of https://btx-museum.de/) **** unfortunately currently still crashes ISDN BTX Terminal T-DisplayTel *** Fax gateway (WIP) * Some other users provide ** Video on demand for ISDN video telephony ** Remote DOOM for ISDN video telephony ** Chiptune audio on demand via voice calls https://osmocom.org/projects/octoi/wiki/Phonebook == What do I need to connect to the network? * Currently, we only offer PRI (E1) service, so you need ** an icE1usb plus some Linux USB host (Raspi, beaglebone, ...) ** some equipment that talks E1, like an _enterprise_ PBX *** Auerswald COMmander2 Basic with S2M is a popular and widely available choice *** this PBX then offers analog and BRI ports for your phones / modems / TAs == The setup used to connect KNF-Kongress * 19" Version of Auerswald COMmander2 Basic PBX * sufficient space internally to put Beaglebone + icE1usb inside image::octoi-event-pbx.jpg[width=1100] == What will I need to connect to the network? * We're working on a small, low-power fully integrated device * Ethernet/IP connection towards Internet / OCTOI hub * 2x S0 (BRI) ports for your ISDN equipment, or analog phones via a/b TA * First prototype looks promising. Main work pending is the uC software. image::osmo-xhfc2su-breakout.jpg[width=1100] // *********************************************************************** // Future Plans // *********************************************************************** == Future Plans * ISDN BRI (S/T) access to OCTOI network ** more end-user friendly; many people have S0 equipment and no E1/S2M PBX at home * Frame Relay switch / hub ** second, additional service, completely independent of the current ISDN service ** initial testing has confirmed HDLC / FR over OCTOI works without problems * X.25 network support ** X.25 access via ISDN D-Channel (X.31) * exhibits at hacker and vintage computing events (like VCFB!) == ISDN BRI (S/T) access to OCTOI * BRI (ISDN basic rate aka "S/T" aka "S0") support in OCTOI protocol * BRI hardware with 2x S/T interface and GPS-DO ** unlike icE1usb: Ethernet/IP support, not USB ** complete _appliance_ for OCTOI client mode: no need for computer ** Status: design of first break-out / evaluation board for ISDN BRI side complete ** Software not even started yet. Idea is to use Nuttx on Atmel SAMV71 == Longer-Term Future Plans * improve FOSS soft-modem situation (linmodem, spandsp) * investigate somewhat limited V.90 performance so far * support for other Q.931 dialects than DSS1 (like 1TR6 or even NI1) * dive into H.320 / H.221 to have software endpoint for ISDN video phones == Thanks Thanks to * Sylvain "tnt" Munaut for icE1usb and all his help * All of the early OCTOI network participants manawyrm, gruetzkopf, roox, DrDeke, tmwawpl, cquirin * noris.net for sponsoring the co-location of the OCTOI hub == Further Reading * https://osmocom.org/projects/octoi/wiki[OCTOI Project Homepage/Wiki] * https://osmocom.org/projects/octoi/wiki/Proposed_efficient_TDMoIP[OCTOI Protocol Description] * https://osmocom.org/projects/retronetworking/wiki[Retronetworking Wiki] * https://osmocom.org/projects/osmo-e1d/wiki/Wiki[osmo-e1d software] * https://osmocom.org/projects/octoi/wiki/Proposed_efficient_TDMoIP[icE1usb hardware] == Grafana / Stats Let's have a look at the statistics... == Modem call Let's demo a modem call == EOF End of File