add osmocon2018-testing_initiative
This commit is contained in:
parent
aabd18f4ce
commit
a8dc590f56
|
@ -0,0 +1,360 @@
|
|||
Osmocom Testing Initiative
|
||||
==========================
|
||||
:author: Harald Welte <laforge@gnumonks.org>
|
||||
:copyright: 2018 by Harald Welte (License: CC-BY-SA)
|
||||
:backend: slidy
|
||||
:max-width: 45em
|
||||
|
||||
== Osmocom testing until mid-2017
|
||||
|
||||
* manual testing
|
||||
* unit tests for some small portion of mostly utility functions
|
||||
** `make check` in the respective repository exceutes them
|
||||
* VTY interface interactive testing using specific python test code
|
||||
|
||||
== split NITB aftermath (the good parts)
|
||||
|
||||
* biggest architectural change since we started in 2008
|
||||
* lots of good reasons and design improvements
|
||||
** finite state machines with proper timeouts / clean-up
|
||||
** proper 3GPP AoIP with interoperability tesing
|
||||
** no synchronous HLR database access
|
||||
** HLR access from OsmoMSC and OsmoSGSN
|
||||
** 2G/3G authentication over GERAN and UTRAN
|
||||
|
||||
== split NITB aftermath (the bad parts)
|
||||
|
||||
[role="incremental"]
|
||||
* never-ending list of breakage
|
||||
** actual regressions of things that used to work before
|
||||
** things that were _known omissions_ during the restructuring
|
||||
* some commercial users stuck with SCCPlite and thus old @osmo-bsc-sccplite@
|
||||
** almost none of the new features or bug fixes there
|
||||
** no automatic testing
|
||||
** back-ports time-consuming
|
||||
|
||||
== split NITB aftermath (lessons learned)
|
||||
|
||||
* overall complexity of Osomcoom cellular is quite stunning now
|
||||
* absence of proper functional testing has caused massive fall-out
|
||||
* the split architecture allows for betteer testing of smaller parts of the system
|
||||
* my personal main focus of the last 11+ months:
|
||||
[role="incremental"]
|
||||
** testing, testing, testing, testing
|
||||
** testing, testing, testing, testing
|
||||
** some more testing
|
||||
** even more testing
|
||||
|
||||
|
||||
== Osmocom CNI testing (1/2)
|
||||
|
||||
[role="incremental"]
|
||||
* unit test (autotest, like we always had)
|
||||
** test individual functions / APIs of libraries / programs
|
||||
** executed during "make check" and hence before any patch can get merged
|
||||
* automatized functional tests in TTCN-3
|
||||
** test _external_ visible behavior on interfaces such as Abis, A, GSUP, GTP, MNCC, PCUIF, CTRL, VTY, ...
|
||||
** executed nightly by Jenkins (could be more frequently)
|
||||
|
||||
== Osmocom CNI testing (2/2)
|
||||
|
||||
[role="incremental"]
|
||||
* osmo-gsm-tester
|
||||
** tests entire Osmoocom network with BTS/BSC/MSC/HLR/PCU/SGSN/GGSN/...
|
||||
** uses real BTS + MS hardware (over coaxial cable)
|
||||
** automatic execution multiple times per day
|
||||
* interop tests
|
||||
** against NG40 RAN + CN simulator from NG4% (A / Gb / Iu level)
|
||||
** not fully automatized yet
|
||||
|
||||
|
||||
|
||||
== Osmocom TTCN-3 Test Suites
|
||||
|
||||
* developed in 2017+2018
|
||||
* compiled using Eclipse TITAN
|
||||
** uses just a command-line compiler + Makefiles
|
||||
** no IDE needed at all, don't let _Eclipse_ fool you
|
||||
* containerized in Docker
|
||||
* executed by Jenkins CI
|
||||
|
||||
== Terminology
|
||||
|
||||
ATS:: Abstract Test Suite
|
||||
MTC:: Main Test Component
|
||||
PTC:: Parallel Test Component
|
||||
IUT:: Implementation Under Test
|
||||
|
||||
== Test Suite Philosophy
|
||||
|
||||
* test one network element (our IUT)
|
||||
* test external behavior (3GPP and non-3GPP)
|
||||
* emulate entire environment from TTCN-3
|
||||
* don't reuse Osmocom C-code protocol implementations in the tests
|
||||
* test against independent TTCN-3 implementations!
|
||||
|
||||
== What to test?
|
||||
|
||||
* successful cases
|
||||
* erroneous cases (no answer, NACK, ...)
|
||||
** many difficult to reproduce with real phones/devices
|
||||
* load / resource exhaustion
|
||||
* spec compliance
|
||||
* focus on functionality actually relevant to IUT
|
||||
|
||||
== Why TTCN-3 + TITAN
|
||||
|
||||
* TTCN-3 specifically designed for telecom protocol testing
|
||||
* TITAN team released many telecom protocols in TTCN-3, such as
|
||||
** BSSAP, L3 (RR/MM/CC), SMS (CP/RP/TP), SS, M3UA, SCCP, GTP, NS, BSSGP, ...
|
||||
** shortens our test development cycle
|
||||
** permits us to test against known working industry implementations
|
||||
|
||||
== Test suites for Osmocom CNI components
|
||||
|
||||
* `osmo-bts`
|
||||
* `osmo-bsc` (for both AoIP and SCCPlite)
|
||||
* `osmo-msc`
|
||||
* `osmo-mgw`
|
||||
* `osmo-hlr`
|
||||
* `osmo-sip-connector`
|
||||
* `osmo-sgsn`
|
||||
* `osmo-ggsn`
|
||||
|
||||
== Test suites in progress
|
||||
|
||||
* `osmo-pcu`
|
||||
* `osmo-bsc_nat`
|
||||
* `osmo-gbproxy`
|
||||
|
||||
|
||||
|
||||
|
||||
== BTS_Tests.ttcn
|
||||
|
||||
* external interfaces
|
||||
** A-bis side: RSL (emulates BSC-side server)
|
||||
** Um side: L1CTL to control MS
|
||||
** PCU side: pcu_socket
|
||||
|
||||
[graphviz]
|
||||
----
|
||||
digraph G {
|
||||
rankdir=LR;
|
||||
{ rank=same; BTS, BSC};
|
||||
BTS [label="IUT\nosmo-bts-trx",shape="box"];
|
||||
ATS [label="ATS\nBTS_Tests.ttcn"];
|
||||
BSC [label="osmo-bsc\nOML only"];
|
||||
BTS -> fake_trx [label="bursts"];
|
||||
fake_trx -> trxcon [label="bursts"];
|
||||
trxcon -> ATS [label="bursts"];
|
||||
|
||||
BTS -> BSC [label="A-bis OML"];
|
||||
BTS -> ATS [label="A-bis RSL"];
|
||||
|
||||
ATS -> BTS [label="pcu_sock"];
|
||||
ATS -> BSC [label="VTY"];
|
||||
ATS -> BTS [label="CTRL"];
|
||||
}
|
||||
----
|
||||
|
||||
|
||||
|
||||
== BSC_Tests.ttcn
|
||||
|
||||
* external interfaces
|
||||
** A-bis side: RSL (emulates BTS-side client)
|
||||
** A-side: BSSAP/SCCP/M3UA (emulates MSC-side)
|
||||
** MGW side: MGCP (emulates MGW side)
|
||||
|
||||
[graphviz]
|
||||
----
|
||||
digraph G {
|
||||
rankdir=LR;
|
||||
{ rank=same; BTS; STP; };
|
||||
BSC [label="IUT\nosmo-bsc",shape="box"];
|
||||
ATS [label="ATS\nBSC_Tests.ttcn"];
|
||||
BTS [label="osmo-bts-omldummy\nOML only"];
|
||||
|
||||
BTS -> BSC [label="A-bis OML"];
|
||||
ATS -> BSC [label="A-bis RSL"];
|
||||
ATS -> BSC [label="CTRL"];
|
||||
ATS -> BSC [label="VTY"];
|
||||
ATS -> STP [label="A BSSAP\nSCCP/M3UA"];
|
||||
BSC -> STP [label="A BSSAP\nSCCP/M3UA"];
|
||||
}
|
||||
----
|
||||
|
||||
== MSC_Tests.ttcn
|
||||
|
||||
* external interfaces
|
||||
** A: BSSAP/SCCP/M3UA (emulates BSC-side)
|
||||
** MNCC: MNCC/unix-domain (emulates ext. MNCC side)
|
||||
** MGW: MGCP (emulates MGW side)
|
||||
** GSUP (impllements HLR side)
|
||||
|
||||
[graphviz]
|
||||
----
|
||||
digraph G {
|
||||
rankdir=LR;
|
||||
MSC [label="IUT\nosmo-msc",shape="box"];
|
||||
ATS [label="ATS\nMSC_Tests.ttcn"];
|
||||
|
||||
ATS -> MSC [label="MNCC"];
|
||||
ATS -> MSC [label="SMPP",style="dashed"];
|
||||
ATS -> MSC [label="CTRL"];
|
||||
ATS -> MSC [label="VTY"];
|
||||
MSC -> ATS [label="GSUP"];
|
||||
ATS -> STP [label="A BSSAP\nSCCP/M3UA"];
|
||||
MSC -> STP [label="A BSSAP\nSCCP/M3UA"];
|
||||
}
|
||||
----
|
||||
|
||||
|
||||
== MGCP_Test.ttcn
|
||||
|
||||
* external interfaces
|
||||
** MGCP (emulates call agent)
|
||||
** RTP (stream source/sink)
|
||||
|
||||
[graphviz]
|
||||
----
|
||||
digraph G {
|
||||
rankdir=LR;
|
||||
MGW [label="IUT\nosmo-mgw",shape="box"];
|
||||
ATS [label="ATS\nMGCP_Test.ttcn"];
|
||||
|
||||
ATS -> MGW [label="RTP"];
|
||||
ATS -> MGW [label="MGCP"];
|
||||
MGW -> ATS [label="RTP"];
|
||||
}
|
||||
----
|
||||
|
||||
== HLR_Tests.ttcn
|
||||
|
||||
* external interfaces
|
||||
** GSUP (emulates VLR/SGSN side)
|
||||
** VTY
|
||||
|
||||
[graphviz]
|
||||
----
|
||||
digraph G {
|
||||
rankdir=LR;
|
||||
HLR [label="IUT\nosmo-hlr",shape="box"];
|
||||
ATS [label="ATS\nHLR_Tests.ttcn"];
|
||||
|
||||
ATS -> HLR [label="GSUP"];
|
||||
ATS -> HLR [label="VTY"];
|
||||
}
|
||||
----
|
||||
|
||||
|
||||
== SIP_Tests.ttcn
|
||||
|
||||
* external interfaces
|
||||
** MNCC (emulates MSC side)
|
||||
** SIP (emulates SIP switch)
|
||||
** VTY
|
||||
|
||||
[graphviz]
|
||||
----
|
||||
digraph G {
|
||||
rankdir=LR;
|
||||
SIP [label="SIP\nosmo-sip-connector",shape="box"];
|
||||
ATS [label="ATS\nSIP_Tests.ttcn"];
|
||||
|
||||
ATS -> SIP [label="MNCC"];
|
||||
ATS -> SIP [label="SIP"];
|
||||
ATS -> SIP [label="VTY"];
|
||||
}
|
||||
----
|
||||
|
||||
|
||||
== SGSN_Tests.ttcn
|
||||
|
||||
* external interfaces
|
||||
** Gb (emulates PCU side NS/BSSGP + MS)
|
||||
** GSUP (emulates HLR)
|
||||
** VTY
|
||||
|
||||
[graphviz]
|
||||
----
|
||||
digraph G {
|
||||
rankdir=LR;
|
||||
SGSN [label="SGSN\nosmo-sgsn",shape="box"];
|
||||
ATS [label="ATS\nSGSN_Tests.ttcn"];
|
||||
|
||||
ATS -> SGSN [label="Gb"];
|
||||
SGSN-> ATS [label="Gp (GTP)"];
|
||||
ATS -> SGSN [label="VTY"];
|
||||
}
|
||||
----
|
||||
|
||||
|
||||
== GGSN_Tests.ttcn
|
||||
|
||||
* external interfaces
|
||||
** Gp: GTP (emulates SGSN)
|
||||
** Gi: IP (emulates Internet)
|
||||
|
||||
[graphviz]
|
||||
----
|
||||
digraph G {
|
||||
rankdir=LR;
|
||||
GGSN [label="GGSN\nosmo-ggsn",shape="box"];
|
||||
ATS [label="ATS\nGGSN_Tests.ttcn"];
|
||||
|
||||
ATS -> GGSN [label="Gp (GTP)"];
|
||||
GGSN -> ATS [label="Gi (IP)"];
|
||||
ATS -> GGSN [label="VTY"];
|
||||
}
|
||||
----
|
||||
|
||||
|
||||
|
||||
== Dockerized Setup
|
||||
|
||||
* one process per container
|
||||
* packages either
|
||||
** IUT (e.g. `osmo-bsc`)
|
||||
** ATS (compiled docker test suite)
|
||||
** other utility (e.g. `trxcon` or `osmo-bts-omldummy`)
|
||||
* why?
|
||||
** no need for local ip/network configuration
|
||||
** standardized / packaged setup on every machine
|
||||
** run older/newer versions of ATS against older/newer IUT
|
||||
|
||||
== Jenkins CI Execution
|
||||
|
||||
. update `docker-playground.git`
|
||||
.. contains `Dockerfile` for ATS + IUT
|
||||
. rebuild IUT container[s] (e.g. `osmo-bts-master`)
|
||||
.. git magic ensures re-build only if `osmo-bts.git` master changed
|
||||
. rebuild ATS container (e.g. `ttcn3-bts-test`)
|
||||
.. git magic ensures re-build only if `osmo-ttcn3-hacks.git` master changed
|
||||
. run `docker-playground/ttcn3-bts-test/jenkins.sh`
|
||||
.. creates docker network
|
||||
.. starts IUT + ATS docker containers
|
||||
.. collects test results
|
||||
|
||||
|
||||
== Jenkins CI Reporting
|
||||
|
||||
* junit-xml generation
|
||||
* store artefacts
|
||||
** pcap file of every test case
|
||||
** ATS log file (TTCN-3 testsuite)
|
||||
** IUT log file[s] (`osmo-*.log`)
|
||||
** IUT config file[s] (`osmo-*.cfg`)
|
||||
* see https://jenkins.osmocom.org/jenkins/view/TTCN3/
|
||||
|
||||
|
||||
== Further Reading
|
||||
|
||||
* http://git.osmocom.org/osmo-ttcn3-hacks/
|
||||
* http://git.osmocom.org/docker-playground/
|
||||
* http://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Notes
|
||||
|
||||
== EOF
|
||||
|
||||
End of File
|
Loading…
Reference in New Issue