297 lines
6.1 KiB
Plaintext
297 lines
6.1 KiB
Plaintext
Osmocom TTCN-3 Test Suites
|
|
==========================
|
|
:author: Harald Welte <laforge@gnumonks.org>
|
|
:copyright: 2018 by Harald Welte (License: CC-BY-SA)
|
|
:backend: slidy
|
|
:max-width: 45em
|
|
|
|
|
|
== Osmocom TTCN-3 Test Suites
|
|
|
|
* developed in 2017+2018
|
|
* compiled using Eclipse TITAN
|
|
** uses just a command-line compiler + Makefiles
|
|
** no IDE needed at all, don't let _Eclipse_ fool you
|
|
* containerized in Docker
|
|
* executed by Jenkins CI
|
|
|
|
== Terminology
|
|
|
|
ATS:: Abstract Test Suite
|
|
MTC:: Main Test Component
|
|
PTC:: Parallel Test Component
|
|
IUT:: Implementation Under Test
|
|
|
|
== Test Suite Philosophy
|
|
|
|
* test one network element (our IUT)
|
|
* test external behavior (3GPP and non-3GPP)
|
|
* emulate entire environment from TTCN-3
|
|
* don't reuse Osmocom C-code protocol implementations in the tests
|
|
* test against independent TTCN-3 implementations!
|
|
|
|
== What to test?
|
|
|
|
* successful cases
|
|
* erroneous cases (no answer, NACK, ...)
|
|
** many difficult to reproduce with real phones/devices
|
|
* load / resource exhaustion
|
|
* spec compliance
|
|
* focus on functionality actually relevant to IUT
|
|
|
|
== Why TTCN-3 + TITAN
|
|
|
|
* TTCN-3 specifically designed for telecom protocol testing
|
|
* TITAN team released many telecom protocols in TTCN-3, such as
|
|
** BSSAP, L3 (RR/MM/CC), SMS (CP/RP/TP), SS, M3UA, SCCP, GTP, NS, BSSGP, ...
|
|
** shortens our test development cycle
|
|
** permits us to test against known working industry implementations
|
|
|
|
== Test suites for Osmocom CNI components
|
|
|
|
* `osmo-bts`
|
|
* `osmo-bsc`
|
|
* `osmo-msc`
|
|
* `osmo-mgw`
|
|
* `osmo-hlr`
|
|
* `osmo-sip-connector`
|
|
* `osmo-sgsn`
|
|
* `osmo-ggsn`
|
|
|
|
== Test suites in progress
|
|
|
|
* `osmo-pcu`
|
|
* `osmo-bsc_nat`
|
|
|
|
|
|
|
|
|
|
== BTS_Tests.ttcn
|
|
|
|
* external interfaces
|
|
** A-bis side: RSL (emulates BSC-side server)
|
|
** Um side: L1CTL to control MS
|
|
** PCU side: pcu_socket
|
|
|
|
[graphviz]
|
|
----
|
|
digraph G {
|
|
rankdir=LR;
|
|
{ rank=same; BTS, BSC};
|
|
BTS [label="IUT\nosmo-bts-trx",shape="box"];
|
|
ATS [label="ATS\nBTS_Tests.ttcn"];
|
|
BSC [label="osmo-bsc\nOML only"];
|
|
BTS -> fake_trx [label="bursts"];
|
|
fake_trx -> trxcon [label="bursts"];
|
|
trxcon -> ATS [label="bursts"];
|
|
|
|
BTS -> BSC [label="A-bis OML"];
|
|
BTS -> ATS [label="A-bis RSL"];
|
|
|
|
ATS -> BTS [label="pcu_sock"];
|
|
ATS -> BSC [label="VTY"];
|
|
ATS -> BTS [label="CTRL"];
|
|
}
|
|
----
|
|
|
|
|
|
|
|
== BSC_Tests.ttcn
|
|
|
|
* external interfaces
|
|
** A-bis side: RSL (emulates BTS-side client)
|
|
** A-side: BSSAP/SCCP/M3UA (emulates MSC-side)
|
|
** MGW side: MGCP (emulates MGW side)
|
|
|
|
[graphviz]
|
|
----
|
|
digraph G {
|
|
rankdir=LR;
|
|
{ rank=same; BTS; STP; };
|
|
BSC [label="IUT\nosmo-bsc",shape="box"];
|
|
ATS [label="ATS\nBSC_Tests.ttcn"];
|
|
BTS [label="osmo-bts-omldummy\nOML only"];
|
|
|
|
BTS -> BSC [label="A-bis OML"];
|
|
ATS -> BSC [label="A-bis RSL"];
|
|
ATS -> BSC [label="CTRL"];
|
|
ATS -> BSC [label="VTY"];
|
|
ATS -> STP [label="A BSSAP\nSCCP/M3UA"];
|
|
BSC -> STP [label="A BSSAP\nSCCP/M3UA"];
|
|
}
|
|
----
|
|
|
|
== MSC_Tests.ttcn
|
|
|
|
* external interfaces
|
|
** A: BSSAP/SCCP/M3UA (emulates BSC-side)
|
|
** MNCC: MNCC/unix-domain (emulates ext. MNCC side)
|
|
** MGW: MGCP (emulates MGW side)
|
|
** GSUP (emulates HLR side)
|
|
|
|
[graphviz]
|
|
----
|
|
digraph G {
|
|
rankdir=LR;
|
|
MSC [label="IUT\nosmo-msc",shape="box"];
|
|
ATS [label="ATS\nMSC_Tests.ttcn"];
|
|
|
|
ATS -> MSC [label="MNCC"];
|
|
ATS -> MSC [label="SMPP",style="dashed"];
|
|
ATS -> MSC [label="CTRL"];
|
|
ATS -> MSC [label="VTY"];
|
|
MSC -> ATS [label="GSUP"];
|
|
ATS -> STP [label="A BSSAP\nSCCP/M3UA"];
|
|
MSC -> STP [label="A BSSAP\nSCCP/M3UA"];
|
|
}
|
|
----
|
|
|
|
|
|
== MGCP_Test.ttcn
|
|
|
|
* external interfaces
|
|
** MGCP (emulates call agent)
|
|
** RTP (stream source/sink)
|
|
|
|
[graphviz]
|
|
----
|
|
digraph G {
|
|
rankdir=LR;
|
|
MGW [label="IUT\nosmo-mgw",shape="box"];
|
|
ATS [label="ATS\nMGCP_Test.ttcn"];
|
|
|
|
ATS -> MGW [label="RTP"];
|
|
ATS -> MGW [label="MGCP"];
|
|
MGW -> ATS [label="RTP"];
|
|
}
|
|
----
|
|
|
|
== HLR_Tests.ttcn
|
|
|
|
* external interfaces
|
|
** GSUP (emulates VLR/SGSN side)
|
|
** VTY
|
|
|
|
[graphviz]
|
|
----
|
|
digraph G {
|
|
rankdir=LR;
|
|
HLR [label="IUT\nosmo-hlr",shape="box"];
|
|
ATS [label="ATS\nHLR_Tests.ttcn"];
|
|
|
|
ATS -> HLR [label="GSUP"];
|
|
ATS -> HLR [label="VTY"];
|
|
}
|
|
----
|
|
|
|
|
|
== SIP_Tests.ttcn
|
|
|
|
* external interfaces
|
|
** MNCC (emulates MSC side)
|
|
** SIP (emulates SIP switch)
|
|
** VTY
|
|
|
|
[graphviz]
|
|
----
|
|
digraph G {
|
|
rankdir=LR;
|
|
SIP [label="SIP\nosmo-sip-connector",shape="box"];
|
|
ATS [label="ATS\nSIP_Tests.ttcn"];
|
|
|
|
ATS -> SIP [label="MNCC"];
|
|
ATS -> SIP [label="SIP"];
|
|
ATS -> SIP [label="VTY"];
|
|
}
|
|
----
|
|
|
|
|
|
== SGSN_Tests.ttcn
|
|
|
|
* external interfaces
|
|
** Gb (emulates PCU side NS/BSSGP + MS)
|
|
** GSUP (emulates HLR)
|
|
** VTY
|
|
|
|
[graphviz]
|
|
----
|
|
digraph G {
|
|
rankdir=LR;
|
|
SGSN [label="SGSN\nosmo-sgsn",shape="box"];
|
|
ATS [label="ATS\nSGSN_Tests.ttcn"];
|
|
|
|
ATS -> SGSN [label="Gb"];
|
|
SGSN-> ATS [label="Gp (GTP)"];
|
|
ATS -> SGSN [label="VTY"];
|
|
}
|
|
----
|
|
|
|
|
|
== GGSN_Tests.ttcn
|
|
|
|
* external interfaces
|
|
** Gp: GTP (emulates SGSN)
|
|
** Gi: IP (emulates Internet)
|
|
|
|
[graphviz]
|
|
----
|
|
digraph G {
|
|
rankdir=LR;
|
|
GGSN [label="GGSN\nosmo-ggsn",shape="box"];
|
|
ATS [label="ATS\nGGSN_Tests.ttcn"];
|
|
|
|
ATS -> GGSN [label="Gp (GTP)"];
|
|
GGSN -> ATS [label="Gi (IP)"];
|
|
ATS -> GGSN [label="VTY"];
|
|
}
|
|
----
|
|
|
|
|
|
|
|
== Dockerized Setup
|
|
|
|
* one process per container
|
|
* packages either
|
|
** IUT (e.g. `osmo-bsc`)
|
|
** ATS (compiled docker test suite)
|
|
** other utility (e.g. `trxcon` or `osmo-bts-omldummy`)
|
|
* why?
|
|
** no need for local ip/network configuration
|
|
** standardized / packaged setup on every machine
|
|
** run older/newer versions of ATS against older/newer IUT
|
|
|
|
== Jenkins CI Execution
|
|
|
|
. update `docker-playground.git`
|
|
.. contains `Dockerfile` for ATS + IUT
|
|
. rebuild IUT container[s] (e.g. `osmo-bts-master`)
|
|
.. git magic ensures re-build only if `osmo-bts.git` master changed
|
|
. rebuild ATS container (e.g. `ttcn3-bts-test`)
|
|
.. git magic ensures re-build only if `osmo-ttcn3-hacks.git` master changed
|
|
. run `docker-playground/ttcn3-bts-test/jenkins.sh`
|
|
.. creates docker network
|
|
.. starts IUT + ATS docker containers
|
|
.. collects test results
|
|
|
|
|
|
== Jenkins CI Reporting
|
|
|
|
* junit-xml generation
|
|
* store artefacts
|
|
** pcap file of every test case
|
|
** ATS log file (TTCN-3 testsuite)
|
|
** IUT log file[s] (`osmo-*.log`)
|
|
** IUT config file[s] (`osmo-*.cfg`)
|
|
* see https://jenkins.osmocom.org/jenkins/view/TTCN3/
|
|
|
|
|
|
== Further Reading
|
|
|
|
* http://git.osmocom.org/osmo-ttcn3-hacks/
|
|
* http://git.osmocom.org/docker-playground/
|
|
* http://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Notes
|
|
|
|
== EOF
|
|
|
|
End of File
|