522 lines
16 KiB
Plaintext
522 lines
16 KiB
Plaintext
TTCN-3 and Eclipse TITAN for testing protocol stacks
|
||
====================================================
|
||
:author: Harald Welte <laforge@gnumonks.org>
|
||
:copyright: 2017 by Harald Welte (License: CC-BY-SA)
|
||
:backend: slidy
|
||
:max-width: 45em
|
||
|
||
== Protocol Testing
|
||
|
||
Important for:
|
||
|
||
* conformance to specification
|
||
* ensuring interoperability
|
||
* network security
|
||
* regression testing
|
||
* performance
|
||
|
||
== Protocol Testing
|
||
|
||
No standard methodology, language, approach, tool
|
||
|
||
* testing implementation against itself
|
||
** works only for symmetric protocols
|
||
** wouldn't cover lots of problems
|
||
* testing against wireshark
|
||
** wireshark often way more tolerant than spec
|
||
* custom implementation
|
||
** in Python (e.g. using scapy)
|
||
** in Erlang (good binary encoder/decoder) or other languages
|
||
* specific tools like packetdrill
|
||
|
||
== Protocol Testing
|
||
|
||
Personal story: During past years,
|
||
|
||
* I implemented tons of [telecom] protocols / stacks at Osmocom.org
|
||
* I was looking for better tools to help [automatic] testing
|
||
** primarily functional testing (correctness / conformance)
|
||
** not so much performance testing
|
||
* I figured Ideal test tool would...
|
||
** allow very productive and expressive way to describe encoding/decoding
|
||
** allow very convenient pattern matching on incoming messages
|
||
** allow exchange of messages asynchronously with implementation under test
|
||
* I stumbled on TTCN-3 occasionally and investigated
|
||
|
||
== The TTCN-3 Language
|
||
|
||
* domain-specific language *just* for protocol conformance tests
|
||
* TTCN history back to 1983 (!), TTCN-3 since 2000
|
||
* used extensively in classic telecom sector (Ericsson, Nokia, etc.)
|
||
* ETSI developed and published abstract test suites in TTCN-3 for
|
||
** IPv6, SIP, DIAMETER, ePassports, Digital Mobiel Radio, 6LoWPAN
|
||
* Other bodies published test suites for
|
||
** CoAP, MQTT, MOST, AUTOSAR
|
||
|
||
But: Until 2015, only proprietary tools / compilers :(
|
||
|
||
== Eclipse TITAN
|
||
|
||
* After TTCN-3 specification in 2000, Ericsson internally develops TTCN-3 toolchain
|
||
* adopted for many Ericsson-internal testing of all kinds of products
|
||
* proprietary software with commercial licenses
|
||
* 300,000 lines of Java + 1.6 Million lines of C++
|
||
* Released as Open Source as "Eclipse TITAN" in 2015
|
||
** Not just TTCN-3 compiler, but also extensive documentations and many protocol modules, test ports as well as Eclipse IDE, Log file viewer/visualizer, etc.
|
||
* `eclipse-titan` part of standard Debian / Ubuntu archive, only one apt-get away
|
||
|
||
Great, we can finally use TTCN-3 in FOSS!
|
||
|
||
== Eclipse TITAN compiler workflow
|
||
|
||
[graphviz]
|
||
----
|
||
digraph G {
|
||
progra [label="Human Developer"];
|
||
ttcn3 [label="TTCN-3 source (ATS)"];
|
||
cpp [label="Generated C++ source"];
|
||
exec [label="Binary Executable (ETS)"];
|
||
tps [label="Other C++ sources, as needed"];
|
||
progra -> ttcn3 [label="writes code"];
|
||
ttcn3 -> cpp [label="ttcn3_compiler"];
|
||
cpp -> exec [label="GNU gcc / g++"];
|
||
tps -> cpp
|
||
}
|
||
----
|
||
|
||
* TITAN actually _compiles_ into executable binaries, it is not using a VM or scripting
|
||
** ATS: Abstract Test Suite (source code)
|
||
** ETS: Executable Test Suite (executable code)
|
||
|
||
== TTCN-3 Language Features (with TITAN)
|
||
|
||
* comprehensive type system
|
||
* parametric templates
|
||
* variety of encoders/decoders
|
||
* automatic / comprehensive logging framework
|
||
* powerful program control statements
|
||
* built-in notion of tests cases, test suites, verdicts, ...
|
||
* runtime / executor for parallel test components + aggregating results
|
||
|
||
== TTCN-3 Basic Types
|
||
|
||
* Simple basic types such as `integer`, `float`, `boolen`
|
||
* Basic string types such as `bitstring`, `octetstring`, `hexstring`, `charstring` (IA5) and `universal charstring` (UCS-4).
|
||
* Structured Types `record`, `set`, `record of`, `set of`
|
||
* Verdict type `verdicttype`
|
||
** can have either value `none`, `pass`, `inconc`, `fail`, or `error`
|
||
** verdict can only _deteriorate_ (`pass` -> `fail`) but never improve (`error` -> `pass`)
|
||
** every test case implicitly has a verdict, no need to explicitly declare a variable of `verdicttype`
|
||
|
||
== TTCN-3 Structured Types
|
||
|
||
A structured type is an abstract type comprised of other types, whcih can be nested.
|
||
An example for a `record` type (similar to a C-language `struct`) is shown below
|
||
|
||
--------
|
||
type record MyMessageType {
|
||
integer field1 optional<1>,
|
||
charstring field2,
|
||
boolean field3
|
||
};
|
||
--------
|
||
<1> optional members may be present or not
|
||
|
||
== TTCN-3 Union Type
|
||
|
||
A union expresses a set of alternative types of which one alternative must be chosen.
|
||
|
||
--------
|
||
type union MyMessageUnion {
|
||
integer field1,
|
||
charstring field2,
|
||
};
|
||
--------
|
||
|
||
Difference to C-language union: `ischosen()` can be used to learn which of the union members is
|
||
chosen/defined!
|
||
|
||
|
||
== Not-used and omit
|
||
|
||
* until a variable or field of structured type is assigned, it is _unbound_
|
||
* whenever a _value_ is expected, TTCN-3 runtime will create an error for _unbound_
|
||
* in case of absence of optional fields, explicit `omit` value must be assigned!
|
||
|
||
== Sub-typing
|
||
|
||
Sub-typing can be used to further constrain a given type. Typical examples include constrained number ranges,
|
||
and string patterns
|
||
|
||
--------
|
||
type integer MyIntRange (1..100);
|
||
type integer MyIntRange8 (0..infinity);
|
||
type charstring MyCharRange (”k”..”w");
|
||
type charstring SideType (”left”, ”right”);
|
||
type integer MyIntListRange (1..5,7,9);
|
||
type record length(0..10) of integer RecOfInt;
|
||
type charstring CrLfTermStrin (pattern ”*\r\n”);
|
||
--------
|
||
|
||
== Templates
|
||
|
||
* Matching incoming messages against some kind of specification is one of the most common tasks in testing protocols
|
||
** some expected fields are static (message type)
|
||
** some expected fields are known (source address)
|
||
** some fields are chosen by sender (some identifier)
|
||
** some fields we don't care (optional headers that may or may not be present)
|
||
* TTCN-3 Templates provide elegant solution for this, avoiding any explicit code to be written
|
||
** templates can even be parametric, i.e. they can be instantiated with "arguments"
|
||
* templates can also be used for sending messages, if they are fully specified/qualified
|
||
|
||
== Templates
|
||
|
||
--------
|
||
// Value list template
|
||
template charstring tr_SingleABorC := (”A”, ”B”, ”C”);
|
||
--------
|
||
|
||
--------
|
||
// Value range
|
||
template float tr_NearPi := (3.14 .. 3.15);
|
||
template integer tr_FitsToOneByte := (0 .. 255);
|
||
template integer tr_GreaterThanZero := (1 .. infinity);
|
||
--------
|
||
|
||
--------
|
||
// Intermixed value list and range matching
|
||
template integer tr_Intermixed := ((0..127), 200, 255);
|
||
--------
|
||
|
||
== Matching inside values
|
||
|
||
--------
|
||
// Using any element matching inside a bitstring value
|
||
// Last 2 bits can be '0' or '1'
|
||
template bitstring tr_AnyBSValue := ’101101??’B;
|
||
--------
|
||
|
||
--------
|
||
// Matches charstrings with the first character "a"
|
||
// and the last one "z"
|
||
template charstring tr_0 := pattern "a*z";
|
||
--------
|
||
|
||
* more capabilities using `complement`, `ifpresent`, `subset`, `superset`, `permutation` constructs not
|
||
covered here
|
||
|
||
|
||
== Parametric Templates
|
||
|
||
See below for an example of a parametric template:
|
||
|
||
--------
|
||
type record MyMessageType {
|
||
integer field1 optional,
|
||
charstring field2,
|
||
boolean field3
|
||
};
|
||
|
||
template MyMessageType trMyTemplte(boolean pl_param) := {
|
||
field1 : = ?, // present, but any value
|
||
field2 : = (”B”, ”O”, ”Q”) ,
|
||
field3 := pl_param
|
||
};
|
||
--------
|
||
|
||
The built-in `match()` function can be used to check if a given value matches a given template. Some TTCN-3
|
||
statements such as `receive()` have built-in capabilities for template matching, avoiding even the explicit
|
||
call of `match()` in many cases.
|
||
|
||
== Template Hierarchy
|
||
|
||
Using modified templates, one can build a hierarchy of templates: From the specific to the unspecific
|
||
|
||
----
|
||
template MyMsgType t_MyMsgAny := {
|
||
msg_type := ?,
|
||
foo := bar
|
||
};
|
||
|
||
template MyMsgType t_MyMsg23 modifies t_MyMsgAny := {
|
||
msg_type := 23,
|
||
};
|
||
----
|
||
where
|
||
|
||
* _t_MyMsgAny_ matches a message with any message type and "foo=bar", while
|
||
* _t_MMyMsg23_ matches only those that have "foo=bar" and "msg_type=23"
|
||
|
||
== Encoders/Decoders
|
||
|
||
* type system, templates, matching are all nice and great, but we need to get data from wire format into
|
||
TTCN-3 abstract types
|
||
* TTTCN-3 specifies importing of formal schema definitios, such as ASN.1, IDL, XSD (XML) and JSON
|
||
* TITAN has additional codecs for those (many) protocols that lack formal syntax
|
||
** `raw` codec for binary protocols (e.g. GTP)
|
||
** `text` codec for text based protocols (e.g. HTTP, MGCP, IMAP, ...)
|
||
* codecs allow you to _express/describe_ the format (declarative programming) rather than the usual imperative approach
|
||
|
||
== TITAN raw codec: UDP Example
|
||
|
||
How to express an UDP header using TITAN raw codec
|
||
|
||
--------
|
||
type integer LIN2_BO_LAST (0..65535) with {
|
||
variant ”FIELDLENGTH(16), COMP(nosign), BYTEORDER(last)”
|
||
};
|
||
type record UDP_header {
|
||
LIN2_BO_LAST srcport,
|
||
LIN2_BO_LAST dstport,
|
||
LIN2_BO_LAST len,
|
||
LIN2_BO_LAST cksum
|
||
} with { variant ”FIELDORDER(msb)” };
|
||
type record UDP packet {
|
||
UDP_header header
|
||
octetstring payload
|
||
} with {
|
||
variant (header) ”LENGTHTO(header, payload), LENGTHINDEX(len)”
|
||
};
|
||
--------
|
||
|
||
== TITAN raw codec: GTP Example
|
||
|
||
How to express an GTP header using TITAN raw codec
|
||
|
||
--------
|
||
type record GRE_Header {
|
||
BIT1 csum_present,
|
||
BIT1 rt_present,
|
||
BIT1 key_present,
|
||
...
|
||
OCT2 protocol_type,
|
||
OCT2 checksum optional,
|
||
OCT2 offset optional,
|
||
OCT4 key otional,
|
||
...
|
||
} with {
|
||
variant (checksum) "PRESENCE(csum_present='1', rt_present='1'B)"
|
||
variant (offset) "PRESENCE(csum_present='1'B, rt_present='1'B)"
|
||
variant (key) "PRESENCE(key_present='1'B)"
|
||
}
|
||
--------
|
||
|
||
|
||
== TITAN text codec: MGCP Example
|
||
|
||
--------
|
||
type charstring MgcpVerb ("EPCF", "CRCX", "MDCX", "DLCX", "RQNT", "NTFY",
|
||
"AUEP", "AUCX", "RSIP") with {
|
||
variant "TEXT_CODING(,convert=upper_case,,case_insensitive)"
|
||
};
|
||
type charstring MgcpTransId (pattern "\d#(1,9)");
|
||
type charstring MgcpEndpoint (pattern "*@*");
|
||
type charstring MgcpVersion (pattern "\d.\d") with {
|
||
variant "BEGIN('MGCP ')"
|
||
};
|
||
type record MgcpCommandLine {
|
||
MgcpVerb verb,
|
||
MgcpTransId trans_id,
|
||
MgcpEndpoint ep,
|
||
MgcpVersion ver
|
||
} with {
|
||
variant "SEPARATOR(' ', '[\t ]+')"
|
||
variant "END('\r\n', '([\r\n])|(\r\n)')"
|
||
};
|
||
--------
|
||
|
||
== Program Control Statements
|
||
|
||
* `if` / `else` like in C
|
||
* `select` statement similar to C `switch`
|
||
* `for`, `while`, `do-while` loops like in C
|
||
* `goto` and `label`
|
||
* `break` and `continue` like in C
|
||
|
||
== Abstract Communications Operations
|
||
|
||
* TTCN-3 test suites communicate with _implementation under test_ through abstract TestPorts
|
||
** TestPorts can be implemented in TTCN-3 or C++ and linked in
|
||
** TestPorts must be _connected_ before using send/receive operaitons
|
||
** TITAN provides TestPorts for e.g. packet socket, IP/UDP/TCP/SCTP socket, ...
|
||
* `<port>.send(<ValueRef>)` performs non-blocking send
|
||
** Literal value, constant, variable, specific value template, ...
|
||
* `<port>.receive(<TemplateRef>)` or `<port>.receive` performs blocking receive
|
||
** literal value, constant, variable, template (with matching!), inline template
|
||
|
||
'... but if receive blocks, how can we wait for any of N events?
|
||
|
||
== Program Control and Behavior
|
||
|
||
* program statements are executed in order
|
||
* blocking statements block the execution of the component
|
||
* occurrence of unexpected event may cause infinite blocking
|
||
|
||
----
|
||
// x must be the first on queue P, y the second
|
||
P.receive(x); // Blocks until x appears on top of queue P
|
||
P.receive(y); // Blocks until y appears on top of queue P
|
||
// When y arrives first then P.receive(x) blocks -> error
|
||
----
|
||
|
||
This is what leads to the `alt` statement:
|
||
`alt` declares a seto alternatives covering all events, which
|
||
|
||
* can happen: expected messages, timeouts, ...
|
||
* must not happen: unexpected faulty messages, no message received, ...
|
||
* all alternatives inside `alt` are blocking operations
|
||
|
||
== The `alt` statement
|
||
|
||
----
|
||
P.send(req)
|
||
T.start;
|
||
// ...
|
||
alt {
|
||
[] P.receive(resp) { /* actions to do and exit alt */ }
|
||
[] any port.receive { /* handle unexpected event */ }
|
||
[] T.timeout { /* handle timer expiry and exit */ }
|
||
}
|
||
----
|
||
|
||
* [] is guard condition enables or disables the alternative
|
||
** usually empty `[]` equals `[true]`
|
||
** can contain a condition like `[x > 0]`
|
||
** very good for e.g. state machines to activate some alternatives only in certain states while others may
|
||
occur in any state
|
||
|
||
|
||
== The `alt` and `repeat` statements
|
||
|
||
The `repeat` statement
|
||
|
||
* takes a new snapshot and re-evaluates the alt statement
|
||
* can appear as last statement in statement blocks of statements
|
||
|
||
----
|
||
P.send(req)
|
||
T.start;
|
||
alt {
|
||
[] P.receive(resp) { /* actions to do and exit alt */ }
|
||
[] P.receive(keep_alive) { /* handle keep alive message */
|
||
repeat }
|
||
[] any port.receive { /* handle unexpected event */ }
|
||
[] T.timeout { /* handle timer expiry and exit */ }
|
||
}
|
||
----
|
||
|
||
== TTCN-3 modules
|
||
|
||
TTCN-3 code is written in _modules_
|
||
|
||
* a test suite consists of one or more modules
|
||
* a module contains _module definitions_ and an optional _control part_
|
||
** _parameters_ (automatically configurable via config file)
|
||
** definition of _data types_, _constants_, _templates_
|
||
** definition of _communications ports_
|
||
** definition of _test components_, _functions_ _altstesp_ and _test cases_
|
||
** _control part_ determines default order/execution of test cases
|
||
* modules can import from each other (think in python terms)
|
||
|
||
== Examples
|
||
|
||
Let's have a look at some real-world examples and do a bit of a walk-through
|
||
before continuing with the slides...
|
||
|
||
== Logging
|
||
|
||
* TITAN runtime contains extensive logging framework
|
||
* config file determines log level for various different subsystems
|
||
** e.g. any encode, decode, receive, transmit operations logged
|
||
** timer starts, expirations
|
||
** any changes to test case verdict
|
||
* explicit logging from code by use of `log()` built-in function
|
||
* `ttcn3_logformat` tool for pretty-printing log files
|
||
* `ttcn3_logmerge` tool for merging/splicing multiple logs
|
||
* log plugins e.g. for generating JUnit-XML available
|
||
** facilitates easy reporting / integration to Jenkins or other CI
|
||
|
||
== Logging
|
||
|
||
Log file format example:
|
||
|
||
----
|
||
// abstract data type before encode
|
||
13:30:41.243536 Sent on GTPC to system @GTP_CodecPort.Gtp1cUnitdata : { peer := { connId := 1, remName := "127.0.23.1", remPort := 2123 }, gtpc := { pn_bit := '0'B, s_bit := '1'B, e_bit := '0'B, spare := '0'B, pt := '1'B, version := '001'B, messageType := '01'O, lengthf := 0, teid := '00000000'O, opt_part := { sequenceNumber := '3AAC'O, npduNumber := '00'O, nextExtHeader := '00'O, gTPC_extensionHeader_List := omit }, gtpc_pdu := { echoRequest := { private_extension_gtpc := omit } } } }
|
||
|
||
// 'msg' contains encoded binary data actually sent via socket
|
||
13:30:41.243799 Outgoing message was mapped to @IPL4asp_Types.ASP_SendTo : { connId := 1, remName := "127.0.23.1", remPort := 2123, proto := { udp := { } }, msg := '32010004000000003AAC0000'O }
|
||
----
|
||
|
||
== Logging
|
||
|
||
The same log file lines if run through `ttcn3_logformat`
|
||
|
||
----
|
||
13:30:41.243536 Sent on GTPC to system @GTP_CodecPort.Gtp1cUnitdata : {
|
||
peer := {
|
||
connId := 1,
|
||
remName := "127.0.23.1",
|
||
remPort := 2123
|
||
},
|
||
gtpc := {
|
||
pn_bit := '0'B,
|
||
s_bit := '1'B,
|
||
e_bit := '0'B,
|
||
spare := '0'B,
|
||
pt := '1'B,
|
||
version := '001'B,
|
||
messageType := '01'O,
|
||
lengthf := 0,
|
||
teid := '00000000'O,
|
||
opt_part := {
|
||
sequenceNumber := '3AAC'O,
|
||
npduNumber := '00'O,
|
||
nextExtHeader := '00'O,
|
||
gTPC_extensionHeader_List := omit
|
||
},
|
||
gtpc_pdu := {
|
||
echoRequest := {
|
||
private_extension_gtpc := omit
|
||
}
|
||
}
|
||
}
|
||
}
|
||
13:30:41.243799 Outgoing message was mapped to @IPL4asp_Types.ASP_SendTo : {
|
||
connId := 1,
|
||
remName := "127.0.23.1",
|
||
remPort := 2123,
|
||
proto := {
|
||
udp := { }
|
||
},
|
||
msg := '32010004000000003AAC0000'O
|
||
}
|
||
----
|
||
|
||
|
||
== Existing TITAN Source
|
||
|
||
* Protocol encoding/decoding
|
||
** BSSAP+, BSSGP, BSSMAP, CoAP, DSS1, DUA, EAP, GRE, GTP, HTTP, ISUP, LLC, M2PA, M2UA, MQTT, MongoDB, NDP, NS,
|
||
NTAF, ROSE, SCTP, SDP, SNDCP, STOMP, STUN, SUA, TLS, WTP, DNS, IP, SMPP, SNMP, IKEv2, DHCP, PPP, RTP, TCP,
|
||
UDP, XMPP, DHCPv6, SMTP, ICMP, RTSP, ICMPv6, DIAMETER, FrameRelay, ProtoBuff, IUA, L2TP, M3UA, MIME,
|
||
WebSocket, H.248, IMAP, IPsec, SRTP, MSRP, ICAP, RADIUS
|
||
* Protocol Emulation
|
||
** M3UA, SCCP, SUA
|
||
* Test Ports
|
||
** GPIO, MTP3, Serial, SocketCAN, SCTP, SIP, HTTP, Telnet, UDP, pcap file, pipe, SQL, TCP, SUNRPC, SSH, STDINOUT, sockets, LDAP
|
||
|
||
|
||
== Further Reading
|
||
|
||
* Ericsson TTCN-3 tutorial http://www.ttcn-3.org/files/TTCN3_P.pdf
|
||
* An Introduction to TTCN-3, 2nd Edition <http://www.wiley.com/go/willcock_TTCN-3_2e>
|
||
* Modules https://github.com/eclipse
|
||
* More Modules http://git.eclipse.org/
|
||
* Debian https://packages.debian.org/search?keywords=eclipse-titan
|
||
* Ubuntu https://packages.ubuntu.com/search?keywords=eclipse-titan
|
||
|
||
== EOF
|
||
|
||
End of File
|