ttcn3 paper: final version

2017/ttcn-netdev2.2/paper/isea.tex

@@ -20,7 +20,7 @@
 /Author (Harald Welte)}
 % The file isea.sty is the style file for ISEA 2015 proceedings.
 %
-\title{TTCN-3 and Eclipse TITAN for testing protocol stacks [draft]}
+\title{TTCN-3 and Eclipse TITAN for testing protocol stacks}
 \author{Harald Welte\\
 sysmocom - systems for mobile communications GmbH\\
 Berlin, Germany\\
@@ -181,7 +181,7 @@ as {\tt universal charstring} (UCS-4).
 Using the {\tt record}, {\tt set}, {\tt union}, {\tt record of} and {\tt set of} TTCN-3 structured types,
 programmers can create abstract container types.
 
-FIXME: example
+% FIXME: example
 
 TTCN-3 also knows a {\tt enumberated} type, like many other programming languages.
 
@@ -240,7 +240,7 @@ template MyMessageType tr_MyTemplate
 Templates can be hierarchical, so from the most generic to the most specific case, you can specify ever more
 concrete templates, depending on the need.
 
-FIXME: Example
+% FIXME: Example
 
 Templates can further be specified as {\tt subset}, {\tt superset}, {\tt permutation} (any order of elements).
 
@@ -249,13 +249,13 @@ against one or multiple receive templates.  Matching against a fixed template wo
 over e.g. comparing with a 'const struct' in a C-language test case.  However, TTCN-3 templates can have
 wild-cards and pattern matching.
 
-FIXME: Example
+% FIXME: Example
 
 
 The built-in {\tt match()} function can be used to determine if a received
 message (or actually any value) matches the template.  Even more so, the fundamental {\tt receive()} function,
 which is used to receive any inbound message from a test port, has built-in matching capability, so explicit
-calls of {\tt match()} are rarely required.  See FIXME later.
+calls of {\tt match()} are rarely required.  See Section \ref{receive} below.
 
 
 \subsection{TTCN-3 + TITAN encoders/decoders}
@@ -331,10 +331,11 @@ type record GRE_Header {
 Much more complex constructs are possible in the TTIAN binary codec, e.g. the extension octet concept found in
 many telecom protocols.
 
-FIXME: examples for a text based protocol
+% FIXME: examples for a text based protocol
 
 
 \subsection{Abstract Communications Operations}
+\label{receive}
 
 Abstract communications happens on the {\em test ports} which connect the test case of the abstract test suite
 (ATS) with the implementation under test (IUT).  TTCN-3 supports  abstract communications operations for both
@@ -413,21 +414,51 @@ interleave {
 }
 \end{lstlisting}
 
+\subsection{Fuzzing extensions}
 
-FIXME: synchronous API calls example
+As the TTCN-3 type notation contains stringent information about what values are permitted, it is normally not
+possible to create illegal values.  TITAN introduces the {\tt erroneous} extension which can be usd to
+generate invalid messages, e.g. those missing mandatory fields, with invalid values, etc.
 
-FIXME: diagram about netfilter conntrack test suite
+\section{Test execution}
 
-FIXME: test execution
+The test suite, comprised of any number of individual test cases is executed by the TITAN executor.  The
+executor takes care of starting any of the parallel test components (on the local host or even on remote
+hosts), performing the test cases as indicated by either the command line, or by the configuration file, or in
+absence of that by the compile-time default list of testcases in the ({\tt control}) section of the TTCN-3
+code.  It also manages opening all configured log files / plugins, as described below.  The test executor
+is typically started using the {\tt ttcn3\_start} program.
 
-FIXME: configuration file / moduleparam
+\subsection{Configuration File}
 
-FIXME: logfile, including match error
+Every TITAN test suite has a configuration file, where many aspects of test execution can be specified.  Among
+others, you can configure the log verbosity of the TITAN-internal logging.  There are different log masks
+for the log file ({\tt FileMask}) and for the console ({\tt FileMask}).  One extremely useful feature is
+the {\tt TTCN\_MATCHING} logging, which will provide detailed information about the exact (even nested) field
+of a value that did not match in any explicit or implicit matching against a template.  Another useful example
+is {\tt TTCN\_ENCDEC}, which will automatically log the input and output of every encode/decode function,
+which is very useful during debugging of codec problems.
 
-FIXME: Junit-XML output plugin
+\subsection{Log Files}
 
+TITAN writes log files in a structured format, which makes them easy to parse by downstream log-processing
+tools.  Every parallel test component will write its own log file.  As logs are timestamped, the {\tt
+ttcn3\_logmerge} tool can be used to {\em splice} the individual log files into each other, providing one
+log file of all components, sorted by timestamps.
+
+The {\tt ttcn3\_logformat} tool can be used to convert the single-line nested value representation into a
+multi-line format with proper indenting which is more easy to understand as a human.  Compare this to e.g.
+JSON or XML pretty-printing.
+
+\subsection{Junit-XML output plugin}
+
+The TITAN runtime understands the concept of log output plugins.  Such plugins can be custom-developed to
+generate whatever output format from test case logs / results as needed.   A number of standard plugins are
+included, among them the capability to write Junit-XML output, which is a standard format for reporting test
+results in a machine-readable format.  This format is understood by the popular Jenkins continuous integration
+software, so that one can easily feed per-testcase results into the Jenkins test results analyzer as part of a
+CI test suite.
 
-FIXME: fuzzing extensions
 
 \section{Acknowledgments}
 The author acknowledges that like everyone in Free Software, he is merely standing on the shoulders of