From be39389a69af4b37ece52b16dd4e38edbe580ed2 Mon Sep 17 00:00:00 2001 From: Harald Welte Date: Mon, 14 Nov 2022 18:30:52 +0100 Subject: [PATCH] 2021/pysim_shell: typos/fixed + Add html rendering --- .../osmodevcon2019-osmo_small_hardware.adoc | 4 + 2021/osmodevcall-gsmr/osmodevcall-gsmr.adoc | 1 + 2021/osmodevcall-pysim_shell/pysim_shell.adoc | 11 +- 2021/osmodevcall-pysim_shell/pysim_shell.html | 4642 +++++++++++++++++ 4 files changed, 4657 insertions(+), 1 deletion(-) create mode 100644 2021/osmodevcall-pysim_shell/pysim_shell.html diff --git a/2019/osmodevcon2019-osmo_small_hardware/osmodevcon2019-osmo_small_hardware.adoc b/2019/osmodevcon2019-osmo_small_hardware/osmodevcon2019-osmo_small_hardware.adoc index 2d60ed6..50c7b31 100644 --- a/2019/osmodevcon2019-osmo_small_hardware/osmodevcon2019-osmo_small_hardware.adoc +++ b/2019/osmodevcon2019-osmo_small_hardware/osmodevcon2019-osmo_small_hardware.adoc @@ -66,6 +66,10 @@ image::sfp-experimenter1.jpg[width="100%"] image::osmo-clock-gen-v1-top.jpg[width="100%"] +== `osmo-clock-gen` future use case + +image::DSC00013.JPG[width="100%"] + == `osmo-clock-conv` * low phase noise, dual-output sine -> square converter diff --git a/2021/osmodevcall-gsmr/osmodevcall-gsmr.adoc b/2021/osmodevcall-gsmr/osmodevcall-gsmr.adoc index ba6ec53..facde7f 100644 --- a/2021/osmodevcall-gsmr/osmodevcall-gsmr.adoc +++ b/2021/osmodevcall-gsmr/osmodevcall-gsmr.adoc @@ -188,6 +188,7 @@ eMLPP:: Enhanced Multi-Level Precedence and Pre-emption * traditionally uses GSM CSD calls with transparent, async teleservice ** network required to support 2.4, 4.8 and 9.6 kbps * EURORADIO layer inside the CSD call +** well, Euroradio in X.224 in T.70 in CSD :) * most recently also possible via TCP over GPRS diff --git a/2021/osmodevcall-pysim_shell/pysim_shell.adoc b/2021/osmodevcall-pysim_shell/pysim_shell.adoc index d6c9286..720be53 100644 --- a/2021/osmodevcall-pysim_shell/pysim_shell.adoc +++ b/2021/osmodevcall-pysim_shell/pysim_shell.adoc @@ -1,4 +1,4 @@ -pySim-shell - next genreation SIM configuration tool +pySim-shell - next generation SIM configuration tool ==================================================== :author: Harald Welte :copyright: 2021 by Harald Welte (License: CC-BY-SA) @@ -97,6 +97,7 @@ What to do about the user interface? * supports output redirection to local files * supports piping output through external programs ("| grep foo") * supports `argparse` to define commands and their options +* supports [persistent] command history * allows to dynamically register and unregister commands * allows full control over tab completion items. @@ -219,6 +220,14 @@ Non-Python commands can be issued with: app("your command") * adding API documentation at least for all of the *new* classes/methods that are used to build pySim-shell, and which you need to extend to cover more files * automatic generation of pySim-shell command reference from `argparse` instances +* keep your ADM1 pins per ICCID in a CSV file and automatically verify them + +== pySim-shell status + +* many, if not most SIM/USIM/ISIM files defined +* only few of them have JSON decoders and encoders yet +* works for interactive use cases +* not yet full feature parity with pySim-prog (no batch programming, ...) == Call for Contributions diff --git a/2021/osmodevcall-pysim_shell/pysim_shell.html b/2021/osmodevcall-pysim_shell/pysim_shell.html new file mode 100644 index 0000000..edd94f5 --- /dev/null +++ b/2021/osmodevcall-pysim_shell/pysim_shell.html @@ -0,0 +1,4642 @@ + + + + +pySim-shell - next generation SIM configuration tool + + + + + + + + +
+

why program your own SIM cards?

+
+
    +
  • + +because you want to operate your own cellular network (2G, 3G, 4G, 5G) + +
      +
    • + +whether FOSS (Osmocom CNI, srsLTE, nextepc/open5gs, free5gc, OpenBTS, yateBTS, …) + +
      • +
    • + +but also proprietary (e.g. Amarisoft) + +
      • +
    +
    • +
  • + +2G can use random 3rd party cards, but it’s not really convenient in most cases + +
    • +
  • + +Anything >= 3G requires you to know the card-specific secrete keys + +
      +
    • + +in reality that translates to having your own programmable SIM cards + +
      • +
    +
    • +
+
+
+
+

some smartcard terminology

+
+
    +
  • + +card filesystem + +
      +
    • + +MF (Master File): The root directory + +
      • +
    • + +DF (Dedicated File): A subdirectory + +
      • +
    • + +ADF (Application Dedicated File): Directory of an application (like USIM, ISIM) + +
      • +
    • + +EF (Entry File): A regulare file + +
        +
      • + +Transparent EF: An unstructured file + +
        • +
      • + +Linear Fixed EF: An file consisting of fixed-length records + +
        • +
      +
      • +
    +
    • +
+
+
+
+

specs vs. proprietary

+
+
    +
  • + +SIM cards are fully specified by a combination of ISU, ETSI and 3GPP specs + +
    • +
  • + +this covers only the operation after the card has been issued, e.g. + +
      +
    • + +reading and writing files accessible without PIN auth, after PIN auth + +
      • +
    • + +performing GSM authentication or UMTS AKA + +
      • +
    +
    • +
  • + +it does not cover how the card is issued/provisioned + +
      +
    • + +secret key material (Ki / K / OP / OPc) is not readable from the card + +
      • +
    • + +it is an implementation detail on how the card manufacturer writes those to the card + +
      • +
    +
    • +
  • + +this leads to the need for card-specific support / code in software like pySim + +
    • +
+
+
+
+

pySim "classic"

+
+
    +
  • + +originally developed in 2009 by Sylvain Munaut + +
    • +
  • + +Linux/FOSS tool for configuration of SuperSIM/MagicSIM + +
    • +
  • + +replicated what proprietary vendor tools did to configure IMSI, Ki, … + +
    • +
  • + +those ultra-cheap Chinese SIM cards at the time (from the SIM card cloning days of COMP128v1) shipped with binary-only windows programs + +
    • +
+
+
+
+

pySim "classic" fundamentals

+
+
    +
  • + +library code + +
      +
    • + +transport - how to exchange APDUs with card + +
      • +
    • + +commands - encoding of ISO7816-4 / TS 51.011 / ETS 102 221 commands + +
      • +
    • + +cards - classes implementing various different programmable SIM cards + +
      • +
    +
    • +
  • + +executable programs + +
      +
    • + +pySim-prog for card programming + +
      • +
    • + +pySim-read for card reading + +
      • +
    +
    • +
+
+
+
+

pySim transports

+
+
    +
  • + +transport - how to exchange APDUs with card + +
      +
    • + +serial transport for serial readers (UART Rx+Tx connected to I/O) + +
      • +
    • + +pcsc transport for pcsc-lite supported readers (e.g. USB CCID) + +
      • +
    +
    • +
  • + +some more exotic options added later on by Vadim Yanitskiy: + +
      +
    • + +calypso for using L1CTL to talk to OsmocomBB on Ti Calypso phone + +
      • +
    • + +modem_atcmd for using AT+CSIM commands talking to a cellular modem + +
      • +
    +
    • +
+
+
+
+

advanced pySim "classic" features

+
+
    +
  • + +batch programming of cards, using CSV file input + +
    • +
  • + +even with ability to execute external tool for driving motorized card readers + +
    • +
+
+
+
+

Limitations of pySim "classic"

+
+
    +
  • + +supports only very few files/parameters that can be programmed + +
      +
    • + +IMSI, MSISDN, Ki and SMSP was sufficient for making classic GSM SIMs work on your own network + +
      • +
    • + +today, people want to configure VoLTE/IMS parameters, 5G SUCI stuff, etc. + +
      • +
    +
    • +
  • + +adding more fields to pySim-classic makes it unusable: Who wants to specify 250 different fields as command line arguments? + +
    • +
+
+
+
+

Usability of pySim "classic"

+
+
    +
  • + +it was designed to program a batch of cards with defaults and random keys + +
    • +
  • + +there are all kinds of automagic bits happening if not all fields are specified + +
    • +
  • + +pySim at least originally assumed that you program a full card, and not that you perform incremental small changes + +
    • +
  • + +no user manual, very limited RADME and wiki instructions + +
    • +
+
+
+
+

the road to pySim-shell

+
+

What to do about the user interface?

+
    +
  • + +GUIs are bad, they are not really scriptable/automatizable + +
    • +
  • + +GUIs are hard to portable, test automatically, etc. + +
    • +
  • + +people setting up FOSS cellular programs most likely use bash/zsh/tcsh on daily basis, so they are familiar with the concept of interactive shells / CLIs + +
    • +
  • + +so let’s do a CLI / shell / REPL! + +
    • +
+
+
+
+

pySim-shell and cmd2

+
+
    +
  • + +uses cmd2 python module for the shell + +
      +
    • + +extremely powerful framework for building CLI/shell type interfaces + +
      • +
    • + +allows us to focus on implementing logic (like individual commands), not the shell itself + +
      • +
    +
    • +
  • + +supports output redirection to local files + +
    • +
  • + +supports piping output through external programs ("| grep foo") + +
    • +
  • + +supports argparse to define commands and their options + +
    • +
  • + +supports [persistent] command history + +
    • +
  • + +allows to dynamically register and unregister commands + +
    • +
  • + +allows full control over tab completion items. + +
    • +
+
+
+
+

pySim-shell UI fundamentals

+
+
    +
  • + +some global commands for things like PIN/CHV verification, enabling, disabling, unblcoking + +
    • +
  • + +you select your way through the card filesystem, like cd on bash/zsh/… + +
      +
    • + +tab-completion for [known] file names + +
      • +
    +
    • +
+
+
Selecting a file
+
+
pySIM-shell (MF/ADF.USIM)> select EF.IMSI
+{
+    "file_descriptor": {
+        "shareable": true,
+        "file_type": "working_ef",
+        "structure": "transparent"
+    },
+    "file_identifier": "6F07",
+    "proprietary_info": {
+        "proprietary_D0": "20",
+        "proprietary_D2": "0F"
+    },
+    "life_cycle_status_int": "operational_activated",
+    "security_attrib_ref_expanded": "6F0603",
+    "file_size": 9,
+    "short_file_id": "38"
+}
+pySIM-shell (MF/ADF.USIM/EF.IMSI)>
+
+
+
+
+

pySim-shell reading/writing

+
+
    +
  • + +context-sensitive action commands depend on the type of file selected + +
      +
    • + +read_binary, update_binary, … for transparent EF + +
      • +
    • + +read_record, update_record, … for linear fixed EF + +
      • +
    +
    • +
+
+
Reading and updating a transparent EF
+
+
pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary
+089910070000400300
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> update_binary 089910070000400301
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary
+089910070000400301
+
+
+
+
+

pySim-shell decoding/encoding

+
+
    +
  • + +working with hex-encoded binary data is not very convenient + +
    • +
  • + +each specified file in the filesystem is represented by a python class + +
      +
    • + +class can provide encoder + decoder methods to go from binary to JSON and vice-versa + +
      • +
    +
    • +
  • + +some shell commands combine reading a file from the card + decoding / encoding + +
      +
    • + +read_binary_decoded, write_binary_decoded, … for transparent EF + +
      • +
    • + +read_record_decoded, write_record_decoded, … for linear fixed EF + +
      • +
    +
    • +
+
+
Reading and updating a decoded transparent EF
+
+
pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary_decoded
+{
+    "imsi": "901700000043000"
+}
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> update_binary_decoded '{"imsi": "901700000043999"}'
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary_decoded
+{
+    "imsi": "901700000043999"
+}
+
+
+
+
+

pySim-shell editing of single parameters

+
+
    +
  • + +many files contain a lot of parameters + +
    • +
  • + +entering all of them as a single line in update_binary_decoded becomes rather cumbersome + +
    • +
  • + +edit_binary_decoded and edit_record_decoded to the rescue + +
      +
    • + +reads data from the card + +
      • +
    • + +decodes it to JSON + +
      • +
    • + +spawns whatever is your favorite text editor + +
      • +
    • + +you do your editing, save + exit + +
      • +
    • + +re-encodes the modified JSON + +
      • +
    • + +writes the result to the card + +
      • +
    +
    • +
+
+
+
+

pySim-shell export

+
+
    +
  • + +one-shot command to iterate over all [known] files below the current DF and dump them + +
    • +
  • + +output format is structured in a way that you can use it as a pySim-shell script + +
    • +
  • + +results in easy method to backup and restore a card + +
    +
    +NOTE +
    +
    +

    +only the ETSI/3GPP standard files that are covered by pySim classes are exported. There may be any +number of proprietary additional files on the card. Particularly K/OP/OPc or OTA keys are not exported this +way +

    +
    +
    +
    • +
+
+
+
+

pySim-shell advanced cmd2 usage

+
+
    +
  • + +run_script: execute any text file as script inside the pySim-shell + +
    • +
  • + +run_pyscript: execute a python script with access to all of the state / classes / methods + +
    • +
  • + +ipy: run an interactive IPython session + +
    • +
  • + +py: run an interactive python interpreter + +
      +
    • + +card represents the pySim.cards.Card instance + +
      • +
    • + +rs represents the pySim.filesystem.RuntimeState + +
      • +
    +
    • +
+
+
Using interactive python shell with access to pySim-shell state
+
+
pySIM-shell (MF)> py
+Python 3.9.2 (default, Feb 28 2021, 17:03:44)
+[GCC 10.2.1 20210110] on linux
+Type "help", "copyright", "credits" or "license" for more information.
+
+End with `Ctrl-D` (Unix) / `Ctrl-Z` (Windows), `quit()`, `exit()`.
+Non-Python commands can be issued with: app("your command")
+
+>>> rs.selected_file
+<pySim.filesystem.CardMF object at 0x7f97952f3070>
+>>> rs.selected_file.desc
+'Master File (directory root)'
+
+
+
+
+

Other recent pySim developments

+
+
    +
  • + +introduction of sphinx for generating documentation + +
    • +
  • + +adding API documentation at least for all of the new classes/methods that are used to build pySim-shell, + and which you need to extend to cover more files + +
    • +
  • + +automatic generation of pySim-shell command reference from argparse instances + +
    • +
  • + +keep your ADM1 pins per ICCID in a CSV file and automatically verify them + +
    • +
+
+
+
+

pySim-shell status

+
+
    +
  • + +many, if not most SIM/USIM/ISIM files defined + +
    • +
  • + +only few of them have JSON decoders and encoders yet + +
    • +
  • + +works for interactive use cases + +
    • +
  • + +not yet full feature parity with pySim-prog (no batch programming, …) + +
    • +
+
+
+
+

Call for Contributions

+
+
    +
  • + +pySim-shell is already very usable, but needs your help! + +
      +
    • + +most files still need encoder + decoder methods to be implemented + +
      • +
    • + +we need automatic testing of encoders + decoders + +
      • +
    • + +we want JSON schema for the JSON representation of each file + +
      • +
    • + +we want to have batch programming in pySim-shell + +
      • +
    +
    • +
+

So if you know a bit of python and care about SIM cards, please send your patches!

+
+
+
+

Further Reading

+
+ +
+
+
+

EOF

+
+

End of File

+
+
+ +