Commit Graph

719 Commits

Author SHA1 Message Date
Harald Welte 799c354827 shell: Proper argparser (for help + manual) activate_file
Change-Id: I5929ae3deff4d15b5db4a1d866576271c57a955f
2022-02-15 15:56:28 +01:00
Harald Welte 2bb17f3df9 pySim-shell: export: Add FCP template to export
The FCP template provides us a lot of context, like the permissions of
a given file.  Let's make it part of the 'export' output, both in raw
and in decoded form.

Change-Id: I05f17bbebd7a9b3535204b821900851a5f66e88f
Closes: OS#5457
2022-02-15 15:41:55 +01:00
Harald Welte 9e241435cc docs/legcay.txt: Point to pySim-shell as replacement
Change-Id: I9ca6b9d8c35e23be2ec8752107bb7d1e4f6f9bc1
2022-02-15 15:38:19 +01:00
Harald Welte 3c9b784825 pySim-shell: support TS 102 222 administrative commands
This adds support for creating/deleting and terminating files,
as well as support for permanent card termination.

Change-Id: I5b1ffb1334afa18d62beb642268066a30deb7ea6
2022-02-15 15:35:36 +01:00
Harald Welte 747a978478 ts_102_221: Implement File Descriptor using construct
This automatically adds encoding support, which is needed for upcoming
CREATE FILE support.

Change-Id: Ia40dba4aab6ceb9d81fd170f7efa8dad1f9b43d0
2022-02-15 15:35:36 +01:00
Harald Welte ee670bc1c6 pySim-shell: Allow selecting of deep paths like DF.GSM/EF.IMSI
With this patch applied, users can directly enter commands like

select DF.GSM/EF.IMSI or
select ADF.USIM/DF.5GS/EF.5GAUTHKEYS

This feature doesn't have tabl completion, so it's mostly useful
for when you know what to select, or for use within scripts.

Change-Id: I681a132eb2df4b2aba4c2ccbdd21c6d5b88443e3
2022-02-15 15:35:36 +01:00
Harald Welte 226b866f51 ts_31_103: TLV definitions for IMS, XCAP and MudMid configuration
Change-Id: I9a90ee978db668a70259eb48085ff5384cf696d6
2022-02-15 15:35:36 +01:00
Harald Welte 540adb0ee6 ts_51_011: EF_CMI: Decoder the alpha_id string
Change-Id: I45efe29ab98972945b4257229a995815f5632536
2022-02-15 15:35:36 +01:00
Harald Welte 1e73d228f4 ts_51_011: Convert EF_ADN and EF_ACC to Construct
this has the benefit of providing encoding support for free.

Change-Id: I31c118082e92892486c3688de2197c0c6dd2750e
2022-02-15 15:35:36 +01:00
Harald Welte bc0e209a9f ts_51_011: Proper decode of EF.SMSP
Full decode of the SSM Parameters File

Change-Id: Iac5bb87ed3350978dc8b207f052510fdba2e4883
2022-02-15 15:35:35 +01:00
Harald Welte 3bb516b2b1 Improve IST/UST check documentation (for the user manual)
Change-Id: I18093d795721f2e729eff858c8922edde9e84451
2022-02-15 15:35:35 +01:00
Harald Welte aceb2a548a ust_service_check: proper treatment of files in sub-directories
We must not only consider files in the current directory (ADF.USIM)
but also in its sub-directories.  This requires us to be able to
determine the path we need to traverse between the currently selected
file (EF.UST) and the respective file in some other directory,
which is implemented via CardFile.build_select_path_to().

Change-Id: I61797fefa9dafa36a8a62c11aa2cfaeecb015740
2022-02-15 15:35:35 +01:00
Harald Welte 419bb496e1 ts_31_102: service annotations for DF.{5GS,WLAN,HNB}
We had service annotations only for ADF.USIM so far, but not for
the related sub-directories.

Change-Id: Iaa56a26ba53eaf18fce14845ae07a27c52a2c58a
Note: The code doesn't make use of them in any reasonable way yet!
2022-02-15 15:35:35 +01:00
Harald Welte fa8b8d1160 ts_31_102: Use perror() instead of poutput() for errors
This adds colorization and ensures they go to stderr and not stdout

Change-Id: I34b8f974b4ff13002679c4700bdf604db7d7f3cd
2022-02-15 15:35:35 +01:00
Harald Welte 82f75c200f ts_31_102: Add more EF.UST checks to 'ust_service_check' command
* check for service dependencies listed in TS 31.102
* print number of errors encountered

Change-Id: Id47f8f2c8de299bbf91243d0c8900d22a7d35b10
2022-02-15 15:35:35 +01:00
Harald Welte d53918c3e1 filesystem: Fix CardMF.get_app_names()
This function was not used and doesn't work without this patch.

Change-Id: Id3dad7d97fe29a25792d2f8f0e879666c1d9c136
2022-02-15 15:35:35 +01:00
Harald Welte 6ca2fa7a5d Split EF.UST handling from EF.IST and EF.SST
The existing code had the following serious problems:
* when trying to update EF.SST or EF.IST, it would write to EF.UST !
* shell commands were called ust_* even for the EST/IST files

Let's introduce the proper separation between what is shared and what
is file-specific.

Change-Id: Ie55669ca37a4762fac9f71b1db528ca67056e8dd
2022-02-15 15:35:35 +01:00
Harald Welte 4c5e2310fa ts_31_102: Add "ust_service_check" command.
This command performs a consistency check between the services activated
in EF.UST/EF.IST and the files that should (or should not) be
active/selectable for the given service.

Produces output like:

Checking service No 48 (inactive)
  ERROR: File EF(EF.MWIS) is selectable but should not!
Checking service No 49 (active)
  ERROR: File EF(EF.CFIS) is not selectable (SW=6a82) but should!

Change-Id: Iea7166959e2015eb8fa34d86036560c9e42ce4d3
2022-02-15 15:35:35 +01:00
Harald Welte d16d904c57 README.md: Remove old usage examples, refer to user manual instead
We want people to use pySim-shell and should not mislead them by
having usage examples of old tools in README.md.  Also, all
documentation should be in the manuals, let's try to have bits
and pieces in various places.

Change-Id: I8c07a2e0778ab95fb42be6074acb80874e681d20
2022-02-15 15:35:35 +01:00
Harald Welte 3729c47651 commands: Add method to select parent DF ("cd ..")
This is useful when walking around the filesystem tree.

Change-Id: Ib256c1b7319f2b5f9a06200fb96854ecb2b7f6bb
2022-02-14 00:51:27 +01:00
Harald Welte a630a3cd28 cosmetic: Remove extraneous empty lines between spec-section-comment and class
This is an artefact of the recent autopep8 re-formatting.

Change-Id: I8b0e7781719d69e18856ada2f482de2c5396bcc3
2022-02-14 00:51:27 +01:00
Harald Welte 6169c72f82 USIM + ISIM: Specify the services associated with each file
This allows us [in a future patch] to perform consistency checking,
whether files exist for services not activated in EF.{UST,IST} or
vice-versa: Services are activated by files are not present or
deactivated.

Change-Id: I94bd1c3f9e977767553000077dd003423ed6dbd1
2022-02-14 00:51:27 +01:00
Harald Welte 9170fbf08d filesystem: Maintain a 'service' attribute for all files on a card
This can be populated by card profiles with the SST/IST/UST service
that is associated with the file.

Change-Id: I3b3f74b691368fa09967ecb377a9f7a6d8af7869
2022-02-14 00:51:22 +01:00
Harald Welte afb8d3f925 pySim-shell: introduce 'apdu' command for sending raw APDU to card
This can be useful when playing around with cards, for example
sending commands for which pySim-shell doesn't yet have proper support.

Change-Id: Ib504431d26ed2b6f71f77a143ff0a7fb4f5ea02e
2022-02-14 00:48:16 +01:00
Harald Welte 08b11abc2f pySim-shell: export: allow export as JSON instead of hex
The primary use case of the --json option is to systematically execute
all of our decoder classes in order to find bugs.  As we don't have
encoders for all files yet, the output generated by 'export --json'
will in many cases not be executable as script again, unlike the normal
'export' output.

Change-Id: Idd820f8e3af70ebcbf82037b56fd2ae9655afbc5
2022-02-14 00:48:16 +01:00
Harald Welte c8c3327b6e ts_102_221: Proper parsing of FCP using pySim.tlv instead of pytlv
pytlv is a nightmare of shortcomings, let's abandon it in favor of
our own meanwhile-created pySim.tlv.  This has the added benefit
that unknown tags finally no longer raise exceptions.

Change-Id: Ic8e0e0ddf915949670d620630d4ceb02a9116471
Closes: OS#5414
2022-02-14 00:48:11 +01:00
Harald Welte e4a6eafc6f tlv: Don't raise exception if somebody passes empty data to TLV decoder
Change-Id: Id46994029d9b3cd6b67f4f7ee619466602cc8142
2022-02-14 00:44:55 +01:00
Harald Welte c975251a48 filesystem: Don't pass empty string to parse_select_response()
This happens e.g. when selecting the ARA-M applet on sysmoISIM-SJA2:

pySIM-shell (MF)> select ADF.ARA-M
-> 00a4040409 a00000015141434c00
<- 9000:
Traceback (most recent call last):
  File "/space/home/laforge/.local/lib/python3.9/site-packages/cmd2/cmd2.py", line 2064, in onecmd_plus_hooks
    stop = self.onecmd(statement, add_to_history=add_to_history)
  File "/space/home/laforge/.local/lib/python3.9/site-packages/cmd2/cmd2.py", line 2494, in onecmd
    stop = func(statement)
  File "/space/home/laforge/projects/git/pysim/./pySim-shell.py", line 750, in do_select
    fcp_dec = self._cmd.rs.select(path, self._cmd)
  File "/space/home/laforge/projects/git/pysim/pySim/filesystem.py", line 1314, in select
    select_resp = f.decode_select_response(data)
  File "/space/home/laforge/projects/git/pysim/pySim/filesystem.py", line 193, in decode_select_response
    return self.parent.decode_select_response(data_hex)
  File "/space/home/laforge/projects/git/pysim/pySim/filesystem.py", line 378, in decode_select_response
    return profile.decode_select_response(data_hex)
  File "/space/home/laforge/projects/git/pysim/pySim/ts_102_221.py", line 796, in decode_select_response
    t.from_tlv(h2b(resp_hex))
  File "/space/home/laforge/projects/git/pysim/pySim/tlv.py", line 231, in from_tlv
    (rawtag, remainder) = self.__class__._parse_tag_raw(do)
  File "/space/home/laforge/projects/git/pysim/pySim/tlv.py", line 258, in _parse_tag_raw
    return bertlv_parse_tag_raw(do)
  File "/space/home/laforge/projects/git/pysim/pySim/utils.py", line 208, in bertlv_parse_tag_raw
    if binary[0] == 0xff and len(binary) == 1 or binary[0] == 0xff and binary[1] == 0xff:
IndexError: bytearray index out of range
EXCEPTION of type 'IndexError' occurred with message: 'bytearray index out of range'

Change-Id: I910e6deba27d1483dff1e986c89f1a1b2165f49b
2022-02-14 00:44:55 +01:00
Harald Welte 81f4b4058b Extend unit test coverage for construct, add [some] tests for TLV
Change-Id: I3470e0b2e978221aa0c1e46a4b65f71f71abef2e
2022-02-14 00:41:24 +01:00
Harald Welte d0519e0c37 construct: Add Construct for variable-length int 'GreedyInteger'
We have a number of integers with variable-length encoding, so
add a Construct for this.  Naming inspired by GreedyBytes.

Related to https://github.com/construct/construct/issues/962

Change-Id: Ic6049b74ea3705fda24855f34b4a1d5f2c9327f7
2022-02-14 00:41:24 +01:00
Harald Welte e8d177d88f tlv: Convert CamelCase class name to snake_case in json
Our hand-written JSON so far is using snake_case identifiers,
while the JSON generated by the pySim.tlv classes use the class
names as keys, which LooksQuiteDifferent.

So let's auto-convert the CamelCase into something that reflects
our existing notion.

Change-Id: Id55929ef03dc48cb668e6ba7e99b6b291680a42f
2022-02-12 08:33:37 +00:00
Harald Welte 9a2a6691b0 tlv: Function for flattening the list-of-dict output of TLV decoder
Before:
{
    "FcpTemplate": [
        {
            "FileDescriptor": {
                "shareable": true,
                "file_type": "df",
                "structure": "no_info_given"
            }
        },
        {
            "FileIdentifier": "3f00"
        },
        {
            "ProprietaryInformation": [
                {
                    "UiccCharacteristics": "71"
                },
                {
                    "AvailableMemory": 123052
                }
            ]
        },
        {
            "LifeCycleStatusInteger": "operational_activated"
        },
        {
            "SecurityAttribReferenced": {
                "ef_arr_file_id": "2f06",
                "ef_arr_record_nr": 2
            }
        },
        {
            "PinStatusTemplate_DO": [
                {
                    "PS_DO": "40"
                },
                {
                    "KeyReference": 1
                },
                {
                    "KeyReference": 129
                }
            ]
        },
        {
            "TotalFileSize": 187809
        }
    ]
}

After:
{
    "FcpTemplate": {
        "FileDescriptor": {
            "shareable": true,
            "file_type": "df",
            "structure": "no_info_given"
        },
        "FileIdentifier": "3f00",
        "ProprietaryInformation": {
            "UiccCharacteristics": "71",
            "AvailableMemory": 123052
        },
        "LifeCycleStatusInteger": "operational_activated",
        "SecurityAttribReferenced": {
            "ef_arr_file_id": "2f06",
            "ef_arr_record_nr": 2
        },
        "PinStatusTemplate_DO": {
            "PS_DO": "40",
            "KeyReference": 129
        },
        "TotalFileSize": 187809
    }
}

Change-Id: Ia5ad8f1d3b0d47ebdb1856b0feaba120bad3eef9
2022-02-11 17:13:01 +01:00
Harald Welte 425038ffbc utils: Fix missing Optional[] in type annotations
Thanks to Vadim for pointing this out

Change-Id: I6e7d3725f28410d66580e88f2271d2b240d1f98e
2022-02-11 13:32:58 +01:00
Harald Welte c91085e744 cosmetic: Switch to consistent four-spaces indent; run autopep8
We had a mixture of tab and 4space based indenting, which is a bad
idea.  4space is the standard in python, so convert all our code to
that.  The result unfortuantely still shoed even more inconsistencies,
so I've decided to run autopep8 on the entire code base.

Change-Id: I4a4b1b444a2f43fab05fc5d2c8a7dd6ddecb5f07
2022-02-11 13:32:58 +01:00
Harald Welte 181c7c5930 ts_102_221: Implement proper parsing of EF.DIR
EF.DIR can not only contain the AID + Label of TS 102 221, but can
also contain any of the DOs specified in ISO7816-4.  Let's imoplement
this based on the modern pySim.tlv parser

Change-Id: I875eb49e1f0370428c2eae69af84f5483bd5b1fc
Closes: OS#5410
2022-02-11 13:01:55 +01:00
Harald Welte ca60ac253e filesystem.py: Accept both a class or an instance as TLV._tlv member
As we've seen in recent patches, this has been a source of bugs, so
let's be tolerant and deal with both.

Change-Id: I0a5ec2a860104ffe4524c647105a42505ac394d6
2022-02-10 18:15:25 +01:00
Harald Welte 6551627cb8 ts_31_102: TLV._tlv must point to the class, not an instance
In Change-Id I6d7c1bf49a8eaf3d8e50fb12888bf3d5b46b6c55 we fixed the
filesystem code to assume the self._tlv memper is a reference to a
class, and not an instance (as this is what the majority of the code
did).

However, it seems thre wer two instances where we actually had _tlv
reference an instance.  Change that to class so it's the same all over
the code base.

Change-Id: Ie4878ad6a92feafe47e375c4f5f3f198921e1e95
2022-02-10 17:53:11 +01:00
Harald Welte 944cd2fcf8 filesystem: Fix TLV decode/encode
We cannot call a method of a class without instantiating it

Change-Id: I6d7c1bf49a8eaf3d8e50fb12888bf3d5b46b6c55
2022-02-10 17:06:30 +01:00
Harald Welte e8947493e6 Better decode of EF.UST, EF.EST and EF.IST
So far, we only returned an array of service numbers like
[ 2, 4, 5, 9 ] which is not very friendly to the human reader.

In EF.SST we already had more verbose decoding including a description
of each service.  Let's add the same principle to EF.UST, EST and IST

The same output above now looks like this:

{
    "1": {
        "description": "Local Phone Book",
        "activated": false
    },
    "2": {
        "description": "Fixed Dialling Numbers (FDN)",
        "activated": true
    },
    "3": {
        "description": "Extension 2",
        "activated": false
    },
    "4": {
        "description": "Service Dialling Numbers (SDN)",
        "activated": true
    },
    "5": {
        "description": "Extension3",
        "activated": true
    },
    "6": {
        "description": "Barred Dialling Numbers (BDN)",
        "activated": false
    },
    "7": {
        "description": "Extension4",
        "activated": false
    },
    "9": {
        "description": "Incoming Call Information (ICI and ICT)",
        "activated": true
    }
}

Change-Id: I34f64d1043698dc385619b2fdda23cb541675f76
2022-02-10 17:06:30 +01:00
Harald Welte 08b2499c35 utils.py: Fix some tuple type annotations
Change-Id: I869b0268383f6babd9b51d0ddfce448a1d2dda1e
2022-02-10 17:06:30 +01:00
Harald Welte 5036877147 utils.py: type annotations for DataObject related methods
Change-Id: I291a429e9fe9f1a3fd95dcba3020b0e982154c97
2022-02-10 17:06:30 +01:00
Harald Welte b060833e9a ts_102_221: Handle nested security condition data objects
ISO 7816-4 Section 5.4.3.2 "Expanded Format" permits for nesting
of security conditions using boolean operators OR, AND, NOT.

Let's implement decoding and encoding of these.

An example decoded looks like:

pySIM-shell (MF/EF.ARR)> read_record_decoded 1
[
    [
        {
            "access_mode": [
                "activate_file_or_record",
                "deactivate_file_or_record",
                "update_erase"
            ]
        },
        {
            "or": [
                {
                    "control_reference_template": "ADM1"
                },
                {
                    "control_reference_template": "ADM2"
                }
            ]
        }
    ],
    [
        {
            "access_mode": [
                "read_search_compare"
            ]
        },
        {
            "always": null
        }
    ]
]

Prior to this patch, pySim would raise "ValueError: Unknown Tag 0xa0 in bytearray"

Change-Id: Icb09cf3a90303a86fc77406b8b0806b5c926f1be
Closes: OS#5411
2022-02-10 15:24:32 +01:00
Harald Welte aaf5931b60 ts_51_011: Fix type annotation for Tuple[int, int]
Thanks to Vadim for pointing this out.

Change-Id: I7ee1309331902bafab3c9fc6bc33ca713f8c7832
2022-02-10 14:53:56 +01:00
Harald Welte 6113fe9929 ts_51_011: Fix typo in EF_MMSUP
The TLV decoder class must be in self._tlv, not self.tlv

Change-Id: Ide6f6c823d5a16e375c324ba9bfa92e02c3b3c89
2022-02-09 20:23:27 +00:00
Harald Welte 06c4a5b2d9 ts_31_102: EF.PNN encoding is identical to that of DF.GSM
so let's use the DF.GSM/EF.PNN decoder

Change-Id: If2ce52fccfca3d8bb2c9801b9812912922600377
2022-02-09 20:23:27 +00:00
Harald Welte 362d2d0433 publish also the HTML manual for pySim
Change-Id: I124d59626f40538da30f729beca0e40bf6b2b915
Closes: OS#5270
2022-02-09 21:02:38 +01:00
Harald Welte 5981aa5c7f contrib/jenkins.sh: Fix PUBLISH
There is no $base in this script, and the current form renders:

make: *** /docs: No such file or directory.  Stop.
Build step 'Execute shell' marked build as failure

Change-Id: Ifcf27f7497daeb285dfb364bff20d0c861c77dcb
Related: OS#5271
2022-02-09 18:02:31 +01:00
Harald Welte 6b08fc3b49 contrib/jenkins.sh: first upload manuals, then execute physical tests
The tests with physical cards should not prevent upload of the manuals

Change-Id: I8aecb4ce211cbcc3890886ef24b04d01c510b6da
2022-02-09 17:01:46 +01:00
Harald Welte bf82cebb7b avoid pylint E0611: No name 'strxor' in module 'Crypto.Util.strxor'
At least on Debian 10 and unstable, I'm getting this error for pylint:
************* Module pySim.utils
pySim/utils.py:570:1: E0611: No name 'strxor' in module 'Crypto.Util.strxor' (no-name-in-module)

despite it clearly existing:

>>> import Crypto.Util.strxor
>>> Crypto.Util.strxor.strxor
<built-in function strxor>

So let's suppress the related pylint error.

Change-Id: Iea89e758782a569be953d19892028f083a92c2f1
2022-02-09 16:38:08 +01:00
Harald Welte f03894aa65 update pyyaml dependency to >= 5.1
5.1 was the version introducing pyyaml.FullLoader which we're using,
see https://pyyaml.org/wiki/PyYAML#history

Change-Id: I0f2fa08ceeac2759218e85ad5bdce3ef951d0b74
2022-02-09 15:06:43 +01:00