Commit Graph

600 Commits

Author SHA1 Message Date
Harald Welte 3729c47651 commands: Add method to select parent DF ("cd ..")
This is useful when walking around the filesystem tree.

Change-Id: Ib256c1b7319f2b5f9a06200fb96854ecb2b7f6bb
2022-02-14 00:51:27 +01:00
Harald Welte a630a3cd28 cosmetic: Remove extraneous empty lines between spec-section-comment and class
This is an artefact of the recent autopep8 re-formatting.

Change-Id: I8b0e7781719d69e18856ada2f482de2c5396bcc3
2022-02-14 00:51:27 +01:00
Harald Welte 6169c72f82 USIM + ISIM: Specify the services associated with each file
This allows us [in a future patch] to perform consistency checking,
whether files exist for services not activated in EF.{UST,IST} or
vice-versa: Services are activated by files are not present or
deactivated.

Change-Id: I94bd1c3f9e977767553000077dd003423ed6dbd1
2022-02-14 00:51:27 +01:00
Harald Welte 9170fbf08d filesystem: Maintain a 'service' attribute for all files on a card
This can be populated by card profiles with the SST/IST/UST service
that is associated with the file.

Change-Id: I3b3f74b691368fa09967ecb377a9f7a6d8af7869
2022-02-14 00:51:22 +01:00
Harald Welte afb8d3f925 pySim-shell: introduce 'apdu' command for sending raw APDU to card
This can be useful when playing around with cards, for example
sending commands for which pySim-shell doesn't yet have proper support.

Change-Id: Ib504431d26ed2b6f71f77a143ff0a7fb4f5ea02e
2022-02-14 00:48:16 +01:00
Harald Welte 08b11abc2f pySim-shell: export: allow export as JSON instead of hex
The primary use case of the --json option is to systematically execute
all of our decoder classes in order to find bugs.  As we don't have
encoders for all files yet, the output generated by 'export --json'
will in many cases not be executable as script again, unlike the normal
'export' output.

Change-Id: Idd820f8e3af70ebcbf82037b56fd2ae9655afbc5
2022-02-14 00:48:16 +01:00
Harald Welte c8c3327b6e ts_102_221: Proper parsing of FCP using pySim.tlv instead of pytlv
pytlv is a nightmare of shortcomings, let's abandon it in favor of
our own meanwhile-created pySim.tlv.  This has the added benefit
that unknown tags finally no longer raise exceptions.

Change-Id: Ic8e0e0ddf915949670d620630d4ceb02a9116471
Closes: OS#5414
2022-02-14 00:48:11 +01:00
Harald Welte e4a6eafc6f tlv: Don't raise exception if somebody passes empty data to TLV decoder
Change-Id: Id46994029d9b3cd6b67f4f7ee619466602cc8142
2022-02-14 00:44:55 +01:00
Harald Welte c975251a48 filesystem: Don't pass empty string to parse_select_response()
This happens e.g. when selecting the ARA-M applet on sysmoISIM-SJA2:

pySIM-shell (MF)> select ADF.ARA-M
-> 00a4040409 a00000015141434c00
<- 9000:
Traceback (most recent call last):
  File "/space/home/laforge/.local/lib/python3.9/site-packages/cmd2/cmd2.py", line 2064, in onecmd_plus_hooks
    stop = self.onecmd(statement, add_to_history=add_to_history)
  File "/space/home/laforge/.local/lib/python3.9/site-packages/cmd2/cmd2.py", line 2494, in onecmd
    stop = func(statement)
  File "/space/home/laforge/projects/git/pysim/./pySim-shell.py", line 750, in do_select
    fcp_dec = self._cmd.rs.select(path, self._cmd)
  File "/space/home/laforge/projects/git/pysim/pySim/filesystem.py", line 1314, in select
    select_resp = f.decode_select_response(data)
  File "/space/home/laforge/projects/git/pysim/pySim/filesystem.py", line 193, in decode_select_response
    return self.parent.decode_select_response(data_hex)
  File "/space/home/laforge/projects/git/pysim/pySim/filesystem.py", line 378, in decode_select_response
    return profile.decode_select_response(data_hex)
  File "/space/home/laforge/projects/git/pysim/pySim/ts_102_221.py", line 796, in decode_select_response
    t.from_tlv(h2b(resp_hex))
  File "/space/home/laforge/projects/git/pysim/pySim/tlv.py", line 231, in from_tlv
    (rawtag, remainder) = self.__class__._parse_tag_raw(do)
  File "/space/home/laforge/projects/git/pysim/pySim/tlv.py", line 258, in _parse_tag_raw
    return bertlv_parse_tag_raw(do)
  File "/space/home/laforge/projects/git/pysim/pySim/utils.py", line 208, in bertlv_parse_tag_raw
    if binary[0] == 0xff and len(binary) == 1 or binary[0] == 0xff and binary[1] == 0xff:
IndexError: bytearray index out of range
EXCEPTION of type 'IndexError' occurred with message: 'bytearray index out of range'

Change-Id: I910e6deba27d1483dff1e986c89f1a1b2165f49b
2022-02-14 00:44:55 +01:00
Harald Welte 81f4b4058b Extend unit test coverage for construct, add [some] tests for TLV
Change-Id: I3470e0b2e978221aa0c1e46a4b65f71f71abef2e
2022-02-14 00:41:24 +01:00
Harald Welte d0519e0c37 construct: Add Construct for variable-length int 'GreedyInteger'
We have a number of integers with variable-length encoding, so
add a Construct for this.  Naming inspired by GreedyBytes.

Related to https://github.com/construct/construct/issues/962

Change-Id: Ic6049b74ea3705fda24855f34b4a1d5f2c9327f7
2022-02-14 00:41:24 +01:00
Harald Welte e8d177d88f tlv: Convert CamelCase class name to snake_case in json
Our hand-written JSON so far is using snake_case identifiers,
while the JSON generated by the pySim.tlv classes use the class
names as keys, which LooksQuiteDifferent.

So let's auto-convert the CamelCase into something that reflects
our existing notion.

Change-Id: Id55929ef03dc48cb668e6ba7e99b6b291680a42f
2022-02-12 08:33:37 +00:00
Harald Welte 9a2a6691b0 tlv: Function for flattening the list-of-dict output of TLV decoder
Before:
{
    "FcpTemplate": [
        {
            "FileDescriptor": {
                "shareable": true,
                "file_type": "df",
                "structure": "no_info_given"
            }
        },
        {
            "FileIdentifier": "3f00"
        },
        {
            "ProprietaryInformation": [
                {
                    "UiccCharacteristics": "71"
                },
                {
                    "AvailableMemory": 123052
                }
            ]
        },
        {
            "LifeCycleStatusInteger": "operational_activated"
        },
        {
            "SecurityAttribReferenced": {
                "ef_arr_file_id": "2f06",
                "ef_arr_record_nr": 2
            }
        },
        {
            "PinStatusTemplate_DO": [
                {
                    "PS_DO": "40"
                },
                {
                    "KeyReference": 1
                },
                {
                    "KeyReference": 129
                }
            ]
        },
        {
            "TotalFileSize": 187809
        }
    ]
}

After:
{
    "FcpTemplate": {
        "FileDescriptor": {
            "shareable": true,
            "file_type": "df",
            "structure": "no_info_given"
        },
        "FileIdentifier": "3f00",
        "ProprietaryInformation": {
            "UiccCharacteristics": "71",
            "AvailableMemory": 123052
        },
        "LifeCycleStatusInteger": "operational_activated",
        "SecurityAttribReferenced": {
            "ef_arr_file_id": "2f06",
            "ef_arr_record_nr": 2
        },
        "PinStatusTemplate_DO": {
            "PS_DO": "40",
            "KeyReference": 129
        },
        "TotalFileSize": 187809
    }
}

Change-Id: Ia5ad8f1d3b0d47ebdb1856b0feaba120bad3eef9
2022-02-11 17:13:01 +01:00
Harald Welte 425038ffbc utils: Fix missing Optional[] in type annotations
Thanks to Vadim for pointing this out

Change-Id: I6e7d3725f28410d66580e88f2271d2b240d1f98e
2022-02-11 13:32:58 +01:00
Harald Welte c91085e744 cosmetic: Switch to consistent four-spaces indent; run autopep8
We had a mixture of tab and 4space based indenting, which is a bad
idea.  4space is the standard in python, so convert all our code to
that.  The result unfortuantely still shoed even more inconsistencies,
so I've decided to run autopep8 on the entire code base.

Change-Id: I4a4b1b444a2f43fab05fc5d2c8a7dd6ddecb5f07
2022-02-11 13:32:58 +01:00
Harald Welte 181c7c5930 ts_102_221: Implement proper parsing of EF.DIR
EF.DIR can not only contain the AID + Label of TS 102 221, but can
also contain any of the DOs specified in ISO7816-4.  Let's imoplement
this based on the modern pySim.tlv parser

Change-Id: I875eb49e1f0370428c2eae69af84f5483bd5b1fc
Closes: OS#5410
2022-02-11 13:01:55 +01:00
Harald Welte ca60ac253e filesystem.py: Accept both a class or an instance as TLV._tlv member
As we've seen in recent patches, this has been a source of bugs, so
let's be tolerant and deal with both.

Change-Id: I0a5ec2a860104ffe4524c647105a42505ac394d6
2022-02-10 18:15:25 +01:00
Harald Welte 6551627cb8 ts_31_102: TLV._tlv must point to the class, not an instance
In Change-Id I6d7c1bf49a8eaf3d8e50fb12888bf3d5b46b6c55 we fixed the
filesystem code to assume the self._tlv memper is a reference to a
class, and not an instance (as this is what the majority of the code
did).

However, it seems thre wer two instances where we actually had _tlv
reference an instance.  Change that to class so it's the same all over
the code base.

Change-Id: Ie4878ad6a92feafe47e375c4f5f3f198921e1e95
2022-02-10 17:53:11 +01:00
Harald Welte 944cd2fcf8 filesystem: Fix TLV decode/encode
We cannot call a method of a class without instantiating it

Change-Id: I6d7c1bf49a8eaf3d8e50fb12888bf3d5b46b6c55
2022-02-10 17:06:30 +01:00
Harald Welte e8947493e6 Better decode of EF.UST, EF.EST and EF.IST
So far, we only returned an array of service numbers like
[ 2, 4, 5, 9 ] which is not very friendly to the human reader.

In EF.SST we already had more verbose decoding including a description
of each service.  Let's add the same principle to EF.UST, EST and IST

The same output above now looks like this:

{
    "1": {
        "description": "Local Phone Book",
        "activated": false
    },
    "2": {
        "description": "Fixed Dialling Numbers (FDN)",
        "activated": true
    },
    "3": {
        "description": "Extension 2",
        "activated": false
    },
    "4": {
        "description": "Service Dialling Numbers (SDN)",
        "activated": true
    },
    "5": {
        "description": "Extension3",
        "activated": true
    },
    "6": {
        "description": "Barred Dialling Numbers (BDN)",
        "activated": false
    },
    "7": {
        "description": "Extension4",
        "activated": false
    },
    "9": {
        "description": "Incoming Call Information (ICI and ICT)",
        "activated": true
    }
}

Change-Id: I34f64d1043698dc385619b2fdda23cb541675f76
2022-02-10 17:06:30 +01:00
Harald Welte 08b2499c35 utils.py: Fix some tuple type annotations
Change-Id: I869b0268383f6babd9b51d0ddfce448a1d2dda1e
2022-02-10 17:06:30 +01:00
Harald Welte 5036877147 utils.py: type annotations for DataObject related methods
Change-Id: I291a429e9fe9f1a3fd95dcba3020b0e982154c97
2022-02-10 17:06:30 +01:00
Harald Welte b060833e9a ts_102_221: Handle nested security condition data objects
ISO 7816-4 Section 5.4.3.2 "Expanded Format" permits for nesting
of security conditions using boolean operators OR, AND, NOT.

Let's implement decoding and encoding of these.

An example decoded looks like:

pySIM-shell (MF/EF.ARR)> read_record_decoded 1
[
    [
        {
            "access_mode": [
                "activate_file_or_record",
                "deactivate_file_or_record",
                "update_erase"
            ]
        },
        {
            "or": [
                {
                    "control_reference_template": "ADM1"
                },
                {
                    "control_reference_template": "ADM2"
                }
            ]
        }
    ],
    [
        {
            "access_mode": [
                "read_search_compare"
            ]
        },
        {
            "always": null
        }
    ]
]

Prior to this patch, pySim would raise "ValueError: Unknown Tag 0xa0 in bytearray"

Change-Id: Icb09cf3a90303a86fc77406b8b0806b5c926f1be
Closes: OS#5411
2022-02-10 15:24:32 +01:00
Harald Welte aaf5931b60 ts_51_011: Fix type annotation for Tuple[int, int]
Thanks to Vadim for pointing this out.

Change-Id: I7ee1309331902bafab3c9fc6bc33ca713f8c7832
2022-02-10 14:53:56 +01:00
Harald Welte 6113fe9929 ts_51_011: Fix typo in EF_MMSUP
The TLV decoder class must be in self._tlv, not self.tlv

Change-Id: Ide6f6c823d5a16e375c324ba9bfa92e02c3b3c89
2022-02-09 20:23:27 +00:00
Harald Welte 06c4a5b2d9 ts_31_102: EF.PNN encoding is identical to that of DF.GSM
so let's use the DF.GSM/EF.PNN decoder

Change-Id: If2ce52fccfca3d8bb2c9801b9812912922600377
2022-02-09 20:23:27 +00:00
Harald Welte 362d2d0433 publish also the HTML manual for pySim
Change-Id: I124d59626f40538da30f729beca0e40bf6b2b915
Closes: OS#5270
2022-02-09 21:02:38 +01:00
Harald Welte 5981aa5c7f contrib/jenkins.sh: Fix PUBLISH
There is no $base in this script, and the current form renders:

make: *** /docs: No such file or directory.  Stop.
Build step 'Execute shell' marked build as failure

Change-Id: Ifcf27f7497daeb285dfb364bff20d0c861c77dcb
Related: OS#5271
2022-02-09 18:02:31 +01:00
Harald Welte 6b08fc3b49 contrib/jenkins.sh: first upload manuals, then execute physical tests
The tests with physical cards should not prevent upload of the manuals

Change-Id: I8aecb4ce211cbcc3890886ef24b04d01c510b6da
2022-02-09 17:01:46 +01:00
Harald Welte bf82cebb7b avoid pylint E0611: No name 'strxor' in module 'Crypto.Util.strxor'
At least on Debian 10 and unstable, I'm getting this error for pylint:
************* Module pySim.utils
pySim/utils.py:570:1: E0611: No name 'strxor' in module 'Crypto.Util.strxor' (no-name-in-module)

despite it clearly existing:

>>> import Crypto.Util.strxor
>>> Crypto.Util.strxor.strxor
<built-in function strxor>

So let's suppress the related pylint error.

Change-Id: Iea89e758782a569be953d19892028f083a92c2f1
2022-02-09 16:38:08 +01:00
Harald Welte f03894aa65 update pyyaml dependency to >= 5.1
5.1 was the version introducing pyyaml.FullLoader which we're using,
see https://pyyaml.org/wiki/PyYAML#history

Change-Id: I0f2fa08ceeac2759218e85ad5bdce3ef951d0b74
2022-02-09 15:06:43 +01:00
Bjoern Riemer e91405e04e implement shell command to update PLMN in IMSI
Add file specific command `update_imsi_plmn` to EF_IMSI to replace
the mcc and mnc part of the imsi for use in bulk_script(s)

Change-Id: I9662ff074acf9dc974ae4c78edac44db286e98fc
2022-01-31 11:55:12 +00:00
Steve Markgraf 9c93cec32a transport/serial: fix for Python 3
Change-Id: I21e5a7ad4f623ed30681dce1ff819679b8714c5b
2022-01-25 01:14:44 +01:00
Harald Welte 0c840f0aab ts_102_221: decode/encode EF.PL
pySIM-shell (MF/EF.PL)> read_binary_decoded
[
    "en",
    null,
    null,
    null,
    null
]

Change-Id: I4e879ef34acee461adb8137a6315d064370b1b10
2022-01-22 12:59:02 +00:00
Harald Welte b3d68c0b98 pySim-shell: alphabetically sort name of files in 'dir' command
Change-Id: Id136909884d3c0eaa2416c6c488a6c4b7ed48119
2022-01-22 12:59:02 +00:00
Harald Welte 2a701eea60 cosmetic: Use EF.FDN instead of EF_FDN in ts_51_011.py
All the files have '.' as separator in their names so far, let's avoid
any inconsistencies

Change-Id: Icabb892408a40ea37c7ebeb7db545b383aa01d99
2022-01-22 12:59:02 +00:00
Harald Welte ff2d86d977 ts_31_102: Add support for EF.ECC (emergency call codes)
decoded output will look like this:
[
    {
        "call_code": "911",
        "service_category": {
            "police": false,
            "ambulance": false,
            "fire_brigade": false,
            "marine_guard": false,
            "mountain_rescue": false,
            "manual_ecall": false,
            "automatic_ecall": false
        },
        "alpha_id": "911"
    },
    {
        "call_code": "112",
        "service_category": {
            "police": false,
            "ambulance": false,
            "fire_brigade": false,
            "marine_guard": false,
            "mountain_rescue": false,
            "manual_ecall": false,
            "automatic_ecall": false
        },
        "alpha_id": "112"
    },
    null,
    null,
    null
]

Change-Id: If8b4972af4f5be1707446d335cfc6e729c973abb
2022-01-22 12:59:02 +00:00
Bjoern Riemer ffee89a031 add missing bit definition for NG-RAN in xAcT
when encoding the AcT value bit 11 is correctly set
when NG-RAN is present in the string representation,
however the decoding of bit 11 was missing.
Adds tests for the decoder as well.

Change-Id: I910df28c4c59ec94cce9603377786325f6d8c1a3
2022-01-22 12:58:00 +00:00
Bjoern Riemer da57ef1529 catch and ignore SwMatchError on probing for AID's
When probing applications on a card by running select_adf_by_aid()
SwMatchError exceptions indicating the non exsistance of that
application on the card should be ignored.

Change-Id: I3aa7deaf46bdf352a201c3089b3714405a06f281
2022-01-20 22:31:32 +01:00
Julian Lemmerich 3e33cc7157 Add pyyaml to requirements.txt
Change-Id: I3430c32aea59af97360b9e766bfe95a146f09fe0
2022-01-13 16:36:14 +01:00
Philipp Maier 0e4515f53d filesystem: use correct AID for applications found by probing
When printing applications found by probing for a specific AID, then the
wrong variable is used to print the AID.

Change-Id: I3d5ec28e46fe00c0d793a1d9ef0a0e0900649a4d
2022-01-04 18:06:25 +01:00
Philipp Maier 8d8bdef637 filesystem: actively probe applications
A profile can cover lots of different applications. Those applications
may not exist on all card models. To exclude applications that are not
installed on the particular card EF.DIR is evaluated. However, there may
be applications that are not registered in EF.DIR but supported by the
profile. To cover those as well, lets try to select the applications we
do not see in EF.DIR. If selecting works we know that the application
exists on the card and we can include them in the RuntimeState.

Change-Id: I3fa77a68664fe50d690a18adfb1ae1a88a189827
2021-12-01 11:52:47 +01:00
johannes.richter e903a40530 fix invalid dependency
* serial is according to pypi: "A framework for serializing/deserializing
 JSON/YAML/XML into python class instances and vice versa"

Change-Id: I154276fbadd70f6be94ba7d99e61f7e9eedbeb33
2021-11-25 16:57:54 +01:00
Lennart Rosam c104095c69 fix: Decoder may raise KeyError
This fixes an issue where a KeyError may be raised when 'A5'
is not present in `fcp`

Change-Id: I5bb6131bd76c7bae2a70034c429cae2b380d164f
2021-11-25 16:55:08 +01:00
Philipp Maier 931bc66331 cards: Make select_adf_by_aid() use prefix AID selection
There is no need for us to expand a partial AID to the full AID before
selecting that ADF. The UICC specifications permit AID selection by
prefix only. So we could pass the prefix to the card, and the card would
do the prefix matching. In order to avoid problems with cards that fail
to do the prefix matching themselves we will still do the AID
completion, but in case we cannot complete the AID (AID not listed in
EF.DIR), we will try with the AID prefix anyway.

From the API user point of view, this allows us to select applications
not listed in EF.DIR

Change-Id: I0747b4e46ca7e30bd96d76053765080367ac1317
2021-11-23 18:35:34 +01:00
Philipp Maier abc2336571 pySim-read: put try/catch block around select_adf_by_aid()
Selecting an application may fail, especially when the application does
not exist on the card.

Change-Id: Ia904a74d672cf9551fb4ee062dd606b350b64cef
2021-11-23 18:35:34 +01:00
Philipp Maier 47833bc176 cards: make _get_aid case insensitive
There is no need to be case sensitive when the xSIM application name is
given as AID.

Change-Id: I9944d9180bf1ba35f44f0be2b05bdb725b5b8da9
2021-11-22 17:37:00 +00:00
Philipp Maier 9e42e7ffce profile: decode_select_response use object instead 'Any'
the return type of decode_select_response is 'Any', lets be more
specific and use 'object'

Change-Id: Ic5c7ace234bc94ab1381d87e091369ade8011cab
2021-11-19 13:21:32 +01:00
Philipp Maier 5998a3a8b3 profile: decode_select_response can be a static method
The method decode_select_response does not access any property of the
object. This means the method can be static.

Change-Id: Idd7aaebcf1ab0099cd40a88b8938604e84d8a88b
2021-11-19 13:21:32 +01:00
Philipp Maier 825b564115 pySim-shell: export command: guess number of records when not specified
The select response of an UICC will always return the number of records
of a file. However, older SIM will not include the number of records in
the select response. In those cases, simply guess the number of records
by reading until the first invalid record is hit.

Change-Id: Ib480797d881b9ec607ec6a86b73d452449f8cf87
Related: OS#5274
2021-11-19 13:21:32 +01:00