Commit Graph

1129 Commits

Author SHA1 Message Date
Philipp Maier e6f8d683e1 utils: specify paremeters of enc_plmn() as Hexstr
To prevent missunderstandings when using enc_plmn(), specify the input
and return parameters as Hexstr.

Change-Id: I57cf8e2de357650aef2a06fbffc7615ccb2a45b4
Related: OS#4963
2021-05-03 14:19:41 +00:00
Vadim Yanitskiy 52efc0372c ModemATCommandLink: fix AttributeError exception in __del__()
self._sl will not be assigned if serial.Serial() fails.

Change-Id: I179a7efd92eaa3469a17b6ef252b3949c44ea6ea
2021-05-03 14:13:14 +00:00
Robert Falkenberg 18fb82bd38 Speed up AT command interface (~130 times faster)
Previous implementation waits 300ms for response after
each command issued. But many commands finish earlier.

This patch improves the command execution time by frequently
checking for the response to complete (i.e. ends with
OK or ERROR), or the occurence of a timeout (default 200ms).

Timeout can be adapted per command to support long response
times of certain commands like AT+COPS=? (network search)

Execution time benchmark (20 AT commands/responses):
Previous: 6.010s (100.0%)
New code: 0.045s (  0.7%)

Change-Id: I69b1cbc0a20d54791e5800bf27ebafc2c8606d93
2021-05-03 10:06:10 +02:00
Vadim Yanitskiy e9fe09baff contrib/jenkins.sh: run pylint to find potential errors
Change-Id: Ib4cdd0fa824329569f973925ba3a46656a8c8296
2021-05-02 20:22:49 +00:00
Vadim Yanitskiy 895fa6f83c [pylint] Fix referencing undefined variable 'shutil'
Let's just use the scope limited TemporaryDirectory() instead, so
the temporary directory will be removed by Python automatically.

pySim/filesystem.py:679:16: E0602: Undefined variable 'shutil' (undefined-variable)

Change-Id: I4ea833fd79f4342c33899124379be509ba1e35ed
2021-05-02 20:22:15 +00:00
Vadim Yanitskiy e8c34ca3e2 [pylint] Declare abstract LinkBase._send_apdu_raw() method as such
pySim/transport/__init__.py:86:15: E1101:
	Instance of 'LinkBase' has no '_send_apdu_raw' member;
	maybe 'send_apdu_raw'? (no-member)

Change-Id: I14fcdceca5d1e35491b6ad98f96b4276b69b2fc1
2021-05-02 20:22:15 +00:00
Vadim Yanitskiy eb395867ab [pylint] Fix float vs integer division in cards.py
Change-Id: Ie4ba72b725a56ba9cfe98cc7bd17dd3653194f36
2021-05-02 20:22:15 +00:00
Vadim Yanitskiy 5e41eeb6fa [pylint] Fix reference to undefined variable 'in_hex'
pySim/ts_31_102.py:384:36: E0602: Undefined variable 'in_hex' (undefined-variable)

Change-Id: I094e61c78621f08108f6ad1c71a05bc1e13a895d
2021-05-02 20:22:15 +00:00
Vadim Yanitskiy 85302d6757 [pylint] Mark abstract MagicSimBase class as such
Change-Id: I315c646d94a1d3282917f5abb0c93efb918b53d7
2021-05-02 20:22:15 +00:00
Vadim Yanitskiy d9a8d2fc43 [pylint] Fix calling non-existing iteritems() of dict
This method has been removed [1] in Python 3.0:

pySim/cards.py:581:14: E1101: Instance of 'dict' has no 'iteritems' member (no-member)
pySim/cards.py:591:24: E1101: Instance of 'dict' has no 'iteritems' member (no-member)

[1] https://wiki.python.org/moin/Python3.0#Built-In_Changes

Change-Id: Iba7ad9ed2a9b197ecedaaed1c6744fe1c721515a
2021-05-02 20:22:15 +00:00
Vadim Yanitskiy 03c67f775b [pylint] Declare some fields in _MagicSimBase class
Fixes the following pylint's warnings:

pySim/cards.py:494:18: E1101: Class '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:509:6: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:529:26: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:537:5: E1101: Instance of '_MagicSimBase' has no '_ki_file' member (no-member)
pySim/cards.py:547:5: E1101: Instance of '_MagicSimBase' has no '_ki_file' member (no-member)
pySim/cards.py:548:8: E1101: Instance of '_MagicSimBase' has no '_ki_file' member (no-member)
pySim/cards.py:559:26: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:560:11: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:576:14: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)

Change-Id: I4db9d21258d6e04140962134c540e36631466322
2021-05-02 20:22:15 +00:00
Vadim Yanitskiy 5ef1650696 [pylint] Mark abstract CardKeyProvider.get() method as such
pySim/card_key_provider.py:67:2: E1111:
	Assigning result of a function call, where the function
	has no return (assignment-from-no-return)

Change-Id: I43bab69f53300fbe837944735cd999fab5405d7a
2021-05-02 20:22:15 +00:00
Vadim Yanitskiy 1a95d2be61 [pylint] fix non-existing 'res' in EF_CNL._encode_record_hex()
pySim/ts_51_011.py:648:45: E0602: Undefined variable 'res' (undefined-variable)
pySim/ts_51_011.py:648:68: E0602: Undefined variable 'res' (undefined-variable)
pySim/ts_51_011.py:649:24: E0602: Undefined variable 'res' (undefined-variable)

Change-Id: I039e2b271dd3ab44f437108c5e8def43e97098a3
2021-05-02 20:22:15 +00:00
Robert Falkenberg 8e15c18498 transport/AT: Make sure PDU has upper case hex digits
Some modems may reject AT+CSIM if PDU contains lower
case hex digits [a-f]. Modem response is "ERROR"
without any error code.

This patch converts each PDU to upper case.

Tested with Sierra Wireless EM7565.
Example:
AT+CSIM=14,"00a40004023F00"
ERROR
AT+CSIM=14,"00A40004023F00"
+CSIM: 4,"612F"

OK


Change-Id: I318e36abc7ae975c62d32b7fe0ec949bf5997d13
2021-05-02 20:11:32 +00:00
Philipp Maier f9cbe090e4 ts_51_011: fix encoding of EF.MSISDN
The json input that is used with EF.MSISDN seems to be somewhat
ambigious. The original code accepts {"msisdn": "+4916012345678"}
only while the output is {"msisdn": [1, 1, "+4916012345678"]}. Lets
add a check and also accept the latter version.

Change-Id: I8f8dd68aac25d3fa3bc1aab06b855f8ec6640258
Related: OS#4963
2021-04-30 17:04:52 +02:00
Martin Hauke 0f1862799f setup: set minimum required versions for contruct and cmd2
* 'String' class renamed to 'PaddedString' in construct v2.9
* 'CommandSet' was introduced with with cmd2 v1.3.0

Change-Id: I170a9e896612086c4b0535cd6d950346897abc3b
2021-04-30 14:43:44 +02:00
Harald Welte 278816251d filesystem.py: Introduce place-holder for BER-TLV files
I always assumed BER-TLV files are transparent EF with BER-TLV contents.

However, this is wrong. ETS TS 102 221 Section 8.2.2.4 specifies them.

TS 102 221 Section 11.3 describes the specific RETRIEVE DATA, SET DATA
commands, which are not yet implemented in pySim.

Change-Id: Ie4701d9f72b05c8a5810e287e55a20f6ea86a574
2021-04-24 11:43:04 +02:00
Harald Welte 71290077b9 ts_31_102, ts_31_103: Add EF.FromPreferred
Change-Id: I7dc989a4ab198f3eaa45ba7060c8087354a544bb
2021-04-24 11:43:04 +02:00
Harald Welte 16ec96a0c9 ts_31_103: Add Rel 16.6 enhancements (MuDMiDConfigData)
Change-Id: I54046375f180017373ab8e06e60ac5a542da706a
2021-04-24 11:43:04 +02:00
Harald Welte 35dfe826c0 ts_31_103: Use EF_SMS, EF_SMSS, EF_SMSR, EF_SMSP from ts_51_011
Change-Id: I688cfddcf5845316f71a9641d4426a20f58c1fba
2021-04-24 11:43:04 +02:00
Harald Welte 3990ebb810 ts_31_102: Define DF_WLAN, DF_ProSe and DF_HNB with their EFs
Change-Id: I2d4662dc021e286a1c3293ab36aaa845b1251388
2021-04-24 11:43:04 +02:00
Harald Welte 2f831035ef ts_31_102: Extend DF_5GS to 3GPP 31.102 R16.6
Change-Id: I344d8247ff81463e5c0140ff17e66322a61ef20f
2021-04-24 11:43:01 +02:00
Philipp Maier b46cb3ffa2 utils: fix encoding of EF.MSISDN
The encoding of EF.MSISDN is a bit unstrutured. The encoder function
does not return a valid result since it lacks the parameters
Capability/Configuration2 Record Identifier and Extension5 Record
Identifier, which are mandatory but can be set to 0xFF. Also the
encoder gets its input from pySim-shell, so it should have some
more input validation, especially when the user encodes an empty
string. The encoder and decoder function also do not have unit-tests.

Since the encoder now adds the missing two bytes by isself this does
not have to be done manually anymore, so cards.py needs to be
re-aligned.

For pySim-shell.py the encoder is used from ts_51_011.py. Unfortunately
it is used wrongly there. The optional Alpha Identifier is required
here as well.

Related: OS#4963
Change-Id: Iee5369b3e3ba7fa1155facc8fa824bc60e33b55b
2021-04-23 15:52:10 +02:00
Harald Welte 977035c429 filsystem.py: Add more information to exceptions
Change-Id: Ia9449ddfaaf5f49e0a65aeeea9435141fd55fe65
2021-04-22 09:11:55 +02:00
Harald Welte 8fe1d202c7 pySim-read: Migrate over to use shared argparse from transport
Now that we have a shared argparse definition for all reader related
options in the transport module, use that.

Change-Id: I12ca1a484a5d6e84820d9761c9701f8a94381f66
2021-04-22 09:11:23 +02:00
Harald Welte 28c2431f9c Move reader related argument parser to transport module
Ideally that shared definition would be used by all programs,
rather than copy+pasting it.  Unfortunately pySim-{read,prog}
are still using optparse and first need to be converted to
argparse.

Change-Id: If77f53850e1ca65f42cf1dca3e0f460dac1b0d1a
2021-04-22 09:11:23 +02:00
Philipp Maier fe1fb037b5 filesystem: fix wrong helpstring for update_record_decoded
the helpstring of update_record_decoded mentions hex bytes for the data
parameter, but it should be mentioned as abstract json data like in
update_binary_decoded

Change-Id: Ibae2ab49054ac5dd6fcccddd28c98d886403dac9
Related: OS#4963
2021-04-21 15:13:52 +02:00
Philipp Maier 0adabf6161 filesystem: fix wrong comment
The property rec_len is not a tuple, it is a set.

Related: OS#4963
Change-Id: I366772c62d0bb5a6400ce5b431eb94ac9248dccc
2021-04-21 15:13:52 +02:00
Philipp Maier 80ce71f58c pySim-shell: separate export summary with a headline
the export summary is printed after the log entry for the last file
without separation. This is confusing because it looks like if the
summary would refer to the last file only. Lets add a headline to make
clear that the last few lines are the "Export summary"

Change-Id: I90771e525b2b114bdb41a8e90d298ca991c09c3d
Related: OS#4963
2021-04-21 15:11:34 +02:00
Philipp Maier 04be9d6033 pysim-testdata/sysmoISIM-SJA2: change card in test rig
The sysmo-usim-sja2 which is currently used for testing somewhat old. In
order to have a defined state here lets exchange it with a new card.
This will also ensure that we have a card with the most recent card
profile in the tester.

Change-Id: Ife13823d0b0a16b1f327dd882f91d0c95af65e43
2021-04-21 12:32:24 +02:00
Robert Falkenberg bba2bd4348 setup: add (nested) library pySim.transport
* setup.py did not install pySim.transport sub-library

Change-Id: Ice026a28c4843121591f3404225c7f6aec4df6c7
2021-04-16 12:43:11 +02:00
Robert Falkenberg 9d16fbca4a Use construct for EF_AD in pySim-{shell, prog, read}.py, cards.py
Also serves as example for RFU (reserved for future use) fields
which should not always be reset to zero in case they have been
set on the uSIM for some reason.
See pySim/ts_51_011.py, class EF_AD.

* Add definitions for RFU {Flag, Bits, Byte, Bytes}
* Use IntEnum for OP_MODE (convenient auto completion)
* Remove obsolete definitions and imports
* Update test results for all SIMs (opmode strings are shortened)

Change-Id: I65e0a426f80a619fec38856a30e590f0e726b554
2021-04-13 11:27:37 +00:00
Harald Welte c8ff026782 pySim-shell: Introduce logical grouping of arguments
Change-Id: Id80a68e615fc1e457bde3dff8817776826fc6d8e
2021-04-11 12:35:25 +02:00
Harald Welte f2e761ce20 pySim-shell: Migrate from optparse to argparse for the main()
We're using argparse internally for all shell commands, and can
use that to auto-generate command reference in the manual.

Let's switch to argparse for the main program, too - and generate
the related reference in the manual.

Change-Id: I77c946dbeb9f746fe3d8051173e59462dc2fb5e2
2021-04-11 12:35:25 +02:00
Harald Welte e4759fd5cd contrib/jenkins.sh: Build and publish PDF manual
Change-Id: I3f01e93dd5a25d26feb3d067a171244a20f0f8e5
2021-04-11 12:20:29 +02:00
Harald Welte 348c195e97 rename manual to be consistent with osmo-gsm-manuals projects
Change-Id: Iad1d96f2b0f95a010b0b30da00d3eef754c1ec44
2021-04-11 12:20:29 +02:00
Harald Welte c9cdce3e02 fix various typos all over the code
Change-Id: Ic8392a951bf94f67b51e35bed95d0e856f7a9250
2021-04-11 12:20:29 +02:00
Harald Welte 790b2709bd lots of file definitions for classic SIM and USIM
Change-Id: I91475df4a5eaca423473aaebba8c69c54c9a0c1a
2021-04-11 12:20:29 +02:00
Harald Welte 9853e247a2 ts_31_102.py: switch EF.Keys from imperative to declarative codec
Change-Id: I215a61c1e44ee2c5df70a8ca96344b4dd4d93c00
2021-04-11 12:20:29 +02:00
Harald Welte 2db5cfb525 filesystem: Introduce 'construct' support
This allows for pure declarative encoders/decoders for linear fixed
and transparent EFs.

Change-Id: Id0765ec3ddb8c044838ce47d0ff8740720a32b15
2021-04-11 12:20:29 +02:00
Harald Welte 703f933b40 pySim-shell: Add open_channel + close_channel commands
Change-Id: I53d9d7f7720eb5f10956bff74ea7ba9fd3b3bd19
2021-04-11 12:20:29 +02:00
Harald Welte a463161ae2 pySim-shell: Adds support for DEACTIVATE FILE + ACTIVATE FILE
Change-Id: I22207dde20f991b0a22dea8f5dd695a0ec99da33
2021-04-11 12:20:29 +02:00
Harald Welte 15fae98d2e pySim-shell: Authenticate (3g) support
This adds support for AUTHENTICATE to the USIM and ISIM application,
based on the newly-introduced 'construct' encoder/decoder support.

Change-Id: Id5697463e29c3dceff98bcf80f5400f7f2bcaa6c
2021-04-11 12:20:29 +02:00
Harald Welte e0f9ef1606 integrate 'construct' python library
'construct' is a declarative symmetric encoder/decoder for user
specified binary formats.  It should come in extremely handy in
tools like pySim.

We start the integration by adding transport methods for transceiving
APDUs with built-in encoding of the command data and decoding of the
response data.

Change-Id: Ibf457aa8b9480a8db5979defcfafd67674303f6c
2021-04-11 12:20:29 +02:00
Robert Falkenberg d0505bdb55 WIP: Add option to set UE operation mode in EF_AD (Administrative Data)
Use ``--opmode=OPMODE`` in cmdline mode or column ``OPMODE`` in csv mode
to specify OPMODE as listed below.

Details:
The ``EF_AD`` field contains administrative data (AD).
It consists of four bytes ``B1``, ``B2``, ``B3``, ``B4``,
and optionally further bytes for future use.

Previous implementation only sets the MNC field appropriately
(located in `B4`) and sets all other bits/bytes to 0.

However, `B1` also defines the *UE operation mode* (see below).
For type approval operations, such as testing with a test uSIM,
this value could be set to `0x80` rather than `0x00`(= normal operation).
This may unlock some UE capabilities that are restricted in
normal operation mode.

Excerpt from [ETSI TS 131 102, 4.2.18](https://www.etsi.org/deliver/etsi_ts/131100_131199/131102/04.15.00_60/ts_131102v041500p.pdf):

```
B1 - UE operation mode:
	Coding:
	Initial value
	- '00' normal operation.
	- '80' type approval operations.
	- '01' normal operation + specific facilities.
	- '81' type approval operations + specific facilities.
	- '02' maintenance (off line).
	- '04' cell test operation.

B2 - Additional information:
	Coding:
	Reserved for future use

B3 - Additional information:
	Coding:
	- B3.b1: OFM setting (Ciphering Indicator)
	- B3.others: Reserved for future use

B4 - Length of MNC in the IMSI:
	Coding:
	- B4.b4..B4.b1: length:  '0010' (= 2) or '0011' (=3)
	- B4.others: Reserved for future use
```

**Legend:** Byte X, bit Y: BX.bY

Further reading: https://nickvsnetworking.com/usim-basics/

Change-Id: Ie9040c6b127c268878a0845ed73d0918ec6bbb08
2021-04-11 09:35:36 +00:00
Harald Welte 08028d02dc README.md: Mention user manual
Change-Id: I6e40166a2ed7ba9f352b466ce85651e20da591e9
2021-04-11 11:21:39 +02:00
Harald Welte 11ee243c47 README.md: We have switched to gerrit for review
Change-Id: I4ee30c411a2dfc2cf39840af2517f28c1989099f
2021-04-11 11:21:39 +02:00
Harald Welte e7887d7af5 README.md: Update dependencies
Change-Id: Ib4028b57aff6bf3d42a83b3942c55f2a1c133172
2021-04-11 11:21:39 +02:00
Harald Welte edee155773 setup.py: add missing comma in list of dependencies
Change-Id: If93d8c67cdfacc1f30307c8cfb38924fb2cbabbe
2021-04-10 18:41:15 +02:00
Harald Welte b00e893e73 pySim-shell: Use poutput_json() whenever applicable
poutput_json() uses a customs JSONEncoder, and hence all JSON
prenting should go through it.

Change-Id: Ie023669e77311350ade4f8dbc65431e71b586052
2021-04-10 18:41:15 +02:00