This change introduces an optional feature that allows to make
SQLite3 use talloc for all internal allocations. This would
facilitate finding memleaks. OsmoHLR needs to be configured
with '--enable-sqlite-talloc'.
full talloc report on 'OsmoHLR' (total 292168 bytes in 449 blocks)
struct osmo_gsup_server contains 162 bytes in 3 blocks (ref 0)
...
struct db_context contains 288407 bytes in 420 blocks (ref 0)
hlr.db contains 7 bytes in 1 blocks (ref 0)
SQLite3 contains 288192 bytes in 418 blocks (ref 0)
db.c:95 contains 48 bytes in 1 blocks (ref 0)
db.c:95 contains 2 bytes in 1 blocks (ref 0)
...
Unfortunately, old SQLite3 versions (such as 3.8.2) run out
of memory when trying to initialize a new database:
DDB ERROR db.c:88 (7) statement aborts at 3: []
DDB ERROR db.c:420 Unable to set Write-Ahead Logging: out of memory
DDB ERROR db.c:88 (7) statement aborts at 3: []
DDB ERROR db.c:238 Unable to prepare SQL statement
'SELECT name FROM sqlite_master WHERE type='table' AND name=?'
...
I've noticed a huge difference in heap usage footprint compared to
generic malloc. At the same time, the recent versions (at least
3.24.0), work just fine.
Change-Id: Icfe67ed0f063b63e6794f9516da3003d01cf20a7
Somehow both 'db_test_SOURCES' and 'db_test_LDADD' ended up in
'src/Makefile.am'. This causes automake / autoconf to complain.
Let's get rid of both useless declarations.
Furthermore, the actual 'db_test_LDADD' in 'tests/Makefile.am'
contained references to the source files from '$(top_srcdir)'.
Most likely, the original intention was to depend on the object
files in '$(top_builddir)'. Let's also fix this.
Change-Id: Ib2e436ed91d9b7551dc5b205329d468c2b0ced04
Replace commented out size check for Ki with a real check, and use it
consistently for Ki, K, OP and OPC. Add a test that sets all keys to the
wrong size and tries to read them.
Related: OS#2565
Change-Id: Ib8e8e9394fb65c6e7932ce9f8bebc321b99f7696
Allow updating the NAM (Network Access Mode) of subscribers with the
VTY. This is important for the subscriber create on demand use case
where subscribers get created without access to PS and CS NAM by
default. Regenerate hlr_vty_reference.xml.
Related: OS#2542
Change-Id: I231e03219355ebe6467d62ae2e40bef9d8303e3b
The last LU time gets read from the database as string, parsed with
strptime to "struct tm", and then gets converted to time_t with mktime.
A recent behavior change in glibc's mktime implementation unconvered,
that we don't have tm.tm_isdst (daylight saving time) set properly. As
"struct tm" was not initialized, and strptime did not write to tm_isdst,
it was set to a random value. When it was not 0, db_test failed on UTC
systems with a more recent glibc (e.g. Ubuntu 19.04).
Fix this by zero-initializing "struct tm" and remove the previous
workaround that made db_test pass on UTC systems.
Related: OS#4026
Change-Id: Iebbbe42fc5cd43324206d9433ede67b39251389c
Apply workaround for db_test failure on ubuntu 19.04 in OBS. When the
timezone is set to UTC (default in OBS), and the glibc version is 2.29,
the test case is failing sometimes (not always) with the following
errors in the test log:
DAUC Cannot convert LU timestamp '2019-05-26 03:05:03' to time_t: Value too large for defined
Force the timezone to be CET when running the test, so it passes again.
I found this workaround in a Fedora bugreport [1], and tested with an
ubuntu 19.04 system running in docker, that setting the environment
variable like this makes the test pass 25 times in a row. Without it, it
fails every two or three times. Then I verified in my OBS namespace,
that the test also passes in OBS again.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1653340
Related: OS#4026
Change-Id: Ic8080ba1914bb364169ab0c563b132a4ab9a9aab
Add a new vty option and allow to optionally generate a random msisdn,
as well as setting the default NAM:
subscriber-create-on-demand (no-msisdn|<3-15>) (none|cs|ps|both)
Thanks to Vadim for the random MSISDN patch [1], which was squashed into
this one.
[1] Change-Id: I475c71f9902950fa7498855a616e1ec231fad6ac
Depends on: Idc74f4d94ad44b9fc1b6d43178f5f33d551ebfb1 (libosmocore)
Change-Id: I0c9fe93f5c24b5e9fefb513c4d049fb7ebd47ecd
Related: OS#2542
Check if a subscriber exists without generating an error log entry if
it does not. This is cheaper than db_subscr_get_by_msisdn(), as it
does not fetch the subscriber entry.
subscriber-create-on-demand will use this function to generate
a random unique MSISDN for new subscribers.
Related: OS#2542
Change-Id: Ibfbc408c966197682ba2b12d166ade4bfeb7abc2
Check if a subscriber exists without generating an error log entry if
it does not. This is cheaper than db_subscr_get_by_imsi(), as it does
not fetch the subscriber entry. subscriber-create-on-demand will use
this function.
Related: OS#2542
Change-Id: I63818c0dd4fd22b41dadeeba2a07a651b5454c54
Allow creating new subscribers without giving them access to CS or PS.
This will be used by the create-subscriber-on-demand feature.
Related: OS#2542
Change-Id: I1a6dd85387723dab5487c53b33d2d9ec6d05d006
So far, the cmdline argument was the only way to set a database config file.
Add a similar config to VTY as 'hlr' / 'database'. The cmdline arg is stronger
than the 'database' cfg item. DB is not reloaded from VTY command.
Change-Id: I87b8673324e1e6225afb758fb4963ff3279ea3d8
This ensures that the rpath of the generated binaries is set to use only
the just-compiled so-files and not any system-wide installed libraries
while avoiding the ugly shell script wrapper.
Change-Id: I927561289b17b313d52fb5c1da55e237fc1d33be
Display the IMEI in "subscriber ... show", allow showing and modifying
subscribers by their IMEI with: "subscriber imei ...". For debug
purposes (and to have proper VTY tests), make it possible to change the
IMEI with "subscriber ... update imei".
IMEIs are saved in the database without the 15th checksum number. When
the checksum gets passed, verify it and cut it off.
Related: OS#2541
Depends: I02b54cf01a674a1911c5c897fbec02240f88b521 (libosmocore)
Change-Id: I1af7b573ca2a1cb22497052665012d9c1acf3b30
Create a test_subscriber.vty.sql file with a dummy entry that has the
ID 100. All entries created in test_subscriber.vty have an ID > 100
now. This will be used in follow-up commit [1] to create a database
entry with an invalid IMEI value to test the related error handling
code path (that entry could not be created through the VTY).
[1]: change-id I1af7b573ca2a1cb22497052665012d9c1acf3b30
"VTY: integrate IMEI"
Related: OS#3733
Change-Id: I48a3a503d7ca96798e2d5f70429b5fc36393420e
Add VTY config option "store-imei". When it is set, store the IMEI
sent from the VLR with CHECK-IMEI in the database.
Related: OS#2541
Change-Id: I09274ecbed64224f7ae305e09ede773931da2a57
Extend the database scheme, add imei to the hlr_subscriber struct and
create db_subscr_update_imei_by_imsi() and db_subscr_get_by_imei(). The
new functions are used in db_test, and in follow-up commits [1], [2].
Upgrade DB schema to version 2. SQLite can only insert new columns at
the end of the table, so this happens when upgrading the database. In
new databases, the column is placed after the IMEISV column (where it
makes more sense in my opinion). This should not have any effect, as
we never rely on the order of the columns in the tables.
Follow-up commit [1] will make use of this column to save the IMEI as
received from the MSC/VLR with the Check-IMEI Procedure. It was
decided to use Check-IMEI instead of the recent Automatic Device
Detection Procedure (which would send the IMEISV) in OS#3733, because
with Check-IMEI we don't need to rely on very recent releases of the
specification.
[1] change-id I09274ecbed64224f7ae305e09ede773931da2a57
"Optionally store IMEI in subscriber table"
[2] change-id I1af7b573ca2a1cb22497052665012d9c1acf3b30
"VTY: integrate IMEI"
Depends: Id2d2a3a93b033bafc74c62e15297034bf4aafe61 (libosmocore)
Related: OS#2541
Change-Id: If232c80bea35d5c6864b889ae92d477eeaa3f45d
It may happen that either the MS or an ESME would become
unresponsive, e.g. due to a bug, or a dropped message.
This is why we have SS session timeout, that prevents
keeping 'stalled' sessions forever.
Let's introduce a VTY option, which can be used to configure
this timer (by default it's set to 30 seconds):
hlr
...
! Use 0 to disable this timer
ncss-guard-timeout 30
Change-Id: I971fc2cee6fd46d4d5d6dac6c634e0b22fff183d
Related: OS#3717
Make use of pragma user_version to store our database schema version.
The present schema is now identitifed as 'version 0', which is also
the default value for databases on which we never ran the statement
'pragma user_version' before.
Only bootstrap the database if it hasn't been bootstrapped yet.
Previously, bootstrap SQL statements ran every time osmo-hlr
opened the database, and any errors were being ignored in SQL.
Instead, we now first run a query which checks whether tables
already exist, and only create them if necessary.
This change will allow future schema updates to work properly.
Prepare for future schema upgrades by adding a new command-line
option which enables upgrades. This option defaults to 'false'
in order to avoid accidental upgrades.
Change-Id: I8aeaa9a404b622657cbc7138106f38aa6ad8d01b
Related: OS#2838
Add the first "official" way to remove the MSISDN from a subscriber entry, to
go back to 'MSISDN: none' like just after 'subscriber create'.
Add VTY command 'subscriber <ID> update msisdn none' to drop the MSISDN from
the subscriber. (Like 'subscriber <ID> update aud3g none')
Add DB_STMT_DELETE_MSISDN_BY_IMSI.
In db_subscr_update_msisdn_by_imsi(), allow passing a NULL msisdn, and if NULL,
call above delete SQL statement.
Change-Id: I15419105ea461137776adb92d384d8985210c90e
When I wrote the osmo-hlr subscriber command, I failed to heed the common
'show foo' scheme and instead created a 'subscriber [...] show' command.
Relieve that weirdness by creating an alias that has 'show' at the start.
Arrange string macros so that the 'show subscriber' cmd doesn't end in a space
(the SUBSCR macro ends in a space ' ' to implicitly include the space to
commands like 'create', 'show', 'update').
Add the new command to test_nodes.vty and test_subscriber.vty.
Change-Id: I01ce9b0868302d40ed05c6a588316a194d6071e4
In libosmocore Change-Id Ia75c7067284ea225cffe13ca71bad05a7747ae66
we fixed the generation (saving) of logging configuration to use
one level of indent, rather than the previous broken implementation
with two levels of indent. This means we have to adjust the VTY
test expectations here.
Change-Id: I9282a59bbfad4cfc86e86c1212c74cccfe187ff8
Keep the rules to run the external-tests in tests/Makefile available for
invocation, to allow conveniently launching the tests manualy without the need
to pass --with-external-tests to ./configure first.
Change-Id: Ic28dbeabddee6b41af12b977e3fe59e663ee51a1
Remove 'logging level all' setting.
Tweak some more logging details (to my current favorite).
Add USSD example for showing the IMSI.
Change-Id: I8296832704d779df5f1b20a595b568c99780e64d
Since libosmocore
commit eb9284ba577d338f74653fcf09ebca0c397823eb
Change-Id I36f17c131cc70ce5a1aef62fd9693097de230cd4
"logging vty: deprecate 'all', introduce 'force-all'"
,
'logging level all' is replaced by 'force-all'.
Adjust the test script to not expect 'logging level all'.
While at it, remove some more expectations that aren't important.
Change-Id: Ia170f8416ebb60c499d2536078f43f28b61d0554
The osmo_verify_transcript_*.py do terminate the osmo-hlr process
in some unusual way, so the database file is not closed properly.
Let's remove temporary files after the tests execution.
Change-Id: I9e4c98e86c1d6b627bfee1acb4fa116460687483
There are some requests that are best served inside the HLR, as it
has access to subscriber information such as MSISDN and IMSI.
This unfortunately required quite some restructuring of the USSD
related structures including the VTY syntax for adding routes.
The default config file has been updated to replicate the *#100#
built-in behavior of old OsmoNITB.
Closes: OS#2566
Change-Id: I1d09fab810a6bb9ab02904de72dbc9e8a414f9f9
Move code to create an Insert Subscriber Data message into a common
function which can be shared by hlr.c and luop.c.
As a consequence, we always encode gsup.cn_domain in the corresponding
msgb and must adjust expected output of the 'gsup' test accordingly.
Change-Id: I6a92ca34cdaadca9eacc774bb1ca386c325ba865
Requested-by: neels
Related: OS#2785
Encode an Insert Subscr Data with is_ps == true to trigger the encoding bug
described in OS#3231, i.e. show that it is fixed.
Move osmo_gsup_addr_send() to a separate .c file, so that it can be overridden
in the regression test to just dump the msgb instead.
I used this test to reproduce issue OS#3231, and now that it's here we might as
well keep it, and possibly expand on it in the future.
Related: OS#3231
Change-Id: Id1453351758f3e1a9ff03bd99fefaf51886e77da
Ever since libosmocore Change-Id I1c931bff1f1723aa82bead9dfe548e4cc5b685e0
was merged, the VTY tests were broken. Let's fix that by adjusting
the expected test output.
Change-Id: I929ca7f366220b5d1238d43eddc96fa9dde916b6
After libosmocore I96a9b6b6a3a5e0b80513aa9eaa727ae8c9c7d7a1 the CTRL interface
returns stricter errors. Adjust the expectations of
test_subscriber_errors.ctrl to fix the external tests on master.
Change-Id: I9337b6b4f3fa8822c91760deb01f18a77a073d19
Don't log "deriving 2G from 3G" when we're actually calculating separately; log
it when we're actually deriving from 3G.
Add log "calculating 2G separately" in the right place.
The test output changes show that each test said "separate 2G" at the top while
logging "deriving 2G from 3G" further down, which was obviously wrong.
Change-Id: I6679d7ef8fdcae39a0c2aff7ac638e63dddb10dc
libosmocore Ie35a02555b76913bb12734a76fc40fde7ffb244d
"ctrl: on parse errors, return a detailed message to sender"
the test_subscriber_errors.ctrl test fails.
Adjust the expected error message.
Change-Id: I3aee1507721cd073f72369150d0fb3cff0fdf66f
Verify that it returns -ENOENT on non-existing IMSI and -ENOKEY for no auth
data.
Move the auc_compute_vectors() stub to the top near the db_get_auc() call, and
just return num_vec to get a successful return value when auth data is present.
Change-Id: Ic0158228afbd78b8ca21f62dffa9f868674682b9
Differentiate between "IMSI unknown" and "IMSI has no auth data": in case of
known IMSI lacking auth data, return -ENOKEY instead of -ENOENT.
Fix API doc comments to reflect what the functions actually return, on top of
adding the -ENOKEY detail.
Adjust db_test expectations from -ENOENT to -ENOKEY where appropriate.
Adjust VTY and CTRL command rc evaluation.
A subsequent patch will use these return values to log details on each of these
situations.
Change-Id: Icf6304d23585f2ed45e050fa27c787f2d66fd3f7
A user on openbsc@ complained that with SQLite 3.8.2, the db_test fails with
--- expected
+++ stderr
-DDB (2067) abort at 18 in [INSERT INTO subscriber (imsi) VALUES ($imsi)]: UNIQUE constraint failed: subscriber.imsi
+DDB (2067) abort at 35 in [INSERT INTO subscriber (imsi) VALUES ($imsi)]: UNIQUE constraint failed: subscriber.imsi
i.e. a trivial difference in the error message issued by SQLite.
For db_test, don't output any SQLite error messages: Add argument
enable_sqlite_logging, pass as true, except in db_test.c.
Remove the SQLite error messages from expected output.
(Note that there is a src/db_test.c program that's not of interest here, this
is about the tests/db/db_test.c)
Change-Id: I2513d71cc0072aef8d08f47d0a1959f311176229
Tweak unit test binaries to still used DEBUG loglevels, so that their expected
outputs remain unchanged (and nicely verbose).
Adjust test_nodes.vty, now expecting the 'notice' log levels upon
'show running-config'.
Change-Id: Ic061e61c9625b49cef8bc2a2c0b936e262c22268
If a database file is missing, osmo-hlr creates it, as is the default sqlite3
API behavior -- before this patch, that db file is created, but lacks useful
tables. Actually also create initial tables in it, as osmo-nitb did.
In effect, the 'vty-test' target in tests/Makefile.am no longer needs to create
a database manually. (The 'ctrl-test' still does, because it also wants to add
subscriber data on top of the bare tables.)
Note: it could be desirable to bail if the desired database file does not
exist. That is however a different semantic from this patch; this is not
changing the fact that a db file is created, this just creates a usable one.
Note: I am about to add osmo-hlr-db-tool to do database migration from
osmo-nitb. For that, it is desirable to bootstrap a usable database, which is
the core reason for this patch.
Don't plainly duplicate hlr.sql to .c, but create db_bootstrap.h as a
BUILT_SOURCE from reading in sql/hlr.sql and mangling via sed to a list of SQL
statement strings. On each db_open(), run this bootstrap sequence.
In sql/hlr.sql, these tweaks are necessary:
* Add 'IF NOT EXISTS' to 'CREATE TABLE', so that the bootstrap sequence can be
run on an already bootstrapped db.
* Drop the final comment at the bottom, which ended up being an empty SQL
statement and causing sqlite3 API errors, seemed to have no purpose anyway.
Note: by composing the statement strings as multiline and including the SQL
comments, sqlite3 actually retains the comments contained in table definitions
and prints them back during 'sqlite3 hlr.db .dump'.
Change-Id: If77dbbfe1af3e66aaec91cb6295b687f37678636
The -I includes should be in CFLAGS, not CPPFLAGS.
I noticed problems with it when trying to add an -I$(builddir) in an upcoming
patch that adds a BUILT_SOURCE, If77dbbfe1af3e66aaec91cb6295b687f37678636.
Change-Id: Ie57a04b7efc7a1e16cf0e3625d8ad2f0ef0089b0
Add a large enough subscriber id and add a test that ensures a leading zero is
not interpreted as octal, and that a leading 0x is invalid and not interpreted
as hexadecimal.
Change-Id: Ib468b7cb595cf52331ebb41e6de0e8f57f69e173
The previous commands are not conforming to how the CTRL interface is intended
to work:
SET enable-ps <IMSI>
SET disable-ps <IMSI>
SET status-ps <IMSI>
'status-ps' is a write-only command even though it returns the status.
'enable-ps' / 'disable-ps' indicate the value instead of a variable name of an
entity. The entity <IMSI> takes the place of the variable value.
See also https://lists.osmocom.org/pipermail/openbsc/2017-September/011236.html
Instead, replace with
SET subscriber.by-imsi-123456.ps-enabled {0,1}
GET subscriber.by-imsi-123456.ps-enabled
and also provide further CTRL functions while at it:
{SET,GET} subscriber.by-{imsi,msisdn,id}-123456.{cs,ps}-enabled {0,1}
GET subscriber.by-{imsi,msisdn,id}-123456.{info,info-aud,info-all}
Provide CTRL tests in the form of transcripts.
Adjust tests/test_subscriber.sql to feature nonzero SQN, to see some values for
SQN in the CTRL transcript tests. (This does not affect the VTY tests, because
that creates its own subscribers, and there's no VTY command to set the SQN.)
This is the first time an application uses CTRL_NODE ids that are defined
outside of libosmocore, see 'Depends' below.
Implementation choice: the first idea was to have a '.' between the 'by-xxx'
and the value, like:
subscriber.by-xxx.123456.function
but the difficulty with subscribers is that they are not in RAM, and I can't
just point node_data at a struct instance that is always there (like, say, a
global bts[0] struct in osmo-bsc). Instead, I want to store the selector and
later decide whether to read from the DB or whatever. With a '.' separating
things, the only way in a ctrl function to obtain both 'by-xxx' and '123456'
for picking a subscriber record would be to parse the entire variable path
string elements, including 'subscriber' and 'function', which would then also
clumsily fix at which node level we hook these commands; there could have been
separate CTRL_NODE_SUBSCR_BY_{IMSI,MSISDN,ID} parent nodes, but we cannot
introspect the current parent node dynamically within a ctrl function handler
(plus I'm not sure whether it's possible and a good idea to have the same
command under multiple parent nodes).
Rather than that, I store the 'by-foo-123' token in the node_data pointer to
have both bits of information pointed at by a single pointer; I use the
incoming command parsing to get this token pre-separated from surrounding node
names, and no need to re-allocate it, since the vector of tokens lives until
after command execution is complete. Each leaf command obtains this token from
cmd->node (aka node_data), and feeds this token to a common static function to
parse selector and value from it and to retrieve a subscriber record as needed.
(BTW, I have mentioned on the mailing list that this way might be necessary to
avoid numeric-only CTRL node names, but we don't need to, and that is not at
all related to this choice of structure.)
Depends: libosmocore I1bd62ae0d4eefde7e1517db15a2155640a1bab58
libosmocore Ic9dba0e4a1eb5a7dc3cee2f181b9024ed4fc7005
Change-Id: I98ee6a06b3aa6a67adb868e0b63b0e04eb42eb50
Vadim Yanitskiy
Oliver Smith
Harald Welte
Pau Espin