Each VLR requesting auth tuples should use a distinct IND pool for 3G
auth. So far we tied the IND to the GSUP peer connection; MSC and SGSN
were always distinct GSUP peers, they ended up using distinct INDs.
However, we have implemented a GSUP proxy, so that, in a distributed
setup, a remotely roaming subscriber has only one direct GSUP peer
proxying for both remote MSC and SGSN. That means as soon as a
subscriber roams to a different site, we would use the GSUP proxy name
to determine the IND instead of the separate MSC and SGSN. The site's
MSC and SGSN appear as the same client, get the same IND bucket, waste
SQNs rapidly and cause auth tuple generation load.
So instead of using the local client as IND, persistently keep a list of
VLR names and assign a different IND to each. Use the
gsup_req->source_name as indicator, which reflects the actual remote
VLR's name (remote MSC or SGSN).
Persist the site <-> IND assignments in the database.
Add an IND test to db_test.c
There was an earlier patch version that separated the IND pools by
cn_domain, but it turned out to add complex semantics, while only
solving one aspect of the "adjacent VLR" problem. We need a solution not
only for CS vs PS, but also for 2,3G vs 4G, and for sites that are
physically adjacent to each other. This patch version does not offer any
automatic solution for that -- as soon as more than 2^IND_bitlen
(usually 32) VLRs show up, it is the responsibility of the admin to
ensure the 'ind' table in the hlr.db does not have unfortunate IND
assignments. So far no VTY commands exist for that, they may be added in
the future.
Related: OS#4319
Change-Id: I6f0a6bbef3a27507605c3b4a0e1a89bdfd468374
Caught by new gcc 11.1.0:
/git/osmo-hlr/src/db.c:257:16: error: ‘rc’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
257 | return rc;
| ^~
Change-Id: I45e1fede234cd4446a7061fb908bf344eee72f5a
In 89fda3024a I added a
vty command to show a summary of filtered subscribers by imsi
or msisdn. In practice there is a also need to be able to
filter on IMEI.
The idea here is not to replace every operation that could be
done directly on the sql database in the vty, but this one
is useful.
Change-Id: Ic4a11d3ebcf8909c68e9f4617e94dc822491e008
Adds the following commands:
show subscribers all - Display summary of all entries in HLR
show subscribers (imsi|msisdn|cs|ps) ... As above but filter on search field/string
show subscribers last seen - Display only subscribers with data in
Last LU update field, and sorts by Last LU.
Change-Id: I7f0573381a6d0d13841ac6d42d50f0e8389decf4
D-GSM will store in the HLR DB whether a locally connected MSC has attached the
subscriber (last_lu_seen[_ps]), or whether the attach happened via a GSUP proxy
from a different site.
Add columns for this separately in this patch.
Change-Id: I98c7b3870559ede84adf56e4bf111f53c7487745
From https://sqlite.org/c3ref/exec.html:
To avoid memory leaks, the application should invoke sqlite3_free()
on error message strings returned through the 5th parameter of
sqlite3_exec() after the error message string is no longer needed.
If the 5th parameter to sqlite3_exec() is not NULL and no errors
occur, then sqlite3_exec() sets the pointer in its 5th parameter
to NULL before returning.
Change-Id: Ic9ed9bad3165bc4a637fe963f51e923f012e19ac
Fixes: CID#208182
Location Updating procedures from both CS and PS overwrite the same
last_lu_seen field of a subscriber. For upcoming D-GSM it will be important to
distinguish those, because only CS attaches qualify for MSISDN lookup.
Add column last_lu_seen_ps, and upon PS LU, do not overwrite last_lu_seen, so
that last_lu_seen now only reflects CS LU.
In the VTY, dump both LU dates distinctively.
Change-Id: Id7fc50567211a0870ac0524f6dee94d4513781ba
Instead of a switch() for each version number with identical switch cases
except for the function name, use an array of function pointers and loop.
Also print a success message after each individual version upgrade, instead of
only one in the end (see change in db_upgrade_test.ok).
Change-Id: I1736af3d9a3f02e29db836966ac15ce49f94737b
Apply the same headers structure that we keep in most Osmocom source trees:
Keep noinst_HEADERS in include/osmocom/hlr and include them using
#include <osmocom/hlr/*.h>
The only header kept in src/ is db_bootstrap.h, because it is generated during
build time. If it was built in include/osmocom/hlr, we would need db.o to
depend on db_bootstrap.h in a different subdir, which automake can't do well.
Change-Id: Ic912fe27f545b85443c5fb713d8c3c8aac23c9ad
The db bootstrap as well as the upgrade code all execute a number of
statements, and have massive code dup around each statement execution. Instead
have one db_run_statements() that takes an array of strings and runs all.
Change-Id: I2721dfc0a9aadcc7f5ac81a1c0fa87452098996f
The osmo-hlr DB schema indicates a hlr_number column and references it as 3GPP
TS 23.008 chapter 2.4.6. However, chapter 2.4.6 refers to the "MSC number",
while the "HLR number" is chapter 2.4.7.
Taking a closer look, 2.4.6 says "The MSC number is [...] stored in the HLR",
while 2.4.7 says "The HLR number may be stored in the VLR". As quite obvious,
the HLR does not store the HLR number. This was a typo from the start.
The osmo-hlr code base so far does not use the hlr_number column at all, so we
get away with renaming the column without any effects on the code base.
However, let's rather make this a new schema version to be safe.
Change-Id: I527e8627b24b79f3e9eec32675c7f5a3a6d25440
A subsequent commit will add a db_upgrade test, which verifies that the db
resulting from an upgrade is identical to one created from scratch in the new
version. That test currently would show a diff: an upgraded 'imei' column has
'default NULL', where a new db created in version 2 has no default value on the
imei column.
Fix the upgrade path to add an imei column without 'default NULL', so that
adding the upgrade test will result in success. The test is added in
I0961bab0e17cfde5b030576c5bc243c2b51d9dc4
Change-Id: I68a00014a3d603fcba8781470bc5285f78b538d0
This change introduces an optional feature that allows to make
SQLite3 use talloc for all internal allocations. This would
facilitate finding memleaks. OsmoHLR needs to be configured
with '--enable-sqlite-talloc'.
full talloc report on 'OsmoHLR' (total 292168 bytes in 449 blocks)
struct osmo_gsup_server contains 162 bytes in 3 blocks (ref 0)
...
struct db_context contains 288407 bytes in 420 blocks (ref 0)
hlr.db contains 7 bytes in 1 blocks (ref 0)
SQLite3 contains 288192 bytes in 418 blocks (ref 0)
db.c:95 contains 48 bytes in 1 blocks (ref 0)
db.c:95 contains 2 bytes in 1 blocks (ref 0)
...
Unfortunately, old SQLite3 versions (such as 3.8.2) run out
of memory when trying to initialize a new database:
DDB ERROR db.c:88 (7) statement aborts at 3: []
DDB ERROR db.c:420 Unable to set Write-Ahead Logging: out of memory
DDB ERROR db.c:88 (7) statement aborts at 3: []
DDB ERROR db.c:238 Unable to prepare SQL statement
'SELECT name FROM sqlite_master WHERE type='table' AND name=?'
...
I've noticed a huge difference in heap usage footprint compared to
generic malloc. At the same time, the recent versions (at least
3.24.0), work just fine.
Change-Id: Icfe67ed0f063b63e6794f9516da3003d01cf20a7
Check if a subscriber exists without generating an error log entry if
it does not. This is cheaper than db_subscr_get_by_msisdn(), as it
does not fetch the subscriber entry.
subscriber-create-on-demand will use this function to generate
a random unique MSISDN for new subscribers.
Related: OS#2542
Change-Id: Ibfbc408c966197682ba2b12d166ade4bfeb7abc2
Check if a subscriber exists without generating an error log entry if
it does not. This is cheaper than db_subscr_get_by_imsi(), as it does
not fetch the subscriber entry. subscriber-create-on-demand will use
this function.
Related: OS#2542
Change-Id: I63818c0dd4fd22b41dadeeba2a07a651b5454c54
Allow creating new subscribers without giving them access to CS or PS.
This will be used by the create-subscriber-on-demand feature.
Related: OS#2542
Change-Id: I1a6dd85387723dab5487c53b33d2d9ec6d05d006
Extend the database scheme, add imei to the hlr_subscriber struct and
create db_subscr_update_imei_by_imsi() and db_subscr_get_by_imei(). The
new functions are used in db_test, and in follow-up commits [1], [2].
Upgrade DB schema to version 2. SQLite can only insert new columns at
the end of the table, so this happens when upgrading the database. In
new databases, the column is placed after the IMEISV column (where it
makes more sense in my opinion). This should not have any effect, as
we never rely on the order of the columns in the tables.
Follow-up commit [1] will make use of this column to save the IMEI as
received from the MSC/VLR with the Check-IMEI Procedure. It was
decided to use Check-IMEI instead of the recent Automatic Device
Detection Procedure (which would send the IMEISV) in OS#3733, because
with Check-IMEI we don't need to rely on very recent releases of the
specification.
[1] change-id I09274ecbed64224f7ae305e09ede773931da2a57
"Optionally store IMEI in subscriber table"
[2] change-id I1af7b573ca2a1cb22497052665012d9c1acf3b30
"VTY: integrate IMEI"
Depends: Id2d2a3a93b033bafc74c62e15297034bf4aafe61 (libosmocore)
Related: OS#2541
Change-Id: If232c80bea35d5c6864b889ae92d477eeaa3f45d
Timestamps are stored in the HLR DB in the new 'last_lu_seen' column
of the 'subscriber' table, in UTC and in granularity of seconds.
At present, osmo-hlr only records these timestamps but otherwise
makes no use of them. Because the timestamps are stored in a
human-readable form, they may already provide value to external
processes which need this information. For example:
sqlite> select imsi,last_lu_seen from subscriber;
901990000000001|2018-12-04 14:17:12
I didn't bother adding additional tests because the code added
with this commit is already being exercised by several calls
to db_subscr_lu() in db_test.c.
This change requires a HLR DB schema update. Existing databases
won't be upgraded automatically. However, osmo-hlr will refuse
to operate with databases which are not upgraded.
Change-Id: Ibeb49d45aec18451a260a6654b8c51b8fc3bec50
Related: OS#2838
Make use of pragma user_version to store our database schema version.
The present schema is now identitifed as 'version 0', which is also
the default value for databases on which we never ran the statement
'pragma user_version' before.
Only bootstrap the database if it hasn't been bootstrapped yet.
Previously, bootstrap SQL statements ran every time osmo-hlr
opened the database, and any errors were being ignored in SQL.
Instead, we now first run a query which checks whether tables
already exist, and only create them if necessary.
This change will allow future schema updates to work properly.
Prepare for future schema upgrades by adding a new command-line
option which enables upgrades. This option defaults to 'false'
in order to avoid accidental upgrades.
Change-Id: I8aeaa9a404b622657cbc7138106f38aa6ad8d01b
Related: OS#2838
Add the first "official" way to remove the MSISDN from a subscriber entry, to
go back to 'MSISDN: none' like just after 'subscriber create'.
Add VTY command 'subscriber <ID> update msisdn none' to drop the MSISDN from
the subscriber. (Like 'subscriber <ID> update aud3g none')
Add DB_STMT_DELETE_MSISDN_BY_IMSI.
In db_subscr_update_msisdn_by_imsi(), allow passing a NULL msisdn, and if NULL,
call above delete SQL statement.
Change-Id: I15419105ea461137776adb92d384d8985210c90e
Thanks to ASAN, it was discovered that some part of heap
is not released on exit:
==19736==ERROR: LeakSanitizer: detected memory leaks
Indirect leak of 94616 byte(s) in 214 object(s) allocated from:
#0 0x4e05c6 (/home/wmn/osmocom/osmo-hlr/src/osmo-hlr+0x4e05c6)
#1 0x7f9b01061dc6 (/usr/lib/x86_64-linux-gnu/libsqlite3.so.0+0x33dc6)
Indirect leak of 1160 byte(s) in 1 object(s) allocated from:
#0 0x4e097d (/home/wmn/osmocom/osmo-hlr/src/osmo-hlr+0x4e097d)
#1 0x7f9b01061d58 (/usr/lib/x86_64-linux-gnu/libsqlite3.so.0+0x33d58)
SUMMARY: AddressSanitizer: 95776 byte(s) leaked in 215 allocation(s).
After a long investigation, it was figured out that *sqlite never
closes the database* due to 'unfinalized statements or unfinished
backups'.
The problem was in db_bootstrap(), where several statements were
prepared, but not finalized in loop. This was also the reason of
*.db-shm / *.db-wal files remaining after the program is closed,
and the reason of the following message
db.c:77 (283) recovered 18 frames from WAL file *.db-wal
Let's fix this and stop ignoring the result of sqlite3_close().
Change-Id: Ibe620d7723b1947d4f60f820bd18435ad0193112
Related: OS#3434
A user on openbsc@ complained that with SQLite 3.8.2, the db_test fails with
--- expected
+++ stderr
-DDB (2067) abort at 18 in [INSERT INTO subscriber (imsi) VALUES ($imsi)]: UNIQUE constraint failed: subscriber.imsi
+DDB (2067) abort at 35 in [INSERT INTO subscriber (imsi) VALUES ($imsi)]: UNIQUE constraint failed: subscriber.imsi
i.e. a trivial difference in the error message issued by SQLite.
For db_test, don't output any SQLite error messages: Add argument
enable_sqlite_logging, pass as true, except in db_test.c.
Remove the SQLite error messages from expected output.
(Note that there is a src/db_test.c program that's not of interest here, this
is about the tests/db/db_test.c)
Change-Id: I2513d71cc0072aef8d08f47d0a1959f311176229
Coverity wants us to evaluate sqlite3_reset, but it is of no use to do so.
Related: coverity CID#178653
Change-Id: I64ac8c148f48be60f9c0d346df0c5152bb527494
If a database file is missing, osmo-hlr creates it, as is the default sqlite3
API behavior -- before this patch, that db file is created, but lacks useful
tables. Actually also create initial tables in it, as osmo-nitb did.
In effect, the 'vty-test' target in tests/Makefile.am no longer needs to create
a database manually. (The 'ctrl-test' still does, because it also wants to add
subscriber data on top of the bare tables.)
Note: it could be desirable to bail if the desired database file does not
exist. That is however a different semantic from this patch; this is not
changing the fact that a db file is created, this just creates a usable one.
Note: I am about to add osmo-hlr-db-tool to do database migration from
osmo-nitb. For that, it is desirable to bootstrap a usable database, which is
the core reason for this patch.
Don't plainly duplicate hlr.sql to .c, but create db_bootstrap.h as a
BUILT_SOURCE from reading in sql/hlr.sql and mangling via sed to a list of SQL
statement strings. On each db_open(), run this bootstrap sequence.
In sql/hlr.sql, these tweaks are necessary:
* Add 'IF NOT EXISTS' to 'CREATE TABLE', so that the bootstrap sequence can be
run on an already bootstrapped db.
* Drop the final comment at the bottom, which ended up being an empty SQL
statement and causing sqlite3 API errors, seemed to have no purpose anyway.
Note: by composing the statement strings as multiline and including the SQL
comments, sqlite3 actually retains the comments contained in table definitions
and prints them back during 'sqlite3 hlr.db .dump'.
Change-Id: If77dbbfe1af3e66aaec91cb6295b687f37678636
To make the db_bind_text() error reporting mention "imsi", change the
DB_STMT_AUC_BY_IMSI to use a named parameter.
Change-Id: I49bd5eb78170cf4cdf8abb386c766d20d9f1cf73
Use the common db_bind_int64() so that the stmt bindings are cleared for any
errors and to get error logging for free.
On error with sqlite3_step(), log the SQL error message, and make sure the stmt
is cleared of bindings and reset.
After sqlite3_step(), verify that exactly one row was modifed, log and return
errors otherwise.
After this patch, the DB interaction closely matches the other (refactored) DB
functions.
Change-Id: I0d870d405e2e0a830360d9ad19f0a3f9e09d8cf2
Add ability to add and remove auc_2g and auc_3g table rows with
db_subscr_update_aud_by_id().
In db_subscr_delete_by_id(), make sure that when deleting a subscriber, also
all auth data associated with that user ID is removed as well. A newly created
subscriber must not obtain the same auth tokens just by getting the same id.
Depends: libosmocore Idf75946eb0a84e145adad13fc7c78bb7a267aa0a
Change-Id: Icb11b5e059fb920447a9aa414db1819a0c020529
Use named parameters in the SQL statements.
Use db_bind_* functions to drop some code dup.
Adopt error handling (rc and logging) to match the other db functions: return
-ENOENT for unknown subscriber, -EIO for SQL failures.
Change-Id: Iad49d29b90a708c6cf55bfb3bcc02d9e29001a15
Use named parameters in the SQL statement.
Use db_bind_* functions to drop some code dup.
Use explicit subscriber id arg instead of subscriber struct.
Match return values and error logging to other db functions.
Change-Id: I35665e84ddbe54a6f218b24033df969ad2e669a0
Allow to set nam_ps and nam_cs from this same function, by adding the is_ps
arg.
Combine both NAM_PS stmts to DB_STMT_UPD_NAM_PS_BY_IMSI, add another such stmt
for CS. Use named parameters instead of parameter indexes.
Improve error return values as well as error logging to clearly indicate
whether the operation could not find the requested IMSI, or other errors
occured.
Adjust the single caller.
This prepares for upcoming VTY and possibly CTRL commands, and the error
handling introduced here has been or will be adopted by other functions in
previous or subsequent patches.
Change-Id: I6e70e15228f5bb10bee6758ae5dc9687d65839bd
Factor out the selected SQL columns as SEL_COLUMNS macro, so that each of the
new DB_STMTs will select identical columns: the old DB_STMT_SEL_BY_IMSI as well
as the new DB_STMT_SEL_BY_MSISDN and DB_STMT_SEL_BY_ID.
Add the new functions db_subscr_get_by_msisdn() and db_subscr_get_by_id() and
factor out common parts with db_subscr_get_by_imsi() to static db_sel().
Change-Id: I6d0ddd1b7e3f6b180b4b1b2663c5725d2a4a9428
db_remove_reset() needs to be called after each stmt run, whether it succeeded
or not.
In case sqlite3_clear_bindings() would fail to unbind a stmt, we would anyway
be beyond recovery. There seem to be no plausible situations where such failure
would occur, unless there have been no bindings in the first place.
In case there was an SQL stmt failure, sqlite3_reset() will re-barf the same
error message, we will always have logged it earlier already in the proper
context.
We are never evaluating the return value, nor would we know how to recover from
non-success.
The conclusions:
- db_remove_reset() does not need to log any errors.
- db_remove_reset() does not need to return success.
Change-Id: I21678463e59f607f5f5c5732963e274392f0fffd
These will be needed by VTY commands to create, delete and modify subscribers.
Auth data editing will follow in another patch.
The FIXME "also remove authentication data from auc_2g and auc_3g" will get
fixed in change-id Icb11b5e059fb920447a9aa414db1819a0c020529.
Change-Id: I725273d36234331093e7fff7d5f12f6be6ab2623
Will be used in upcoming patches, e.g. change-IDs
- I6e70e15228f5bb10bee6758ae5dc9687d65839bd
- I83a47289a48ac37da0f712845d422e897a5e8171
Change-Id: I705a15eef242c98feb6e95a883916f6cf8173d70
SQLite3 seems to be commonly compiled without log callback support. It is then
misleading to see a seeming error message about this on each osmo-hlr startup.
Avoid the impression that we would miss out on important logging: query
sqlit3_compileoption_get() whether SQLITE_CONFIG_SQLLOG is enabled. Try to
register the callback only if present, if not, say so on DEBUG log.
See https://sqlite.org/compile.html "SQLITE_ENABLE_SQLLOG"
Change-Id: I78d75dc351eb587b0a022f82f147e9a31c0324c5
In multiple lines, the statement becomes more readable.
I'd like to get this change out of the way before upcoming SQL statement edits
and additions.
Change-Id: Icf09f4bbb298a516aa52c81e3ca67d9d91d8c7c2
There are more uses for a generalized db_bind_text(), and in an upcoming patch
there will be similar functions like db_bind_int().
Also, add argument param_name, optionally indicating a named SQL parameter to
bind to, which will be used in subsequent patches. So far, all callers pass
NULL to yield previous db_bind_imsi() behavior of binding to the first param.
Change-Id: I87bc46a23a724677e8319d6a4b032976b7ba9394
There are upcoming additions, and some seem too general without a proper common
prefix in the identifiers, like 'CREATE'.
Change-Id: I51b677db31a1ebbbc45dc7925074de7493fbde1f
Add ind_bitlen column to auc_3g to record each USIM's IND size according to
3GPP TS 33.102 -- default is 5 bits, as suggested by the spec.
Introduce auc_3g_ind to each connecting GSUP client to use as IND index for
generating auth tuples sent to this client.
With osmo_gsup_server_add_conn(), implement a scheme where clients receive
fixed auc_3g_ind indexes based on the order in which they connect; each new
connection takes the lowest unused auc_3g_ind, so in case one of the clients
restarts, it will most likely receive the same auc_3g_ind, and if one client
disconnects, no other clients' auc_3g_ind are affected.
Add gsup_server_test.c to test the auc_3g_ind index distribution scheme.
Depends: libosmocore I4eac5be0c0b2cede04464c4c3a0873102d952453 for llist_first
Related: OS#1969
Change-Id: If4501ed4ff8e923fa6fe8b80c44c5ad647a8ed60
Also, fix log formatting: SQL statements do not have '\n' at the end.
Note: sqlite should be compiled with SQLITE_ENABLE_SQLLOG for this code
to work at all.
Change-Id: I5e53de54ad1b9da18e1f414932cfd21be71ab154
* move common cleanup code into separate function
* add helper function for IMSI binding
* use errno.h instead of numbers
Change-Id: Iec81b56ab1ccc948807854a3947b04355a555c10
Parse commandline options, supporting general Osmocom options as copied from
osmo-nitb (bsc_hack.c): version, logging and daemonize options.
Set the HLR database file from cmdline option, log the filename in db_open().
(VTY config file in next patch.)
Change-Id: I279d517e1310e398b0a2382349e62be8e65364c1
We also introduce a 'gsup_router' which enables us to route
a transaction to a given VLR. It works based on the SERIAL attribute
communicated at time of the IPA multiplex setup as part of the CCM
sub-protocol.
Neels Hofmeyr
Pau Espin
Keith Whyte
Oliver Smith
Neels Hofmeyr
Harald Welte