The choice of libdbi was one of the biggest early mistakes in (back
then) OpenBSC development. A database abstraction library that
prevents you from using proper prepared statements. Let's finally
abandon it and use sqlite3 directly, just like we do in osmo-hlr.
I decided to remove the database migration code as it would be relatively
cumbersome to port all of it to direct sqlite3 with prepared statements,
and it is prone to introduction of all kinds of errors. Since we don't
have a body of older database files and comprehensive migration tests,
it is safer to not offer migration code of uncertain quality. The last
schema revision (5) was introduced 5 years ago in 2017 (osmo-msc
v1.1.0), so it is considered an exceptionally rare case. People can
install osmo-msc 1.1.0 through 1.8.0 to upgrade to v5 before using
this new 'direct sqlite3' version of osmo-msc.
Change-Id: Ia334904289f92d014e7bd16b02b3b5817c12c790
Related: OS#5559, OS#5563, OS#5564
Let's disable category here since we don't care about its formatting here.
In any case, every test relying on logging output validation should
always explicitly state the config to avoid issues in the future if
default values change.
Change-Id: I089c0001fc75e81558c3e860827e4d434cf1eab3
Related: OS#5034
man memcp doesn't define exact values for returned integer, it only
specifices a meaning for the sign of it.
So it happens that different versions/implementations actually return
different values when this test is run, making it fail.
Let's simply drop that info from logs since anyways it's not useful.
Change-Id: I771fb8f4fc56f337b16561d005ff1803a386d1c6
Thanks to db_sms_test, it was discovered that storing an SMS with
empty TP-User-Data (TP-UDL=1) causes buffer overruns in libdbi
and it's SQLite3 driver (libdbdsqlite3):
DDB NOTICE test_db_sms_store('Empty TP-UD'): ==7791== Invalid write of size 2
==7791== at 0x857DC60: dbd_quote_binary (in /usr/lib/x86_64-linux-gnu/dbd/libdbdsqlite3.so)
==7791== by 0x5B2B321: dbi_conn_quote_binary_copy (in /usr/lib/x86_64-linux-gnu/libdbi.so.1.1.0)
==7791== by 0x4073B1: db_sms_store (db.c:701)
==7791== by 0x405BB5: test_db_sms_store (db_sms_test.c:310)
==7791== by 0x405BB5: main (db_sms_test.c:546)
==7791== Address 0x7ed1cf0 is 0 bytes after a block of size 0 alloc'd
==7791== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7791== by 0x857DC4B: dbd_quote_binary (in /usr/lib/x86_64-linux-gnu/dbd/libdbdsqlite3.so)
==7791== by 0x5B2B321: dbi_conn_quote_binary_copy (in /usr/lib/x86_64-linux-gnu/libdbi.so.1.1.0)
==7791== by 0x4073B1: db_sms_store (db.c:701)
==7791== by 0x405BB5: test_db_sms_store (db_sms_test.c:310)
==7791== by 0x405BB5: main (db_sms_test.c:546)
...
DDB NOTICE test_db_sms_get('Empty TP-UD'): ==8051== Invalid read of size 1
==8051== at 0x5B30510: _dbd_decode_binary (in /usr/lib/x86_64-linux-gnu/libdbi.so.1.1.0)
==8051== by 0x857D957: dbd_fetch_row (in /usr/lib/x86_64-linux-gnu/dbd/libdbdsqlite3.so)
==8051== by 0x5B2C86E: dbi_result_seek_row (in /usr/lib/x86_64-linux-gnu/libdbi.so.1.1.0)
==8051== by 0x40828F: next_row (db.c:188)
==8051== by 0x40828F: db_sms_get (db.c:805)
==8051== by 0x406C29: test_db_sms_get (db_sms_test.c:390)
==8051== by 0x405C14: main (db_sms_test.c:547)
==8051== Address 0x8f74641 is 0 bytes after a block of size 1 alloc'd
==8051== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8051== by 0x5DBEB49: strdup (strdup.c:42)
==8051== by 0x857D93C: dbd_fetch_row (in /usr/lib/x86_64-linux-gnu/dbd/libdbdsqlite3.so)
==8051== by 0x5B2C86E: dbi_result_seek_row (in /usr/lib/x86_64-linux-gnu/libdbi.so.1.1.0)
==8051== by 0x40828F: next_row (db.c:188)
==8051== by 0x40828F: db_sms_get (db.c:805)
==8051== by 0x406C29: test_db_sms_get (db_sms_test.c:390)
==8051== by 0x405C14: main (db_sms_test.c:547)
==8051==
success, as expected
DDB NOTICE verify_sms('Empty TP-UD'): user_data_len mismatch: E0 vs A3
Apparently, dbi_conn_quote_binary_copy() doesn't properly handle
zero-length input. Let's guard against this.
Observed with:
- libdbi-dev 0.9.0-1
- libdbd-sqlite3:amd64 0.9.0-2ubuntu2
Change-Id: If0b2bb557118c5f0e520a2e6c2816336f6028661
Since OsmoMSC has built-in SMSC, it needs to store the messages
somewhere. Currently we use libdbi and SQLite3 back-end for that.
For a long time, the db_sms_* API remained uncovered by unit tests.
This change aims to fix that, and does cover the following calls:
- db_sms_store(),
- db_sms_get(),
- db_sms_get_next_unsent(),
- db_sms_mark_delivered(),
- db_sms_delete_sent_message_by_id(),
- db_sms_delete_by_msisdn(),
- db_sms_delete_oldest_expired_message().
Due to performance reasons, the test database is initialized in
RAM using the magic filename ':memory:'. This is a feature of
SQLite3 (and not libdbi), see:
https://www.sqlite.org/inmemorydb.html
Of course, this unit test helped to discover some problems:
1) Storing an SMS with empty TP-User-Data (TP-UDL=0) causes
buffer overruns in both db_sms_store() and db_sms_get().
2) TP-User-Data-Length is always being interpreted in octets,
regardless of DCS (Data Coding Scheme). This results in
storing garbage in the database if the default 7-bit
encoding is used. Fortunately, the 'user_data' buffer
in structure 'gsm_sms' is large emough, so we don't
experience buffer overruns.
3) db_sms_delete_oldest_expired_message() doesn't work
as expected. Instead of removing the *oldest* expired
message, it tries to remove the *newest* one.
The current test expectations do reflect these problems.
All of them will be fixed in the follow-up patches.
Change-Id: Id94ad35b6f78f839137db2e17010fbf9b40111a3
Harald Welte
Pau Espin
Vadim Yanitskiy