x = application code, _ = time spent in the database
Developed ~2000 by D. Richard Hipp
(Dwayne Richard Hipp)
Developed as kind of SQL database stub (my words)
Put into public domain
Took off
Developed ~2000 by D. Richard Hipp
(Dwayne Richard Hipp)
Developed as kind of SQL database stub (my words)
Put into public domain
Took off
Any SQLite3 user here?
…MariaDB/MySQL, PostgreSQL, MSSQL, Oracle…
…MariaDB/MySQL, PostgreSQL, MSSQL, Oracle…
attach up to 125 databases
up to 2,147,483,646 tables
up to 2^64 rows in a tables
140 terabytes
Not the small version of [you name it]
Not the small version of [you name it]
Shipped as a single C header/source file
Super simple to add to your application
It just works, defaults already OK
Optimized/fine-tuned via compile time options
Not the small version of [you name it]
Zero config relational database
Can be statically linked into your application
Not the small version of [you name it]
Zero config relational database
Can be statically linked into your application
The app decides the thread model, 0..n
Multiple writers not the primary use case
(file locking)
Relational data layout and key value storage
Support of transactions
Therefore often the better own file format
Not the small version of [you name it]
Zero config relational database
Can be statically linked into your application
In process relational database
Relational application file format
Json
Full text search
R-Tree tables
CSV tables from file
….
A great opens source project
Excellent tested (FAA DO-178 B or C)
Well documented
An in-memory data store
….
Not the small version of [you name it]
In process relational database
Zero config relational database
Relational application file format
Can be statically linked into your app
And much more … and even more …
Since I have it in the title ….
Since every presentation needs some charts ….
x = runtime of application code
_ = run time spent in the database
x = application code, _ = time spent in the database
x = application code, _ = time spent in the database
Base case: some ORM and/or library …
xxxxx_______________________________________________
x = application code, _ = time spent in the database
Base case: some ORM and/or library …
xxxxx_______________________________________________
Optimizing application code
xx________________________________________________
x = application code, _ = time spent in the database
Base case: some ORM and/or library …
xxxxx_______________________________________________
Optimizing application code
xx________________________________________________
Understanding/using SQL + database normalization
xx_______________
If you care about performance …
... care about the things that matter.
Care about the things that matter
Different type of performance
Time to market …
Daily migration of schema and data due to new customer requests, …
Run with whatever back-end wanted/needed, like bug tracer, wiki, shop software, …
Care about the things that matter
If it is runtime, usually for C++ , embedded
SQL
database normalization
indexes
query plan execution
and of course somehow sane C++
Since C++11 SQL is a pleasure to write in C++ code
Since C++11 SQL is a pleasure to write in C++ code
constexpr const char*
mySqlStatement()
{
R"~(
select/insert/update ...
... what ever ...
... where ever ...
;
)~"
}
Thanks to raw string literals !
4th generation declarative language
Relational algebra
Tuple relational calculus
Aggregate functions
Scalar functions
DDD
Data definition language
Data manipulation language
Data control language (*)
* In sqlite just file permissions
Entity is a table, columns are attributes
Attributes might have constraints
There might be relations between entities
1:1, 1 to many / many to 1, many to many
The goal:
keep redundancy to zero in the storage
ensure data correctness on storage level
Create tables, typed attributes(*) and relations
CREATE TABLE artist(
id INTEGER PRIMARY KEY,
name TEXT
);
CREATE TABLE track(
id INTEGER,
name TEXT,
artist INTEGER ,
FOREIGN KEY(artist) REFERENCES artist(id)
);
A track without artist is impossible to have.
A well defined storage is always in a well defined state
No headache, it just works
In relational databases we trust
INSERT, UPDATE, DELETE
SELECT, ORDERED BY, GROUPED BY
Aggregate functions (avg, count, sum, …)
Scalar functions (trim, length, …)
JOINS (relations)
Just another programming challenge
An elegant SQL statement replaces boring spaghetti code and is much faster.
Data driven design/programming
Table driven design/programming
Ask the data storage questions …
Do list manipulation on the data storage …
make own aggregate/scalar functions in C++
event driven (trigger) …
on insert/update/delete ….
in process … Ideas? :-)
The basics are simple
Basics can be learned in a hand full of hours
Not everyone needs to be an expert
Do not abstract SQL away (invent a own DSL)
Put data there where it should be
ERM has often more payoff than Class/Object diagram
WITH RECURSIVE
xaxis(x) AS (VALUES(-2.0)
UNION ALL SELECT x+0.05 FROM xaxis WHERE x<1.2),
yaxis(y) AS (VALUES(-1.0)
UNION ALL SELECT y+0.1 FROM yaxis WHERE y<1.0),
m(iter, cx, cy, x, y) AS (
SELECT 0, x, y, 0.0, 0.0 FROM xaxis, yaxis
UNION ALL
SELECT iter+1, cx, cy, x*x-y*y + cx, 2.0*x*y + cy FROM m
WHERE (x*x + y*y) < 4.0 AND iter<28),
m2(iter, cx, cy) AS (
SELECT max(iter), cx, cy FROM m GROUP BY cx, cy),
a(t) AS (
SELECT group_concat( substr(' .+*#', 1+min(iter/7,4), 1), '')
FROM m2 GROUP BY cy)
SELECT group_concat(rtrim(t),x'0a') FROM a;
Might be a lite bit fast
not to memorize all details
to see how simple it is
lay back
fasten your seat belts
relax
sqlite3* db = nullptr;
auto rc = sqlite3_open (name, &db);
if(rc != SQLITE_OK) {
std::cerr << "Unable to open database '" << name << "': "
<< sqlite3_errmsg (db);
sqlite3_close (db);
std::exit(EXIT_FAILURE);
}
using the database and than, of course
// do not forgett to close !!!
sqlite3_close (db);
using database =
std::unique_ptr<sqlite3, decltype(&sqlite3_close)> ;
database open_database(const char* name)
{
sqlite3* db = nullptr;
auto rc = sqlite3_open (name, &db);
if(rc != SQLITE_OK) {
std::cerr << "Unable to open database '" << name << "': "
<< sqlite3_errmsg (db);
sqlite3_close (db);
std::exit(EXIT_FAILURE);
}
return database{db, sqlite3_close} ;
}
From now on just use
auto db = open_database(":memory:");
void execute (not_null<sqlite3*> db, const char* sql)
{
char* errmsg = 0;
int rc = sqlite3_exec (db.get (), sql, 0, 0, &errmsg);
if (rc != SQLITE_OK) {
std::cerr << "Unable to execute '" << sql << "': "
<< errmsg ;
sqlite3_free(errmsg) ;
std::exit(EXIT_FAILURE);
}
}
// Note: The not_null is from the MS GSL.
constexpr const char* create_things()
{
return R"~(BEGIN TRANSACTION ;
CREATE TABLE things(id INTEGER PRIMARY KEY, name TEXT,value REAL);
INSERT INTO things VALUES(1,'one', 1.1);
INSERT INTO things VALUES(2,'two', 2.2);
COMMIT TRANSACTION ;
)~";
}
int main()
{
auto db = open_database(":memory:");
execute (db.get(), create_things());
// run your program....
}
And now I have all - or no - things in memory.
JASG: Just another scope guard
JASG: Just another scope guard
struct Transaction
{
Transaction(not_null<sqlite3*> db) : _db{db}{
execute(_db, "BEGIN TRANSACTION;") ;
}
~Transaction() {
if(_db) execute(_db, "ROLLBACK TRANSACTION;") ;
}
void commit() {
if(_db) execute(_db, "COMMIT TRANSACTION;") ;
_db = nullptr ;
}
Transaction (Transaction&&) = default ;
Transaction (Transaction&) = delete ;
Transaction& operator=(Transaction&) = delete ;
Transaction& operator=(Transaction&&) = delete ;
private:
sqlite3* _db ;
};
JASG: Just another scope guard
constexpr const char* create_things()
{
return R"~(
CREATE TABLE things(id INTEGER PRIMARY KEY, name TEXT,value REAL);
INSERT INTO things VALUES(1,'one', 1.1);
INSERT INTO things VALUES(2,'two', 2.2); )~";
}
int main()
{
auto db = open_database(":memory:");{
Transaction transaction{db};
execute (db.get(), create_things());
transaction.commit();
}
// now there is some data
}
each SQL command needs to be compiled, expensive
compiled SQL commands can have parameters
each SQL command needs to be compiled, expensive
compiled SQL commands can have parameters
-- sqlite syntax
INSERT INTO things VALUES(?,?,?); -- index 1, 2 , 3
-- or with names
INSERT INTO things VALUES(@id,@name,@value);
-- indexes still work
sqlite3_stmt* stmt = nullptr;
int rc = sqlite3_prepare_v2 (db,
sql.c_str (), sql.length(),
&stmt, nullptr);
if (rc != SQLITE_OK) {
std::cerr << "Unable to create statement '" << sql << "': "
<< sqlite3_errmsg(db.get ());
std::exit(EXIT_FAILURE);
}
use stmt
don’t forget
sqlite3_finalize(stmt) ;
using statement =
std::unique_ptr<sqlite3_stmt, decltype(&sqlite3_finalize)> ;
statement create_statement(not_null<sqlite3*> db,
const std::string& sql)
{
sqlite3_stmt* stmt = nullptr;
int rc = sqlite3_prepare_v2 (db,
sql.c_str (), sql.length(),
&stmt, nullptr);
if (rc != SQLITE_OK) {
std::cerr << "Unable to create statement '" << sql << "': "
<< sqlite3_errmsg(db.get ());
std::exit(EXIT_FAILURE);
}
return statement(stmt, sqlite3_finalize);
}
From now on just use
auto stmt = create_statement(db.get(),
"INSERT INTO things VALUES(@id,@name,@value);");
Extra lightning talks on how to do this exists …
Extra lightning talks on how to do this exists …
The point here is just to show sqlite3_bind_TYPE
→ rc sqlite3_bind_TYPE (stmt, index, TYPE_value)
void parameter(not_null<sqlite3_stmt*> stmt,
int index, int64_t value){
if (sqlite3_bind_int64 (stmt, index, value) != SQLITE_OK){
throw "TODO" ;
}
}
// void parameter( ... double value) sqlite3_bind_real
void parameter(not_null<sqlite3_stmt*> stmt,
int index, const std::string& value) {
if(sqlite3_bind_text (stmt.get(), index,
value.c_str (), value.size (),
SQLITE_TRANSIENT) != SQLITE_OK){
throw "TODO" ;
}
}
// void parameter( ... blob& value) sqlite3_bind_blob
// for temporary, SQLITE_TRANSIENT, sqlite get a copy
→ rc sqlite3_bind_TYPE (stmt, index, TYPE_value)
int/int64/double
null , no value
text/text16/text64
blob/blob16/blob64
Text and blob, SQLITE_TRANSIENT vs SQLITE_STATIC
sqlite3_step(stmt)
executes a statement.
There was an error
OK for INSERT/UPDATE/…
A row of data for you
Done, no more data
One way to dot it:
Give me a callback
If there is data, call the callback
If the callback says continue, step next
→ No need to write loops as user.
The callback can be a lambda, a function, function object, …
using stmt_callback = //callback to handle current row
std::function<bool(not_null<sqlite3_stmt*>)> ;
void run(not_null<sqlite3_stmt*> stmt,
stmt_callback callback = stmt_callback{})
{
using reset_guard // JASG: Just another scope guard
= std::unique_ptr<sqlite3_stmt, decltype (&sqlite3_reset)>;
auto reset = reset_guard (stmt.get(), &sqlite3_reset);
auto next_step = [&](int rc){
if (rc == SQLITE_OK || rc == SQLITE_DONE)
return false ;
else if (rc == SQLITE_ROW)
if(callback) return callback(stmt);
// else ... some error handling
return false ;
};
while(next_step(sqlite3_step(stmt))) ;
}
A callback example
bool dump_current_row(not_null<sqlite3_stmt*> stmt)
{
for (int i = 0 ; i < sqlite3_column_count(stmt); ++i) {
// get the current field value and print it
std::cout << "|" ;
}
std::cout << "\n" ;
return true ;
}
auto columntype = sqlite3_column_type(stmt, i) ;
if(columntype == SQLITE_NULL) {
std::cout << "<NULL>" ;
}
else if (columntype == SQLITE_INTEGER){
std::cout << sqlite3_column_int64(stmt, i);
}
else if (columntype == SQLITE_FLOAT){
std::cout << sqlite3_column_double(stmt, i) ;
}
else if (columntype == SQLITE_TEXT ){
auto first = sqlite3_column_text (stmt, i);
std::size_t s = sqlite3_column_bytes (stmt, i);
std::cout << "'" << (s > 0 ?
std::string((const char*)first, s) : "") << "'";
}
else if (columntype == SQLITE_BLOB ){
std::cout << "<BLO000B>" ;
}
sqlite3_column_count
sqlite3_column_type
sqlite3_column_bytes(16)
sqlite3_column_int(64)
sqlite3_column_double
sqlite3_column_text(16)
sqlite3_column_blob
sqlite3_open
sqlite3_execute
sqlite3_prepare_v2
sqlite3_step + return value
sqlite3_bind_TYPE .. app → sqlite
sqlite3_column_TYPE + helpers .. app ← sqlite
constexpr const char* create_things()
{
return R"~(
CREATE TABLE things(id INTEGER PRIMARY KEY, name TEXT,value REAL);
INSERT INTO things VALUES(1,'one', 1.1);
INSERT INTO things VALUES(2,'two', 2.2); )~";
}
int main()
{
auto db = open_database(":memory:");{
Transaction transaction{db};
execute (db.get(), create_things());
transaction.commit();
}
auto stmt = create_statement(db.get(), "SELECT * FROM things;");
run(stmt.get(), dump_current_row);
}
//
// 1|'one'|1.1|
// 2|'two'|2.2|
Column handling in the callback
Column handling in the callback
auto on_row = [](not_null<sqlite3_stmt*> stmt)
{
for (int i = 0 ; i < sqlite3_column_count(stmt.get()); ++i) {
// get the current field value .
}
return true ;
}
Column handling in the callback
auto on_row = [](not_null<sqlite3_stmt*> stmt)
{
for (int i = 0 ; i < sqlite3_column_count(stmt.get()); ++i) {
// get the current field value ..
}
return true ;
}
for (auto column : columns(stmt)) …
Column handling in the callback
auto on_row = [](not_null<sqlite3_stmt*> stmt)
{
for (int i = 0 ; i < sqlite3_column_count(stmt.get()); ++i) {
// get the current field value ...
}
return true ;
}
for (auto column : columns(stmt)) …
Create initial data, return the insert statement
statement create_things2(not_null<sqlite3*> db) {
Transaction transaction(db) ;
execute(db, R"~(CREATE TABLE things
(id INTEGER PRIMARY KEY, name TEXT,value REAL); )~");
auto insert_thing = create_statement(db,
"INSERT INTO things VALUES(@id,@name,@value);");
// create the additive identity thing
parameter(insert_thing.get(), 1, int64_t{0}) ;
parameter(insert_thing.get(), 2, "") ;
parameter(insert_thing.get(), 3, double{0.0}) ;
run (insert_thing.get()) ;
transaction.commit() ;
// return creator
return insert_thing ;
}
void somehwere()
{
auto db = open_database(":memory:");
auto add_thing = create_things2(db.get());
// add a second thing
parameter(add_thing.get(), 1, int64_t{1}) ;
parameter(add_thing.get(), 2, "first") ;
parameter(add_thing.get(), 3, "second") ; // Mistake !!
run(add_thing.get());
}
But we just wanted to help and deduce the type for setting a parameter …
I know what a thing is, it is int/text/double!
Printing is easy!
bool print_thing(not_null<sqlite3_stmt*> stmt) {
// get the int64
auto id = [&](){return sqlite3_column_int64(stmt, 0);} ;
//get the text
auto name = [&](){ auto first = sqlite3_column_text (stmt, 1);
std::size_t s = sqlite3_column_bytes (stmt, 1);
return s > 0 ? std::string ((const char*)first, s)
: std::string{};
};
// get the double
auto value = [&]() {return sqlite3_column_double(stmt, 2);};
// and process the data
std::cout << id() << ", " << name() << ", " << value() << std::endl;
return true ;
}
// parameter(insert_thing.get(), 1, int64_t{0}) ;
// parameter(insert_thing.get(), 2, "") ;
// parameter(insert_thing.get(), 3, double{0.0}) ;
// parameter(add_thing.get(), 1, int64_t{1}) ;
// parameter(add_thing.get(), 2, "first") ;
// parameter(add_thing.get(), 3, "second") ; // Mistake !!
auto stmt = create_statement(db.get(), "SELECT * FROM things;");
run(stmt.get(), dump_current_row);
run (stmt.get(), print_thing);
Generic print row gets it right, it cares about types
0|''|0|
1|'first'|'second'| # dump_current_row know the type
print_thing, where we have been sure what to expect
0, , 0
1, first, 0 # this might become expensive
We work with wrong data, and no one told us!
print_thing, where we have been sure what to expect
0, , 0
1, first, 0 # this might become expensive
If you do not care about your types,
someone else will do it for you.
... but maybe not in the way you want!
auto on_row = [](not_null<sqlite3_stmt*> stmt)
{
for (int i = 0 ; i < sqlite3_column_count(stmt.get()); ++i) {
// get the current field value ...
}
return true ;
}
for (auto column : columns(stmt)) …
#include <iostream>
#include <sl3/database.hpp>
int main()
{
using namespace sl3;
Database db(":memory:");
db.execute("CREATE TABLE tbl(f1 INTEGER, f2 TEXT, f3 REAL);");
// insert with type insurance
auto cmd = db.prepare("INSERT INTO tbl(f1, f2, f3) VALUES(?,?,?);",
DbValues ({Type::Int,Type::Text, Type::Real})) ;
//this will work,
cmd.execute({ {1}, {"one"}, {1.1} } );
try{
// this will throw since "2" is a wrong type
cmd.execute({ {"2"}, {"two"}, {2.1} } );
} catch (const Error& e) {
std::cout << e << std::endl;
}
}
Provide a secure (*), convenient interface
// ommit types in prepare, do what evet
//db.prepare("INSERT INTO tbl(f1, f2, f3) VALUES(?,?,?);") ;
for(auto& row :db.select("SELECT * FROM tbl;"); {
for (auto& field : row) {
std::cout << typeName (field.getType()) << "/"
<< typeName (field.getStorageType())
<< ": " << field << ", ";
}
std::cout << std::endl;
}
Variant/Int: 1, Variant/Text: one, Variant/Real: 1.1,
Variant/Text: 2, Variant/Text: two, Variant/Real: 2.1,
* no undefined behaviour/value
request the wrong value type will throw
and have a secure, convenient interface
try {
// SELECT will throw if any types is not as wanted
Dataset ds = db.select ("SELECT * FROM tbl;",
{ Type::Int, Type::Text, Type::Real });
for(auto& row : ds) {
// here I can trust the type, (could be NULL :-)
row.at(0).getInt();
row.at(1).getText();
row.at(2).getReal();
}
}
catch (const Error& e)
{
std::cout << e << std::endl ;
}
100% C++ wrapper
no direct include of sqlite header
change MB of include again some lines
access to all sqlite features possible
all sqlite data structures accessable, nothing is hidden
usable, but use primarily as inspiration
It’s under ongoing work, C++98/11/14/17 …
interface might change (if no one ask me to not do it)
If parts of this presentation looked interesting but have been handled to short/fast…
If parts of this presentation looked interesting but have been handled to short/fast…
Request an in depth tutorial or workshop
→ ask you local C++ community!
If parts of this presentation looked interesting but have been handled to short/fast…
Request an in depth tutorial or workshop
→ ask you local C++ community!
www.swedencpp.se :-)