sql for the iot - percona...about these notes nothing in these notes is required –to write...

99
SQL for the IoT

Upload: others

Post on 26-Jun-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SQL for the IoT

Page 2: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Move the query to the data,not the data to the query.

Page 3: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Internals

Page 4: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Why Do You Care?

●So you can better understand what your query is

doing.

●So that you can appreciate how much work the

database is saving you.

●So that you can fix things when they go wrong.

●So that you can write better SQL that runs faster

and uses less memory and disk space.

Page 5: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

About These Notes

●Nothing in these notes is required

–to write applications that use SQL and/or SQLite

–to query an SQL and/or SQLite database

–to maintain or enhance SQL and/or SQLite-based software

●Everything in these notes is required in order to

be an SQL and/or SQLite guru.

Page 6: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Key Concept

●SQL is a peculiar programming language

–Each SQL statement is a separate program

–SQL describes what instead of how

●An RDBMS consists of...

–Compiler to translate SQL into procedures

–Virtual Machine to evaluate the procedures

Page 7: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Example

SELECT * FROM table1;

Translates into:

Open database file containing table1Rewind the filewhile not at end-of-fileread all columns out of current recordreturn the columns to the calleradvance file to the next record

end-whileclose the file

Page 8: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Imagine what this translates into:

SELECT eqptid, enclosureidFROM eqpt

WHERE typeid IN (SELECT typeid FROM typespecWHERE attrid=(

SELECT attrid FROM attributeWHERE name='detect_autoactuate'

)AND value=1INTERSECTSELECT typeid FROM typespecWHERE attrid=(

SELECT attrid FROM attributeWHERE name='algorithm'

)AND value IN ('sensor','wetbulb')

)

Page 9: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Or This....SELECT h.url, h.title, f.url,

(SELECT b.parentFROM moz_bookmarks b

JOIN moz_bookmarks t ON t.id = b.parent AND t.parent != ?1WHERE b.type = 1 AND b.fk = h.idORDER BY b.lastModified DESC LIMIT 1) AS parent,

(SELECT b.titleFROM moz_bookmarks b

JOIN moz_bookmarks t ON t.id = b.parent AND t.parent != ?1WHERE b.type = 1 AND b.fk = h.idORDER BY b.lastModified DESC LIMIT 1) AS bookmark,

(SELECT GROUP_CONCAT(t.title, ',')FROM moz_bookmarks b

JOIN moz_bookmarks t ON t.id = b.parent AND t.parent = ?1WHERE b.type = 1 AND b.fk = h.id) AS tags,

h.visit_count, h.typed, h.frecencyFROM moz_places_temp h

LEFT OUTER JOIN moz_favicons f ON f.id = h.favicon_idWHERE h.frecency <> 0UNION ALLSELECT h.url, h.title, f.url,

(SELECT b.parentFROM moz_bookmarks b

JOIN moz_bookmarks t ON t.id = b.parent AND t.parent != ?1WHERE b.type = 1 AND b.fk = h.idORDER BY b.lastModified DESC LIMIT 1) AS parent,

(SELECT b.titleFROM moz_bookmarks b

JOIN moz_bookmarks t ON t.id = b.parent AND t.parent != ?1WHERE b.type = 1 AND b.fk = h.idORDER BY b.lastModified DESC LIMIT 1) AS bookmark,

(SELECT GROUP_CONCAT(t.title, ',')FROM moz_bookmarks b

JOIN moz_bookmarks t ON t.id = b.parent AND t.parent = ?1WHERE b.type = 1 AND b.fk = h.id) AS tags,

h.visit_count, h.typed, h.frecencyFROM moz_places h

LEFT OUTER JOIN moz_favicons f ON f.id = h.favicon_idWHERE h.id NOT IN (SELECT id FROM moz_places_temp)

AND h.frecency <> 0ORDER BY 9 DESC

Page 10: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

The Whole Point Of SQL...

●A few lines of SQL generates the equivalent of

hundreds or thousands of lines of procedural

code.

●By adding an index, entirely new procedures are

used without recoding.

●The SQL Query Optimizer is tasked with picking

the algorithm

– so that you, the developer, don't have to

Page 11: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Ins and Outs of SQL

Compile SQLinto a program

Run theprogramSQL

Prep'edStmt

Result

Front Half Back Half

Page 12: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Ins and Outs of SQL

Compile SQLinto a program

Run theprogramSQL

Prep'edStmt

Result

sqlite3_prepare_v2()

sqlite3_step()sqlite3_stmt object

Page 13: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Ins and Outs of SQL

Compile SQLinto a program

Run theprogramSQL

Prep'edStmt

Result

sqlite3_prepare_v2()

sqlite3_step()

sqlite3_exec(){sqlite3_prepare_v2();while( sqlite3_step()!=DONE ){};sqlite3_finalize();

}

Page 14: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Architecture Of SQLite

Virtual Machine

B-Tree

Pager

OS Interface

Parser

Code Generator

Page 15: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Virtual Machine

B-Tree

Pager

OS Interface

Compile SQLinto a program

Run theprogramSQL

Prep'edStmt

Result

Parser

Code Generator

Page 16: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Virtual Machine

B-Tree

Pager

OS Interface

Compile SQLinto a program

Run theprogramSQL

Prep'edStmt

Result

Parser

Code Generator

Page 17: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Virtual Machine

B-Tree

Pager

OS Interface

Compile SQLinto a program

Run theprogramSQL

Prep'edStmt

Result

Parser

Code Generator

●“Storage Engine”

●Fast, transactional,

ordered, key/value store

Page 18: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Architecture Of SQLite

Parser

Code Generator

Virtual Machine

B-Tree

Pager

OS Interface

●Byte code interpreter

●Big switch statement

inside a for loop.

●Other engines walk a

tree of structures

●Similar to JVM or Parrot

or P-Code

Page 19: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

addr opcode p1 p2 p3 p4 p5 comment

---- ------------- ---- ---- ---- ------------- -- -------------

0 Init 0 12 0 00 Start at 12

1 OpenRead 0 2 0 3 00 root=2 iDb=0; tab

2 Explain 0 0 0 SCAN TABLE tab 00

3 Rewind 0 10 0 00

4 Column 0 0 1 00 r[1]=tab.Fruit

5 Ne 2 9 1 (BINARY) 69 if r[2]!=r[1] goto 9

6 Column 0 2 3 00 r[3]=tab.Price

7 RealAffinity 3 0 0 00

8 ResultRow 3 1 0 00 output=r[3]

9 Next 0 4 0 01

10 Close 0 0 0 00

11 Halt 0 0 0 00

12 Transaction 0 0 1 0 01

13 TableLock 0 2 0 tab 00 iDb=0 root=2 write=0

14 String8 0 2 0 Orange 00 r[2]='Orange'

15 Goto 0 1 0 00

EXPLAIN SELECT price FROM tab WHERE fruit='Orange'

Page 20: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

addr opcode p1 p2 p3 p4 p5 comment

---- ------------- ---- ---- ---- ------------- -- -------------

0 Init 0 12 0 00 Start at 12

1 OpenRead 0 2 0 3 00 root=2 iDb=0; tab

2 Explain 0 0 0 SCAN TABLE tab 00

3 Rewind 0 10 0 00

4 Column 0 0 1 00 r[1]=tab.Fruit

5 Ne 2 9 1 (BINARY) 69 if r[2]!=r[1] goto 9

6 Column 0 2 3 00 r[3]=tab.Price

7 RealAffinity 3 0 0 00

8 ResultRow 3 1 0 00 output=r[3]

9 Next 0 4 0 01

10 Close 0 0 0 00

11 Halt 0 0 0 00

12 Transaction 0 0 1 0 01

13 TableLock 0 2 0 tab 00 iDb=0 root=2 write=0

14 Variable 1 2 0 ?1 00 r[2]=parameter(1,?1)

15 Goto 0 1 0 00

EXPLAIN SELECT price FROM tab WHERE fruit=?1

Page 21: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Architecture Of SQLite

Parser

Code Generator

Virtual Machine

B-Tree

Pager

OS Interface

●Ordered key/value pairs

with unique keys

●O(logN) insert, seek,

and delete

●O(1) next and previous

Page 22: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Architecture Of SQLite

Parser

Code Generator

Virtual Machine

B-Tree

Pager

OS Interface

●Atomic commit and

rollback

●Uniform size pages

numbered from 1

●No interpretation of

page content

●Cache

Page 23: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Architecture Of SQLite

Parser

Code Generator

Virtual Machine

B-Tree

Pager

OS Interface

●Platform-specific

interface to the OS

●Run-time changeable

●Portability layer

Page 24: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Logical View of SQL Table Storage

64bit integer key“rowid”

Arbitrary length data in“record” format

Page 25: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Variable Length Integers

1xxxxxxx - high bit set. 7 bits of data

0xxxxxxx - high bit clear. 7 bits of data

xxxxxxxx - 8 bits of data

128 to 16383

2097152 to 268435455

34359738368 to 4398046511103

562949953421312 to 72057594037927935

0 to 127

16384 to 2097151

268435456 to 34359738367

4398046511104 to 562949953421311

Less than 0 or greater than 72057594037927935

✔ Small positive ROWIDs stored more efficiently

Page 26: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

B+tree Structure(used by SQL tables)

Root page

Leaf pages

Integer key

Pointer to lower page

Binary content

Page 27: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

B+tree Structure(used by SQL tables)

Non-leaf pages hold only keys

● Key + Data in leaves● Combined key+data is a “cell”● As few as one “cell” per page.

Between 50 and 8000keys/page dependingon page size.

Integer key

Pointer to lower page

Binary content

Some keys appear more thanonce in the tree.

Page 28: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Mapping B-trees Into Pages

Page 2

Page 3

Page 4

Page 1

Page 5

Page 6

Page 7

Page 29: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

To see how pages are used:

●make showdb

●./showdb database.db pgidx

Page 30: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Available pages: 1..1146

1: root leaf of table [sqlite_master]

2: root interior node of table [blob]

3: root interior node of index [sqlite_autoindex_blob_1]

4: root interior node of table [delta]

5: root interior node of table [rcvfrom]

6: root leaf of index [sqlite_autoindex_rcvfrom_1]

7: root leaf of table [config]

8: root leaf of index [sqlite_autoindex_config_1]

9: root leaf of table [shun]

10: root leaf of index [sqlite_autoindex_shun_1]

11: root leaf of table [private]

...264: leaf of table [blob], child 201 of page 2

265: leaf of table [blob], child 202 of page 2

266: overflow 1 from cell 0 of page 268

267: overflow 2 from cell 0 of page 268

268: leaf of table [blob], child 203 of page 2

...

Page 31: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Overflow

Integer key

Pointer to another pageBinary content

Page 32: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Implications Of Overflow

●Move small and

common fields near the

beginning.

●Put larger and

infrequently accessed

fields at the end.

●Move large BLOBs to a

separate table and

access via join.

Page 33: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Logical View of SQL Index Storage

Arbitrary length key in“record” format

Zero data

Page 34: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

B-tree Structure(used by indexes & WITHOUT ROWID tables)

● Key only. No data. The key is the data.● Larger binary keys, hence lower fan-out● Each key appears in the table only once Binary key

Pointer to lower page

Page 35: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

sqlite_master

CREATE TABLE sqlite_master(

type text,

name text,

tbl_name text,

rootpage integer,

sql text

);

✔ sqlite_master always rooted at page 1

Page 36: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

sqlite_master

sqlite> CREATE TABLE t1(x);

sqlite> .mode line

sqlite> SELECT * FROM sqlite_master;

type = table

name = t1

tbl_name = t1

rootpage = 2

sql = CREATE TABLE t1(x)

Page 37: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Virtual Machine

●Byte code interpreter

●Defines the “record

format”

Content of tables Key of indices

Record Format

Parser

Code Generator

Virtual Machine

B-Tree

Pager

OS Interface

Page 38: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Record Format

headersize

type1

type2

...type

Ndata

1data

2...

dataN

Variable length integers

type-dependent content

Header

Data

Page 39: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Integer Type Codes

Page 40: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Record Format Example

CREATE TABLE t1(a,b,c);INSERT INTO t1 VALUES(177, NULL, 'hello');

Page 41: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

mac02:bld drh$ sqlite3 test.db

SQLite version 3.8.5 2014-06-24 20:19:21

Enter ".help" for usage hints.

sqlite> CREATE TABLE t1(a,b,c);

sqlite> INSERT INTO t1 VALUES(177, NULL, 'hello');

sqlite> .q

mac02:bld drh$ showdb test.db 2bd

Pagesize: 1024

Available pages: 1..2

Header on btree page 2:

000: 0d 13 table leaf

001: 00 00 0 Offset to first freeblock

003: 00 01 1 Number of cells on this page

005: 03 f3 1011 Offset to cell content area

007: 00 0 Fragmented byte count

Cell[0]:

3f3: 0b payload-size: 11

3f4: 01 rowid: 1

3f5: 04 record-header-size: 4

3f6: 02 typecode[0]: 2 - int16

3f7: 00 typecode[1]: 0 - NULL

3f8: 17 typecode[2]: 23 - text(5)

3f9: 00 b1 data[0]: 177

3fb: 68 65 6c 6c 6f data[2]: 'hello'

mac02:bld drh$

Page 42: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL

);

Key Data

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

Page 43: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT price FROM tab WHERE fruit='Peach'

Page 44: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT price FROM tab WHERE rowid=4

Page 45: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

CREATE INDEX idx1 ON tab(fruit)

1:1

Key DataKey

Page 46: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT price FROM tab WHERE fruit='Peach'

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

Page 47: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

SELECT price FROM tabWHERE fruit='Orange'

Page 48: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA'

Page 49: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE INDEX idx2 ON tab(state)

idx2 tab

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

1

2

4

5

18

19

23

state

FL

NC

SC

CA

FL

NC

CA

Key DataKey

Page 50: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

1

2

4

5

18

19

23

state

FL

NC

SC

CA

FL

NC

CA

idx2 tab

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA'

Page 51: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

idx3 tab

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

stateNC

SC

CAFL

NC

CREATE INDEX idx3 ON tab(fruit, state)

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

Key DataKey

Page 52: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

stateNC

SC

CAFL

NC

CREATE INDEX idx3 ON tab(fruit, state)

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

Page 53: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

state

FL

NC

SC

CAFL

NC

CA

CREATE INDEX idx3 ON tab(fruit, state)

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

Page 54: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA'

state

FL

NC

SC

CAFL

NC

CA

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

Page 55: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

idx5 tab

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

CREATE INDEX idx5 ON tab(fruit, state, price)

price

0.85

0.45

0.60

0.801.25

2.45

1.05

Key DataKey

Page 56: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA'

Page 57: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA' 1

Page 58: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA' 2

Page 59: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

1

2

4

5

18

19

23

state

FL

NC

SC

CA

FL

NC

CA

idx2 tab

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA' 3

Page 60: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA'

state

FL

NC

SC

CAFL

NC

CA

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

4

Page 61: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

SELECT price FROM tabWHERE fruit='Orange'

AND state='CA' 5

Page 62: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SELECT price FROM tabWHERE fruit='Orange'

OR state='CA'

Page 63: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT price FROM tabWHERE fruit='Orange'

OR state='CA'

Page 64: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

SELECT price FROM tabWHERE fruit='Orange'

OR state='CA'

rowid

1

2

4

5

18

19

23

state

FL

NC

SC

CA

FL

NC

CA

UN

ION

6

Page 65: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

idx4 tab

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

state

NC

SC

CA

FL

NC

CREATE INDEX idx4 ON tab(state,fruit)

rowid

fruit

2

4

5

18

19Apple

Peach

Grape

Lemon

Strawberry

FLCA

123

OrangeOrange

Key DataKey

Page 66: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

idx4 tab

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

state

NC

SC

CA

FL

NC

rowid

fruit

2

4

5

18

19Apple

Peach

Grape

Lemon

Strawberry

FLCA

123

OrangeOrange

SELECT price FROM tabWHERE fruit='Orange' 7

Page 67: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT * FROM tab ORDER BY fruitsorte

r

Page 68: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT * FROM tab ORDER BY rowid

Page 69: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SELECT * FROM tab ORDER BY rowid DESC

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

Page 70: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SELECT * FROM tab ORDER BY fruit

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

Page 71: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SELECT * FROM tab ORDER BY fruit LIMIT 2

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

Page 72: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

SELECT * FROM tab ORDER BY fruit LIMIT 2sorte

r

Page 73: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SELECT * FROM tab ORDER BY fruit DESC

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

Page 74: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

SELECT * FROM tab ORDER BY fruit

Page 75: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SELECT * FROM tab ORDER BY fruit, state

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

Page 76: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SELECT * FROM tab ORDER BY fruit DESC, state DESC

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

Page 77: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

SELECT * FROM tab ORDER BY fruit, state DESC

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

Page 78: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

SELECT * FROM tabWHERE fruit='Orange'ORDER BY state

Page 79: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL

);

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

Page 80: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE TABLE tab(Rowid INTEGER PRIMARY KEY,Fruit TEXT,State TEXT,Price REAL

);

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

Page 81: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL,PRIMARY KEY(Fruit, State)

);

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

Key DataKey

Page 82: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL,PRIMARY KEY(Fruit, State)

);

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL,UNIQUE(Fruit, State)

);

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL

);CREATE UNIQUE INDEX idx ON tab(Fruit,State);

All mean the samething (to SQLite)

Page 83: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL,PRIMARY KEY(Fruit, State)

) WITHOUT ROWID;

stateNC

SC

CAFL

NC

fruitApple

Peach

GrapeLemon

Strawberry

FLCA

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

Key Data

Page 84: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL,PRIMARY KEY(Fruit, State)

);

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

stateNC

SC

CAFL

NC

rowid

fruit2

4

518

19

Apple

Peach

GrapeLemon

Strawberry

FLCA

123

OrangeOrange

stateNC

SC

CAFL

NC

fruitApple

Peach

GrapeLemon

Strawberry

FLCA

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

With Rowid:

Without Rowid:

Page 85: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL,Amt INT,PRIMARY KEY(Fruit, State)

) WITHOUT ROWID;

CREATE INDEX tab_amt ON tab(amt);

state

NC

SCCA

FL

NC

fruit

Apple

PeachGrape

Lemon

Strawberry

FL

CA

Orange

Orange

amt

375

1000

138200

750

980825

stateNC

SC

CAFL

NC

fruitApple

Peach

GrapeLemon

Strawberry

FLCA

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

amt

375

1000

138

200750

980

825

Key DataKey

Page 86: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

state

NC

SCCA

FL

NC

fruit

Apple

PeachGrape

Lemon

Strawberry

FL

CA

Orange

Orange

amt

375

1000

138200

750

980825

stateNC

SC

CAFL

NC

fruitApple

Peach

GrapeLemon

Strawberry

FLCA

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

amt

375

1000

138

200750

980

825

CREATE TABLE tab(Price REAL,State TEXT,Fruit TEXT,Amt INT,PRIMARY KEY(Fruit, State)

) WITHOUT ROWID;

Same As

CREATE TABLE tab(Fruit TEXT,State TEXT,Price REAL,Amt INT,PRIMARY KEY(Fruit, State)

) WITHOUT ROWID;

CREATE INDEX tab_amt ON tab(amt);

Key DataKey

Page 87: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

state

NC

SCCA

FL

NC

fruit

Apple

PeachGrape

Lemon

Strawberry

FL

CA

Orange

Orange

amt

375

1000

138200

750

980825

stateNC

SC

CAFL

NC

fruitApple

Peach

GrapeLemon

Strawberry

FLCA

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

amt

375

1000

138

200750

980

825

SELECT price FROM tab WHERE fruit='Peach';

Page 88: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

rowid

fruit state price1245181923

OrangeApplePeachGrapeLemon

Strawberry

Orange

FLNCSCCAFLNCCA

0.850.450.600.801.252.451.05

rowid

fruit

1

2

4

518

19

23Orange

Apple

Peach

GrapeLemon

Strawberry

Orange

SELECT price FROM tab WHERE fruit='Peach';

state

NC

SCCA

FL

NC

fruit

Apple

PeachGrape

Lemon

Strawberry

FL

CA

Orange

Orange

amt

375

1000

138200

750

980825

stateNC

SC

CAFL

NC

fruitApple

Peach

GrapeLemon

Strawberry

FLCA

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

amt

375

1000

138

200750

980

825

Page 89: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

state

NC

SCCA

FL

NC

fruit

Apple

PeachGrape

Lemon

Strawberry

FL

CA

Orange

Orange

amt

375

1000

138200

750

980825

stateNC

SC

CAFL

NC

fruitApple

Peach

GrapeLemon

Strawberry

FLCA

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

amt

375

1000

138

200750

980

825

SELECT fruit, state FROM tab WHERE amt<250;

Page 90: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

state

NC

SCCA

FL

NC

fruit

Apple

PeachGrape

Lemon

Strawberry

FL

CA

Orange

Orange

amt

375

1000

138200

750

980825

stateNC

SC

CAFL

NC

fruitApple

Peach

GrapeLemon

Strawberry

FLCA

OrangeOrange

price

0.85

0.45

0.60

0.801.25

2.45

1.05

amt

375

1000

138

200750

980

825

SELECT fruit, state, price FROM tab WHERE amt<250;

Page 91: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

When To Use WITHOUT ROWID

●Non-INTEGER and/or composite PRIMARY

KEYs

●No need for AUTOINCREMENT, Direct BLOB I/O,

or the update hook

●Average row size less than 1/20th of page size

–Check this using sqlite3_analyzer.exe

●Run experiments to see if it helps

Page 92: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Index Column Order

SELECT x, y, z FROM t1 WHERE w=5 AND x=6 ORDER BY x, y;

CREATE INDEX t1i1 ON t1(w,x,y,z);

Page 93: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Index Column Order

SELECT x, y, z FROM t1 WHERE w=5 AND x=6 ORDER BY x, y;

CREATE INDEX t1i1 ON t1(w,x,y,z);

WHERE clause equality terms first

Page 94: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Index Column Order

SELECT x, y, z FROM t1 WHERE w=5 AND x=6 ORDER BY x, y;

CREATE INDEX t1i1 ON t1(w,x,y,z);

ORDER BY terms second.No gaps!Can overlap WHERE clause.

Page 95: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Index Column Order

SELECT x, y, z FROM t1 WHERE w=5 AND x=6 ORDER BY x, y;

CREATE INDEX t1i1 ON t1(w,x,y,z);

Result set columns can appearanywhere and in any order.

Page 96: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Index Column Order

SELECT * FROM t1 WHERE w=5 AND x>=6 AND y=23;

CREATE INDEX t1i1 ON t1(w,x,y,z);

Equality terms first Range constraints at the

end

Terms past the range constraint arenot usable for search.

Page 97: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Avoid Redundant Indices

CREATE TABLE t2(x,y,z);

CREATE INDEX t2i1 ON t2(x,y,z);

CREATE INDEX t2i2 ON t2(x,y);

CREATE INDEX t2i3 ON t2(x);

Page 98: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Avoiding & Fixing Planner Problems

●Create appropriate indexes

●Avoid low-quality indexes

●Run ANALYZE

●Instrument your code

●Use unlikely() and likelihood() hints

●Use CROSS JOIN to force loop nesting order

●Disable WHERE terms using unary “+”

●INDEXED BY to force a particular index

Page 99: SQL for the IoT - Percona...About These Notes Nothing in these notes is required –to write applications that use SQL and/or SQLite –to query an SQL and/or SQLite database –to

Query Plan Stability Guarantee

SQLite will always generate the same query

plan for the same SQL text provided:

1) The schema is not changed

2) You do not rerun ANALYZE

3) Omit SQLITE_ENABLE_STAT[34]

4) Keep the same version of SQLite