admin/SQLite-fork
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Adding the sqlite sourcecode

Browse Source
This commit is contained in:
Jiří Stach committed 2025-07-17 11:54:48 +02:00
1 parent bdb54a1f8f
commit 406b584760
2208 files changed
+1172785 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/F2FS.txt
+87
View File
@@ -0,0 +1,87 @@
SQLite's OS layer contains the following definitions used in F2FS related
calls:
#define F2FS_IOCTL_MAGIC 0xf5
#define F2FS_IOC_START_ATOMIC_WRITE _IO(F2FS_IOCTL_MAGIC, 1)
#define F2FS_IOC_COMMIT_ATOMIC_WRITE _IO(F2FS_IOCTL_MAGIC, 2)
#define F2FS_IOC_START_VOLATILE_WRITE _IO(F2FS_IOCTL_MAGIC, 3)
#define F2FS_IOC_ABORT_VOLATILE_WRITE _IO(F2FS_IOCTL_MAGIC, 5)
#define F2FS_IOC_GET_FEATURES _IOR(F2FS_IOCTL_MAGIC, 12, u32)
#define F2FS_FEATURE_ATOMIC_WRITE 0x0004
After opening a database file on Linux (including Android), SQLite determines
whether or not a file supports F2FS atomic commits as follows:
u32 flags = 0;
rc = ioctl(fd, F2FS_IOC_GET_FEATURES, &flags);
if( rc==0 && (flags & F2FS_FEATURE_ATOMIC_WRITE) ){
/* File supports F2FS atomic commits */
}else{
/* File does NOT support F2FS atomic commits */
}
where "fd" is the file-descriptor open on the database file.
Usually, when writing to a database file that supports atomic commits, SQLite
accumulates the entire transaction in heap memory, deferring all writes to the
db file until the transaction is committed.
When it is time to commit a transaction on a file that supports atomic
commits, SQLite does:
/* Take an F_WRLCK lock on the database file. This prevents any other
** SQLite clients from reading or writing the file until the lock
** is released. */
rc = fcntl(fd, F_SETLK, ...);
if( rc!=0 ) goto failed;
rc = ioctl(fd, F2FS_IOC_START_ATOMIC_WRITE);
if( rc!=0 ) goto fallback_to_legacy_journal_commit;
foreach (dirty page){
rc = write(fd, ...dirty page...);
if( rc!=0 ){
ioctl(fd, F2FS_IOC_ABORT_VOLATILE_WRITE);
goto fallback_to_legacy_journal_commit;
}
}
rc = ioctl(fd, F2FS_IOC_COMMIT_ATOMIC_WRITE);
if( rc!=0 ){
ioctl(fd, F2FS_IOC_ABORT_VOLATILE_WRITE);
goto fallback_to_legacy_journal_commit;
}
/* If we get there, the transaction has been successfully
** committed to persistent storage. The following call
** relinquishes the F_WRLCK lock. */
fcntl(fd, F_SETLK, ...);
Assumptions:
1. After either of the F2FS_IOC_ABORT_VOLATILE_WRITE calls return,
the database file is in the state that it was in before
F2FS_IOC_START_ATOMIC_WRITE was invoked. Even if the ioctl()
fails - we're ignoring the return code.
This is true regardless of the type of error that occurred in
ioctl() or write().
2. If the system fails before the F2FS_IOC_COMMIT_ATOMIC_WRITE is
completed, then following a reboot the database file is in the
state that it was in before F2FS_IOC_START_ATOMIC_WRITE was invoked.
Or, if the write was commited right before the system failed, in a
state indicating that all write() calls were successfully committed
to persistent storage before the failure occurred.
3. If the process crashes before the F2FS_IOC_COMMIT_ATOMIC_WRITE is
completed then the file is automatically restored to the state that
it was in before F2FS_IOC_START_ATOMIC_WRITE was called. This occurs
before the posix advisory lock is automatically dropped - there is
no chance that another client will be able to read the file in a
half-committed state before the rollback operation occurs.
doc/compile-for-unix.md
+70
View File
@@ -0,0 +1,70 @@
# Notes On Compiling SQLite On All Kinds Of Unix
Here are step-by-step instructions on how to build SQLite from
canonical source on any modern machine that isn't Windows. These
notes are tested (on 2024-10-11) on Ubuntu and on MacOS, but they
are general and should work on most any modern unix platform.
See the companion document ([](./compile-for-windows.md>)) for
guidance on building for Windows.
1. Install a C-compiler. GCC or Clang both work fine. If you are
reading this document, you've probably already done that.
2. *(Optional):* Install TCL development libraries. In this note,
we'll do a private install in the $HOME/local directory,
but you can make adjustments to install TCL wherever you like.
This document assumes you are working with TCL version 9.0.
See also the [](./tcl-extension-testing.md) document that contains
more details on compiling Tcl for use with SQLite.
<ol type="a">
<li>Get the TCL source archive, perhaps from
<https://www.tcl.tk/software/tcltk/download.html>
or <https://sqlite.org/tmp/tcl9.0.0.tar.gz>.
<li>Untar the source archive. CD into the "unix/" subfolder
of the source tree.
<li>Run: `mkdir $HOME/local`
<li>Run: `./configure --prefix=$HOME/local`
<li>Run: `make install`
</ol>
<p>
As of 2024-10-25, TCL is not longer required for many
common build targets, such as "sqlite3.c" or the "sqlite3"
command-line tool. So you can skip this step if that is all
you want to build. TCL is still required to run "make test"
and similar, or to build the TCL extension, of course.
4. Download the SQLite source tree and unpack it. CD into the
toplevel directory of the source tree.
5. Run: `./configure --enable-all --with-tclsh=$HOME/local/bin/tclsh9.0`
You do not need to use --with-tclsh if the tclsh you want to use is the
first one on your PATH or if you are building without TCL.
6. Run the "`Makefile`" makefile with an appropriate target.
Examples:
<ul>
<li> `make sqlite3.c`
<li> `make sqlite3`
<li> `make sqldiff`
<li> `make sqlite3_rsync`
</ul>
<p>None of the targets above require TCL. TCL is needed
for the following targets:
<ul>
<li> `make tclextension-install`
<li> `make devtest`
<li> `make releasetest`
<li> `make sqlite3_analyzer`
</ul>
It is not required that you run the "tclextension-install" target prior to
running tests. However, the tests will run more smoothly if you do.
The version of SQLite used for the TCL extension does *not* need to
correspond to the version of SQLite under test. So you can install the
SQLite TCL extension once, and then use it to test many different versions
of SQLite.
7. For a debugging build of the CLI, where the ".treetrace" and ".wheretrace"
commands work, add the the --with-debug argument to configure.
doc/compile-for-windows.md
+191
View File
@@ -0,0 +1,191 @@
# Notes On Compiling SQLite On Windows 11
Below are step-by-step instructions on how to build SQLite from
canonical source on a new Windows 11 PC, as of 2024-10-09.
See [](./compile-for-unix.md) for a similar guide for unix-like
systems, including MacOS.
1. Install Microsoft Visual Studio. The free "community edition"
will work fine. Do a standard install for C++ development.
SQLite only needs the
"cl" compiler and the "nmake" build tool.
<ul><li><b>Note:</b>
VS2015 or later is required for the procedures below to
all work. You *might* be able to get the build to work with
earlier versions of MSVC, but in that case the TCL installation
of step 3 will be required, since the "jimsh0.c" program of
Autosetup that is used as a substitute for "tclsh.exe" won't
compile with versions of Visual Studio prior to VS2015. In any
event, building SQLite from canonical source code on Windows
is not supported for earlier versions of Visual Studio.</ul>
2. Under the "Start" menu, find "All Apps" then go to "Visual Studio 20XX"
and find "x64 Native Tools Command Prompt for VS 20XX". Pin that
application to your task bar, as you will use it a lot. Bring up
an instance of this command prompt and do all of the subsequent steps
in that "x64 Native Tools" command prompt. (Or use "x86" if you want
a 32-bit build.) The subsequent steps will not work in a vanilla
DOS prompt. Nor will they work in PowerShell.
3. *(Optional):* Install TCL development libraries.
This note assumes that you will
install the TCL development libraries in the "`c:\Tcl`" directory.
Make adjustments
if you want TCL installed somewhere else. SQLite needs both the
"tclsh90.exe" command-line tool as part of the build process, and
the "tcl90.lib" and "tclstub.lib" libraries in order to run tests.
This document assumes you are working with TCL version 9.0.
See [](./tcl-extension-testing.md#windows) for guidance on how
to compile TCL version 8.6 for use with SQLite.
<ol type="a">
<li>Get the TCL source archive, perhaps from
<https://www.tcl.tk/software/tcltk/download.html>
or <https://sqlite.org/tmp/tcl9.0.0.tar.gz>.
<li>Untar or unzip the source archive. CD into the "win/" subfolder
of the source tree.
<li>Run: `nmake /f makefile.vc INSTALLDIR=c:\Tcl release`
<li>Run: `nmake /f makefile.vc INSTALLDIR=c:\Tcl install` <br>
Notes:
<ol type="i">
<li> The previous two `nmake` commands must be run separately.
<li> Also, the INSTALLDIR=... argument is required on both.
</ol>
<li><i>Optional:</i> CD to `c:\Tcl\bin` and make a copy of
`tclsh90.exe` over into just `tclsh.exe`.
<li><i>Optional:</i>
Add `c:\Tcl\bin` to your %PATH%. To do this, go to Settings
and search for "path". Select "edit environment variables for
your account" and modify your default PATH accordingly.
You will need to close and reopen your command prompts after
making this change.
</ol>
As of 2024-10-25, TCL is not longer required for many
common build targets, such as "sqlite3.c" or the "sqlite3.exe"
command-line tool. So you can skip this step if that is all
you want to build. TCL is still required to run "make test"
and similar, or to build the TCL extension, of course.
4. Download the SQLite source tree and unpack it. CD into the
toplevel directory of the source tree.
5. Run the "`Makefile.msc`" makefile with an appropriate target.
Examples:
<ul>
<li> `nmake /f makefile.msc`
<li> `nmake /f makefile.msc sqlite3.c`
<li> `nmake /f makefile.msc sqlite3.exe`
<li> `nmake /f makefile.msc sqldiff.exe`
<li> `nmake /f makefile.msc sqlite3_rsync.exe`
</ul>
<p>No TCL is required for the nmake targets above. But for the ones
that follow, you will need a TCL installation, as described in step 3
above. If you install TCL in some directory other than C:\\Tcl, then
you will also need to add the "TCLDIR=<i>&lt;dir&gt;</i>" option on the
nmake command line to tell nmake where your TCL is installed.
<ul>
<li> `nmake /f makefile.msc tclextension-install`
<li> `nmake /f makefile.msc devtest`
<li> `nmake /f makefile.msc releasetest`
<li> `nmake /f makefile.msc sqlite3_analyzer.exe`
</ul>
It is not required that you run the "tclextension-install" target prior to
running tests. However, the tests will run more smoothly if you do.
The version of SQLite used for the TCL extension does *not* need to
correspond to the version of SQLite under test. So you can install the
SQLite TCL extension once, and then use it to test many different versions
of SQLite.
7. For a debugging build of the CLI, where the ".treetrace" and ".wheretrace"
commands work, add the DEBUG=3 argument to nmake. Like this:
<ul>
<li> `nmake /f makefile.msc DEBUG=3 clean sqlite3.exe`
</ul>
## 32-bit Builds
Doing a 32-bit build is just like doing a 64-bit build with the
following minor changes:
1. Use the "x86 Native Tools Command Prompt" instead of
"x64 Native Tools Command Prompt". "**x86**" instead of "**x64**".
2. Use a different installation directory for TCL.
The recommended directory is `c:\tcl32`. Thus you end up
with two TCL builds:
<ul>
<li> `c:\tcl` &larr; 64-bit (the default)
<li> `c:\tcl32` &larr; 32-bit
</ul>
3. Ensure that `c:\tcl32\bin` comes before `c:\tcl\bin` on
your PATH environment variable. You can achieve this using
a command like:
<ul>
<li> `set PATH=c:\tcl32\bin;%PATH%`
</ul>
## Building a DLL
The command the developers use for building the deliverable DLL on the
[download page](https://sqlite.org/download.html) is as follows:
> ~~~~
nmake /f Makefile.msc sqlite3.dll USE_NATIVE_LIBPATHS=1 "OPTS=-DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_FTS5=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1=1 -DSQLITE_ENABLE_GEOPOLY=1 -DSQLITE_ENABLE_SESSION=1 -DSQLITE_ENABLE_PREUPDATE_HOOK=1 -DSQLITE_ENABLE_SERIALIZE=1 -DSQLITE_ENABLE_MATH_FUNCTIONS=1"
~~~~
That command generates both the sqlite3.dll and sqlite3.def files. The same
command works for both 32-bit and 64-bit builds.
## Statically Linking The TCL Library
Some utility programs associated with SQLite need to be linked
with TCL in order to function. The [sqlite3_analyzer.exe program](https://sqlite.org/sqlanalyze.html)
is an example. You can build as described above, and then
enter:
> ~~~~
nmake /f Makefile.msc sqlite3_analyzer.exe
~~~~
And you will end up with a working executable. However, that executable
will depend on having the "tcl98.dll" library somewhere on your %PATH%.
Use the following steps to build an executable that has the TCL library
statically linked so that it does not depend on separate DLL:
1. Use the appropriate "Command Prompt" window - either x86 or
x64, depending on whether you want a 32-bit or 64-bit executable.
2. Untar the TCL source tarball into a fresh directory. CD into
the "win/" subfolder.
3. Run: `nmake /f makefile.vc OPTS=static shell`
4. CD into the "Release*" subfolder that is created (note the
wildcard - the full name of the directory might vary). There
you will find the "tcl90s.lib" file. Copy this file into the
same directory that you put the "tcl90.lib" on your initial
installation. (In this document, that directory is
"`C:\Tcl32\lib`" for 32-bit builds and
"`C:\Tcl\lib`" for 64-bit builds.)
5. CD into your SQLite source code directory and build the desired
utility program, but add the following extra argument to the
nmake command line:
<blockquote><pre>
STATICALLY_LINK_TCL=1
</pre></blockquote>
<p>So, for example, to build a statically linked version of
sqlite3_analyzer.exe, you might type:
<blockquote><pre>
nmake /f Makefile.msc STATICALLY_LINK_TCL=1 sqlite3_analyzer.exe
</pre></blockquote>
6. After your executable is built, you can verify that it does not
depend on the TCL DLL by running:
<blockquote><pre>
dumpbin /dependents sqlite3_analyzer.exe
</pre></blockquote>
doc/json-enhancements.md
+144
View File
@@ -0,0 +1,144 @@
# JSON Functions Enhancements (2022)
This document summaries enhancements to the SQLite JSON support added in
early 2022.
## 1.0 Change summary:
1. New **->** and **->>** operators that work like MySQL and PostgreSQL (PG).
2. JSON functions are built-in rather than being an extension. They
are included by default, but can be omitted using the
-DSQLITE_OMIT_JSON compile-time option.
## 2.0 New operators **->** and **->>**
The SQLite language adds two new binary operators **->** and **->>**.
Both operators are similar to json_extract(). The left operand is
JSON and the right operand is a JSON path expression (possibly abbreviated
for compatibility with PG - see below). So they are similar to a
two-argument call to json_extract().
The difference between -> and ->> (and json_extract()) is as follows:
* The -> operator always returns JSON.
* The ->> operator converts the answer into a primitive SQL datatype
such as TEXT, INTEGER, REAL, or NULL. If a JSON object or array
is selected, that object or array is rendered as text. If a JSON
value is selected, that value is converted into its corresponding
SQL type
* The json_extract() interface returns JSON when a JSON object or
array is selected, or a primitive SQL datatype when a JSON value
is selected. This is different from MySQL, in which json_extract()
always returns JSON, but the difference is retained because it has
worked that way for 6 years and changing it now would likely break
a lot of legacy code.
In MySQL and PG, the ->> operator always returns TEXT (or NULL) and never
INTEGER or REAL. This is due to limitations in the type handling capabilities
of those systems. In MySQL and PG, the result type a function or operator
may only depend on the type of its arguments, never the value of its arguments.
But the underlying JSON type depends on the value of the JSON path
expression, not the type of the JSON path expression (which is always TEXT).
Hence, the result type of ->> in MySQL and PG is unable to vary according
to the type of the JSON value being extracted.
The type system in SQLite is more general. Functions in SQLite are able
to return different datatypes depending on the value of their arguments.
So the ->> operator in SQLite is able to return TEXT, INTEGER, REAL, or NULL
depending on the JSON type of the value being extracted. This means that
the behavior of the ->> is slightly different in SQLite versus MySQL and PG
in that it will sometimes return INTEGER and REAL values, depending on its
inputs. It is possible to implement the ->> operator in SQLite so that it
always operates exactly like MySQL and PG and always returns TEXT or NULL,
but I have been unable to think of any situations where returning the
actual JSON value this would cause problems, so I'm including the enhanced
functionality in SQLite.
The table below attempts to summarize the differences between the
-> and ->> operators and the json_extract() function, for SQLite, MySQL,
and PG. JSON values are shown using their SQL text representation but
in a bold font.
<table border=1 cellpadding=5 cellspacing=0>
<tr><th>JSON<th>PATH<th>-&gt; operator<br>(all)<th>-&gt;&gt; operator<br>(MySQL/PG)
<th>-&gt;&gt; operator<br>(SQLite)<th>json_extract()<br>(SQLite)
<tr><td> **'{"a":123}'** <td>'$.a'<td> **'123'** <td> '123' <td> 123 <td> 123
<tr><td> **'{"a":4.5}'** <td>'$.a'<td> **'4.5'** <td> '4.5' <td> 4.5 <td> 4.5
<tr><td> **'{"a":"xyz"}'** <td>'$.a'<td> **'"xyz"'** <td> 'xyz' <td> 'xyz' <td> 'xyz'
<tr><td> **'{"a":null}'** <td>'$.a'<td> **'null'** <td> NULL <td> NULL <td> NULL
<tr><td> **'{"a":[6,7,8]}'** <td>'$.a'<td> **'[6,7,8]'** <td> '[6,7,8]' <td> '[6,7,8]' <td> **'[6,7,8]'**
<tr><td> **'{"a":{"x":9}}'** <td>'$.a'<td> **'{"x":9}'** <td> '{"x":9}' <td> '{"x":9}' <td> **'{"x":9}'**
<tr><td> **'{"b":999}'** <td>'$.a'<td> NULL <td> NULL <td> NULL <td> NULL
</table>
Important points about the table above:
* The -> operator always returns either JSON or NULL.
* The ->> operator never returns JSON. It always returns TEXT or NULL, or in the
case of SQLite, INTEGER or REAL.
* The MySQL json_extract() function works exactly the same
as the MySQL -> operator.
* The SQLite json_extract() operator works like -> for JSON objects and
arrays, and like ->> for JSON values.
* The -> operator works the same for all systems.
* The only difference in ->> between SQLite and other systems is that
when the JSON value is numeric, SQLite returns a numeric SQL value,
whereas the other systems return a text representation of the numeric
value.
### 2.1 Abbreviated JSON path expressions for PG compatibility
The table above always shows the full JSON path expression: '$.a'. But
PG does not accept this syntax. PG only allows a single JSON object label
name or a single integer array index. In order to provide compatibility
with PG, The -> and ->> operators in SQLite are extended to also support
a JSON object label or an integer array index for the right-hand side
operand, in addition to a full JSON path expression.
Thus, a -> or ->> operator that works on MySQL will work in
SQLite. And a -> or ->> operator that works in PG will work in SQLite.
But because SQLite supports the union of the disjoint capabilities of
MySQL and PG, there will always be -> and ->> operators that work in
SQLite that do not work in one of MySQL and PG. This is an unavoidable
consequence of the different syntax for -> and ->> in MySQL and PG.
In the following table, assume that "value1" is a JSON object and
"value2" is a JSON array.
<table border=1 cellpadding=5 cellspacing=0>
<tr><th>SQL expression <th>Works in MySQL?<th>Works in PG?<th>Works in SQLite
<tr><td>value1-&gt;'$.a' <td> yes <td> no <td> yes
<tr><td>value1-&gt;'a' <td> no <td> yes <td> yes
<tr><td>value2-&gt;'$[2]' <td> yes <td> no <td> yes
<tr><td>value2-&gt;2 <td> no <td> yes <td> yes
</table>
The abbreviated JSON path expressions only work for the -> and ->> operators
in SQLite. The json_extract() function, and all other built-in SQLite
JSON functions, continue to require complete JSON path expressions for their
PATH arguments.
## 3.0 JSON moved into the core
The JSON interface is now moved into the SQLite core.
When originally written in 2015, the JSON functions were an extension
that could be optionally included at compile-time, or loaded at run-time.
The implementation was in a source file named ext/misc/json1.c in the
source tree. JSON functions were only compiled in if the
-DSQLITE_ENABLE_JSON1 compile-time option was used.
After these enhancements, the JSON functions are now built-ins.
The source file that implements the JSON functions is moved to src/json.c.
No special compile-time options are needed to load JSON into the build.
Instead, there is a new -DSQLITE_OMIT_JSON compile-time option to leave
them out.
doc/jsonb.md
+290
View File
@@ -0,0 +1,290 @@
# The JSONB Format
This document describes SQLite's JSONB binary encoding of
JSON.
## 1.0 What Is JSONB?
Beginning with version 3.45.0 (circa 2024-01-01), SQLite supports an
alternative binary encoding of JSON which we call "JSONB". JSONB is
a binary format that stored as a BLOB.
The advantage of JSONB over ordinary text RFC 8259 JSON is that JSONB
is both slightly smaller (by between 5% and 10% in most cases) and
can be processed in less than half the number of CPU cycles. The built-in
[JSON SQL functions] of SQLite can accept either ordinary text JSON
or the binary JSONB encoding for any of their JSON inputs.
The "JSONB" name is inspired by [PostgreSQL](https://postgresql.org), but the
on-disk format for SQLite's JSONB is not the same as PostgreSQL's.
The two formats have the same name, but they have wildly different internal
representations and are not in any way binary compatible.
The central idea behind this JSONB specification is that each element
begins with a header that includes the size and type of that element.
The header takes the place of punctuation such as double-quotes,
curly-brackes, square-brackets, commas, and colons. Since the size
and type of each element is contained in its header, the element can
be read faster since it is no longer necessary to carefully scan forward
looking for the closing delimiter. The payload of JSONB is the same
as for corresponding text JSON. The same payload bytes occur in the
same order. The only real difference between JSONB and ordinary text
JSON is that JSONB includes a binary header on
each element and omits delimiter and separator punctuation.
### 1.1 Internal Use Only
The details of the JSONB are not intended to be visible to application
developers. Application developers should look at JSONB as an opaque BLOB
used internally by SQLite. Nevertheless, we want the format to be backwards
compatible across all future versions of SQLite. To that end, the format
is documented by this file in the source tree. But this file should be
used only by SQLite core developers, not by developers of applications
that only use SQLite.
## 2.0 The Purpose Of This Document
JSONB is not intended as an external format to be used by
applications. JSONB is designed for internal use by SQLite only.
Programmers do not need to understand the JSONB format in order to
use it effectively.
Applications should access JSONB only through the [JSON SQL functions],
not by looking at individual bytes of the BLOB.
However, JSONB is intended to be portable and backwards compatible
for all future versions of SQLite. In other words, you should not have
to export and reimport your SQLite database files when you upgrade to
a newer SQLite version. For that reason, the JSONB format needs to
be well-defined.
This document is therefore similar in purpose to the
[SQLite database file format] document that describes the on-disk
format of an SQLite database file. Applications are not expected
to directly read and write the bits and bytes of SQLite database files.
The SQLite database file format is carefully documented so that it
can be stable and enduring. In the same way, the JSONB representation
of JSON is documented here so that it too can be stable and enduring,
not so that applications can read or writes individual bytes.
## 3.0 Encoding
JSONB is a direct translation of the underlying text JSON. The difference
is that JSONB uses a binary encoding that is faster to parse compared to
the detailed syntax of text JSON.
Each JSON element is encoded as a header and a payload. The header
determines type of element (string, numeric, boolean, null, object, or
array) and the size of the payload. The header can be between 1 and
9 bytes in size. The payload can be any size from zero bytes up to the
maximum allowed BLOB size.
### 3.1 Payload Size
The upper four bits of the first byte of the header determine size of the
header and possibly also the size of the payload.
If the upper four bits have a value between 0 and 11, then the header is
exactly one byte in size and the payload size is determined by those
upper four bits. If the upper four bits have a value between 12 and 15,
that means that the total header size is 2, 3, 5, or 9 bytes and the
payload size is unsigned big-endian integer that is contained in the
subsequent bytes. The size integer is the one byte that following the
initial header byte if the upper four bits
are 12, two bytes if the upper bits are 13, four bytes if the upper bits
are 14, and eight bytes if the upper bits are 15. The current design
of SQLite does not support BLOB values larger than 2GiB, so the eight-byte
variant of the payload size integer will never be used by the current code.
The eight-byte payload size integer is included in the specification
to allow for future expansion.
The header for an element does *not* need to be in its simplest
form. For example, consider the JSON numeric value "`1`".
That element can be encode in five different ways:
* `0x13 0x31`
* `0xc3 0x01 0x31`
* `0xd3 0x00 0x01 0x31`
* `0xe3 0x00 0x00 0x00 0x01 0x31`
* `0xf3 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x31`
The shortest encoding is preferred, of course, and usually happens with
primitive elements such as numbers. However the total size of an array
or object might not be known exactly when the header of the element is
first generated. It is convenient to reserve space for the largest
possible header and then go back and fill in the correct payload size
at the end. This technique can result in array or object headers that
are larger than absolutely necessary.
### 3.2 Element Type
The least-significant four bits of the first byte of the header (the first
byte masked against 0x0f) determine element type. The following codes are
used:
<ol>
<li type="0"><p><b>NULL</b> &rarr;
The element is a JSON "null". The payload size for a true JSON NULL must
must be zero. Future versions of SQLite might extend the JSONB format
with elements that have a zero element type but a non-zero size. In that
way, legacy versions of SQLite will interpret the element as a NULL
for backwards compatibility while newer versions will interpret the
element in some other way.
<li value="1"><p><b>TRUE</b> &rarr;
The element is a JSON "true". The payload size must be zero for a actual
"true" value. Elements with type 1 and a non-zero payload size are
reserved for future expansion. Legacy implementations that see an element
type of 1 with a non-zero payload size should continue to interpret that
element as "true" for compatibility.
<li value="2"><p><b>FALSE</b> &rarr;
The element is a JSON "false". The payload size must be zero for a actual
"false" value. Elements with type 2 and a non-zero payload size are
reserved for future expansion. Legacy implementations that see an element
type of 2 with a non-zero payload size should continue to interpret that
element as "false" for compatibility.
<li value="3"><p><b>INT</b> &rarr;
The element is a JSON integer value in the canonical
RFC 8259 format, without extensions. The payload is the ASCII
text representation of that numeric value.
<li value="4"><p><b>INT5</b> &rarr;
The element is a JSON integer value that is not in the
canonical format. The payload is the ASCII
text representation of that numeric value. Because the payload is in a
non-standard format, it will need to be translated when the JSONB is
converted into RFC 8259 text JSON.
<li value="5"><p><b>FLOAT</b> &rarr;
The element is a JSON floating-point value in the canonical
RFC 8259 format, without extensions. The payload is the ASCII
text representation of that numeric value.
<li value="6"><p><b>FLOAT5</b> &rarr;
The element is a JSON floating-point value that is not in the
canonical format. The payload is the ASCII
text representation of that numeric value. Because the payload is in a
non-standard format, it will need to be translated when the JSONB is
converted into RFC 8259 text JSON.
<li value="7"><p><b>TEXT</b> &rarr;
The element is a JSON string value that does not contain
any escapes nor any characters that need to be escaped for either SQL or
JSON. The payload is the UTF8 text representation of the string value.
The payload does <i>not</i> include string delimiters.
<li value="8"><p><b>TEXTJ</b> &rarr;
The element is a JSON string value that contains
RFC 8259 character escapes (such as "<tt>\n</tt>" or "<tt>\u0020</tt>").
Those escapes will need to be translated into actual UTF8 if this element
is [json_extract|extracted] into SQL.
The payload is the UTF8 text representation of the escaped string value.
The payload does <i>not</i> include string delimiters.
<li value="9"><p><b>TEXT5</b> &rarr;
The element is a JSON string value that contains
character escapes, including some character escapes that part of JSON5
and which are not found in the canonical RFC 8259 spec.
Those escapes will need to be translated into standard JSON prior to
rendering the JSON as text, or into their actual UTF8 characters if this
element is [json_extract|extracted] into SQL.
The payload is the UTF8 text representation of the escaped string value.
The payload does <i>not</i> include string delimiters.
<li value="10"><p><b>TEXTRAW</b> &rarr;
The element is a JSON string value that contains
UTF8 characters that need to be escaped if this string is rendered into
standard JSON text.
The payload does <i>not</i> include string delimiters.
<li value="11"><p><b>ARRAY</b> &rarr;
The element is a JSON array. The payload contains
JSONB elements that comprise values contained within the array.
<li value="12"><p><b>OBJECT</b> &rarr;
The element is a JSON object. The payload contains
pairs of JSONB elements that comprise entries for the JSON object.
The first element in each pair must be a string (types 7 through 10).
The second element of each pair may be any types, including nested
arrays or objects.
<li value="13"><p><b>RESERVED-13</b> &rarr;
Reserved for future expansion. Legacy implements that encounter this
element type should raise an error.
<li value="14"><p><b>RESERVED-14</b> &rarr;
Reserved for future expansion. Legacy implements that encounter this
element type should raise an error.
<li value="15"><p><b>RESERVED-15</b> &rarr;
Reserved for future expansion. Legacy implements that encounter this
element type should raise an error.
</ol>
Element types outside the range of 0 to 12 are reserved for future
expansion. The current implement raises an error if see an element type
other than those listed above. However, future versions of SQLite might
use of the three remaining element types to implement indexing or similar
optimizations, to speed up lookup against large JSON arrays and/or objects.
### 3.3 Design Rationale For Element Types
A key goal of JSONB is that it should be quick to translate
to and from text JSON and/or be constructed from SQL values.
When converting from text into JSONB, we do not want the
converter subroutine to burn CPU cycles converting elements
values into some standard format which might never be used.
Format conversion is "lazy" - it is deferred until actually
needed. This has implications for the JSONB format design:
1. Numeric values are stored as text, not a numbers. The values are
a direct copy of the text JSON values from which they are derived.
2. There are multiple element types depending on the details of value
formats. For example, INT is used for pure RFC-8259 integer
literals and INT5 exists for JSON5 extensions such as hexadecimal
notation. FLOAT is used for pure RFC-8259 floating point literals
and FLOAT5 is used for JSON5 extensions. There are four different
representations of strings, depending on where the string came from
and how special characters within the string are escaped.
A second goal of JSONB is that it should be capable of serving as the
"parse tree" for JSON when a JSON value is being processed by the
various [JSON SQL functions] built into SQLite. Before JSONB was
developed, operations such [json_replace()] and [json_patch()]
and similar worked in three stages:
1. Translate the text JSON into a internal format that is
easier to scan and edit.
2. Perform the requested operation on the JSON.
3. Translate the internal format back into text.
JSONB seeks to serve as the internal format directly - bypassing
the first and third stages of that process. Since most of the CPU
cycles are spent on the first and third stages, that suggests that
JSONB processing will be much faster than text JSON processing.
So when processing JSONB, only the second stage of the three-stage
process is required. But when processing text JSON, it is still necessary
to do stages one and three. If JSONB is to be used as the internal
binary representation, this is yet another reason to store numeric
values as text. Storing numbers as text minimizes the amount of
conversion work needed for stages one and three. This is also why
there are four different representations of text in JSONB. Different
text representations are used for text coming from different sources
(RFC-8259 JSON, JSON5, or SQL string values) and conversions only
happen if and when they are actually needed.
### 3.4 Valid JSONB BLOBs
A valid JSONB BLOB consists of a single JSON element. The element must
exactly fill the BLOB. This one element is often a JSON object or array
and those usually contain additional elements as its payload, but the
element can be a primitive value such a string, number, boolean, or null.
When the built-in JSON functions are attempting to determine if a BLOB
argument is a JSONB or just a random BLOB, they look at the header of
the outer element to see that it is well-formed and that the element
completely fills the BLOB. If these conditions are met, then the BLOB
is accepted as a JSONB value.
doc/lemon.html
+1282
View File
File diff suppressed because it is too large. Load diff
doc/pager-invariants.txt
+76
View File
@@ -0,0 +1,76 @@
*** Throughout this document, a page is deemed to have been synced
automatically as soon as it is written when PRAGMA synchronous=OFF.
Otherwise, the page is not synced until the xSync method of the VFS
is called successfully on the file containing the page.
*** Definition: A page of the database file is said to be "overwriteable" if
one or more of the following are true about the page:
(a) The original content of the page as it was at the beginning of
the transaction has been written into the rollback journal and
synced.
(b) The page was a freelist leaf page at the start of the transaction.
(c) The page number is greater than the largest page that existed in
the database file at the start of the transaction.
(1) A page of the database file is never overwritten unless one of the
following are true:
(a) The page and all other pages on the same sector are overwriteable.
(b) The atomic page write optimization is enabled, and the entire
transaction other than the update of the transaction sequence
number consists of a single page change.
(2) The content of a page written into the rollback journal exactly matches
both the content in the database when the rollback journal was written
and the content in the database at the beginning of the current
transaction.
(3) Writes to the database file are an integer multiple of the page size
in length and are aligned to a page boundary.
(4) Reads from the database file are either aligned on a page boundary and
an integer multiple of the page size in length or are taken from the
first 100 bytes of the database file.
(5) All writes to the database file are synced prior to the rollback journal
being deleted, truncated, or zeroed.
(6) If a master journal file is used, then all writes to the database file
are synced prior to the master journal being deleted.
*** Definition: Two databases (or the same database at two points it time)
are said to be "logically equivalent" if they give the same answer to
all queries. Note in particular the content of freelist leaf
pages can be changed arbitrarily without effecting the logical equivalence
of the database.
(7) At any time, if any subset, including the empty set and the total set,
of the unsynced changes to a rollback journal are removed and the
journal is rolled back, the resulting database file will be logical
equivalent to the database file at the beginning of the transaction.
(8) When a transaction is rolled back, the xTruncate method of the VFS
is called to restore the database file to the same size it was at
the beginning of the transaction. (In some VFSes, the xTruncate
method is a no-op, but that does not change the fact the SQLite will
invoke it.)
(9) Whenever the database file is modified, at least one bit in the range
of bytes from 24 through 39 inclusive will be changed prior to releasing
the EXCLUSIVE lock.
(10) The pattern of bits in bytes 24 through 39 shall not repeat in less
than one billion transactions.
(11) A database file is well-formed at the beginning and at the conclusion
of every transaction.
(12) An EXCLUSIVE lock must be held on the database file before making
any changes to the database file.
(13) A SHARED lock must be held on the database file before reading any
content out of the database file.
doc/tcl-extension-testing.md
+264
View File
@@ -0,0 +1,264 @@
# Test Procedures For The SQLite TCL Extension
## 1.0 Background
The SQLite TCL extension logic (in the
"[tclsqlite.c](/file/src/tclsqlite.c)" source
file) is statically linked into "textfixture" executable
which is the program used to do most of the testing
associated with "make test", "make devtest", and/or
"make releasetest". So the functionality of the SQLite
TCL extension is thoroughly vetted during normal testing. The
procedures below are designed to test the loadable extension
aspect of the SQLite TCL extension, and in particular to verify
that the "make tclextension-install" build target works and that
an ordinary tclsh can subsequently run "package require sqlite3".
This procedure can also be used as a template for how to set up
a local TCL+SQLite development environment. In other words, it
can be be used as a guide on how to compile per-user copies of
Tcl that are used to develop, test, and debug SQLite. In that
case, perhaps make minor changes to the procedure such as:
* Make TCLBUILD directory is permanent.
* Enable debugging symbols on the Tcl library build.
* Reduce the optimization level to -O0 for easier debugging.
* Also compile "wish" to go with each "tclsh".
<a id="unix"></a>
## 2.0 Testing On Unix-like Systems (Including Mac)
See also the [](./compile-for-unix.md) document which provides another
perspective on how to compile SQLite on unix-like systems.
### 2.1 Setup
<ol type="1">
<li value="1">
[Fossil][] installed.
<li> Check out source code and set environment variables:
<ol type="a">
<li> **TCLSOURCE** &rarr;
The top-level directory of a [Fossil][] check-out of the
[TCL source tree][tcl-fossil].
<li> **SQLITESOURCE** &rarr;
A Fossil check-out of the SQLite source tree.
<li> **TCLHOME** &rarr;
A directory that does not exist at the start of the test and which
will be deleted at the end of the test, and that will contain the
test builds of the TCL libraries and the SQLite TCL Extensions.
It is the top-most installation directory, i.e. the one provided
to Tcl's `./configure --prefix=/path/to/tcl`.
<li> **TCLVERSION** &rarr;
The `X.Y`-form version of Tcl being used: 8.6, 9.0, 9.1...
</ol>
</ol>
### 2.2 Testing TCL 8.x and 9.x on unix
From a checked-out copy of [the core Tcl tree][tcl-fossil]
<ol type="1">
<li value="3">`TCLVERSION=8.6` <br>
&uarr; A version of your choice. This process has been tested with
values of 8.6, 9.0, and 9.1 (as of 2025-04-16). The out-of-life
version 8.5 fails some of `make devtest` for undetermined reasons.
<li>`TCLHOME=$HOME/tcl/$TCLVERSION`
<li>`TCLSOURCE=/path/to/tcl/checkout`
<li>`SQLITESOURCE=/path/to/sqlite/checkout`
<li>`rm -fr $TCLHOME` <br>
&uarr; Ensure that no stale Tcl installation is laying around.
<li> `cd $TCLSOURCE`
<li> `fossil up core-8-6-branch` <br>
&uarr; The branch corresponding to `$TCLVERSION`, e.g.
`core-9-0-branch` or `trunk`.
<li> `fossil clean -x`
<li> `cd unix`
<li> `./configure --prefix=$TCLHOME --disable-shared` <br>
&uarr; The `--disable-shared` is to avoid the need to set `LD_LIBRARY_PATH`
when using this Tcl build.
<li> `make install`
<li> `cd $SQLITESOURCE`
<li> `fossil clean -x`
<li> `./configure --with-tcl=$TCLHOME --all`
<li> `make tclextension-install` <br>
&uarr; Verify extension installed at
`$TCLHOME/lib/tcl${TCLVERSION}/sqlite<SQLITE_VERSION>`.
<li> `make tclextension-list` <br>
&uarr; Verify TCL extension correctly installed.
<li> `make tclextension-verify` <br>
&uarr; Verify that the correct version is installed.
<li> `$TCLHOME/bin/tclsh[89].[0-9] test/testrunner.tcl release --explain` <br>
&uarr; Verify thousands of lines of output with no errors. Or
consider running "devtest" without --explain instead of "release".
</ol>
### 2.3 Cleanup
<ol type="1">
<li value="29"> `rm -rf $TCLHOME`
</ol>
<a id="windows"></a>
## 3.0 Testing On Windows
See also the [](./compile-for-windows.md) document which provides another
perspective on how to compile SQLite on Windows.
### 3.1 Setup for Windows
(These docs are not as up-to-date as the Unix docs, above.)
<ol type="1">
<li value="1">
[Fossil][] installed.
<li>
Unix-like command-line tools installed. Example:
[unxutils](https://unxutils.sourceforge.net/)
<li> [Visual Studio](https://visualstudio.microsoft.com/vs/community/)
installed. VS2015 or later required.
<li> Check out source code and set environment variables.
<ol type="a">
<li> **TCLSOURCE** &rarr;
The top-level directory of a Fossil check-out of the TCL source tree.
<li> **SQLITESOURCE** &rarr;
A Fossil check-out of the SQLite source tree.
<li> **TCLBUILD** &rarr;
A directory that does not exist at the start of the test and which
will be deleted at the end of the test, and that will contain the
test builds of the TCL libraries and the SQLite TCL Extensions.
<li> **ORIGINALPATH** &rarr;
The original value of %PATH%. In other words, set as follows:
`set ORIGINALPATH %PATH%`
</ol>
</ol>
### 3.2 Testing TCL 8.6 on Windows
<ol type="1">
<li value="5"> `mkdir %TCLBUILD%\tcl86`
<li> `cd %TCLSOURCE%\win`
<li> `fossil up core-8-6-16` <br>
&uarr; Or some other version of Tcl8.6.
<li> `fossil clean -x`
<li> `set INSTALLDIR=%TCLBUILD%\tcl86`
<li> `nmake /f makefile.vc release` <br>
&udarr; You *must* invoke the "release" and "install" targets
using separate "nmake" commands or tclsh86t.exe won't be
installed.
<li> `nmake /f makefile.vc install`
<li> `cd %SQLITESOURCE%`
<li> `fossil clean -x`
<li> `set TCLDIR=%TCLBUILD%\tcl86`
<li> `set PATH=%TCLBUILD%\tcl86\bin;%ORIGINALPATH%`
<li> `set TCLSH_CMD=%TCLBUILD%\tcl86\bin\tclsh86t.exe`
<li> `nmake /f Makefile.msc tclextension-install` <br>
&uarr; Verify extension installed at %TCLBUILD%\\tcl86\\lib\\tcl8.6\\sqlite3.*
<li> `nmake /f Makefile.msc tclextension-verify`
<li>`tclsh86t test/testrunner.tcl release --explain` <br>
&uarr; Verify thousands of lines of output with no errors. Or
consider running "devtest" without --explain instead of "release".
</ol>
### 3.3 Testing TCL 9.0 on Windows
<ol>
<li value="20"> `mkdir %TCLBUILD%\tcl90`
<li> `cd %TCLSOURCE%\win`
<li> `fossil up core-9-0-0` <br>
&uarr; Or some other version of Tcl9
<li> `fossil clean -x`
<li> `set INSTALLDIR=%TCLBUILD%\tcl90`
<li> `nmake /f makefile.vc release` <br>
&udarr; You *must* invoke the "release" and "install" targets
using separate "nmake" commands or tclsh90.exe won't be
installed.
<li> `nmake /f makefile.vc install`
<li> `cd %SQLITESOURCE%`
<li> `fossil clean -x`
<li> `set TCLDIR=%TCLBUILD%\tcl90`
<li> `set PATH=%TCLBUILD%\tcl90\bin;%ORIGINALPATH%`
<li> `set TCLSH_CMD=%TCLBUILD%\tcl90\bin\tclsh90.exe`
<li> `nmake /f Makefile.msc tclextension-install` <br>
&uarr; Verify extension installed at %TCLBUILD%\\tcl90\\lib\\sqlite3.*
<li> `nmake /f Makefile.msc tclextension-verify`
<li> `tclsh90 test/testrunner.tcl release --explain` <br>
&uarr; Verify thousands of lines of output with no errors. Or
consider running "devtest" without --explain instead of "release".
</ol>
### 3.4 Cleanup
<ol type="1">
<li value="35"> `rm -rf %TCLBUILD%`
</ol>
## 4.0 Testing the TEA(ish) Build (unix only)
This part requires following the setup instructions for Unix systems,
at the top of this document.
The former TEA, now TEA(ish), build of this extension uses the same
code as the builds described above but is provided in a form more
convenient for downstream Tcl users.
It lives in `autoconf/tea` and, as part of the `autoconf` bundle,
_cannot be tested directly from the canonical tree_. Instead it has to
be packaged.
### 4.1 Teaish Setup
Follow the same Tcl- and environment-related related setup described
in the first section of this document, up to and including the
installation of Tcl (unless, of course, it was already installed using
those same instructions).
### 4.2 Teaish Testing
<ol>
<li>`cd $SQLITESOURCE`
<li>Run either `make snapshot-tarball` or `make amalgamation-tarball`
&uarr;
Those steps will leave behind a temp dir called `mkpkg_tmp_dir`,
under which the extension is most readily reached. It can optionally
be extracted from the generated tarball, but that tarball was
generated from this dir, and reusing this dir is a time saver
during development.
<li> `cd mkpkg_tmp/tea`
<li> `./configure --with-tcl=$TCLHOME`
<li> `make test install` <br>
&uarr; Should run to completion without any errors.
<li> `make uninstall` <br>
&uarr; Will uninstall the extension. This _can_ be run
in the same invocation as the `install` target, but only
if the `-j#` make flag is _not_ used. If it is, the
install/uninstall steps will race and make a mess of things.
Parallel builds do not help in this build, anyway, as there's
only a single C file to compile.
</ol>
When actively developing and testing the teaish build, which requires
going through the tarball generation, there's a caveat about the
`mkpkg_tmp_dir` dir: it will be deleted every time a tarball is
built, the shell console which is parked in that
directory for testing needs to add `cd $PWD &&` to the start of the
build commands, like:
>
```
[user@host:.../mkpkg_tmp_dir/tea]$ \
cd $PWD && ./configure CFLAGS=-O0 --with-tcl=$TCLHOME \
&& make test install uninstall
```
### 4.3 Teaish Cleanup
<ol type="1">
<li> `rm -rf $TCLHOME`
<li> `cd $SQLITESOURCE; rm -fr mkpkg_tmp_dir; fossil clean -x`
</ol>
[Fossil]: https://fossil-scm.org/home
[tcl-fossil]: https://core.tcl-lang.org/tcl
doc/testrunner.md
+404
View File
@@ -0,0 +1,404 @@
# The testrunner.tcl Script
<ul type=none>
<li> 1. <a href=#overview>Overview</a>
<li> 2. <a href=#binary_tests>Binary Tests</a>
<ul type=none>
<li> 2.1. <a href=#organization_tests>Organization of Tcl Tests</a>
<li> 2.2. <a href=#run_tests>Commands to Run Tests</a>
<li> 2.3. <a href=#binary_test_failures>Investigating Binary Test Failures</a>
</ul>
<li> 3. <a href=#source_code_tests>Source Tests</a>
<ul type=none>
<li> 3.1. <a href=#commands_to_run_tests>Commands to Run SQLite Tests</a>
<li> 3.2. <a href=#zipvfs_tests>Running ZipVFS Tests</a>
<li> 3.3. <a href=#source_code_test_failures>Investigating Source Code Test Failures</a>
<li> 3.4. <a href=#fuzzdb>External Fuzzcheck Databases</a>
</ul>
<li> 4. <a href=#testrunner_options>Extra testrunner.tcl Options</a>
<li> 5. <a href=#cpu_cores>Controlling CPU Core Utilization</a>
</ul>
<a name=overview></a>
# 1. Overview
The testrunner.tcl program is a Tcl script used to run multiple SQLite
tests in parallel, thus reducing testing time on multi-core machines.
It supports the following types of tests:
* Tcl test scripts.
* Fuzzcheck tests, including using an external fuzzcheck database.
* Tests run with `make` commands. Examples:
- `make devtest`
- `make releasetest`
- `make sdevtest`
- `make testrunner`
The testrunner.tcl program stores output of all tests and builds run in
log file **testrunner.log**, created in the current working directory.
Search this file to find details of errors. Suggested search commands:
* `grep "^!" testrunner.log`
* `grep failed testrunner.log`
The testrunner.tcl program also populates SQLite database **testrunner.db**.
This database contains details of all tests run, running and to be run.
A useful query might be:
```
SELECT * FROM script WHERE state='failed'
```
You can get a summary of errors in a prior run by invoking commands like
these:
```
tclsh $(TESTDIR)/testrunner.tcl errors
tclsh $(TESTDIR)/testrunner.tcl errors -v
```
Running the command:
```
tclsh $(TESTDIR)/testrunner.tcl status
```
in the directory containing the testrunner.db database runs various queries
to produce a succinct report on the state of a running testrunner.tcl script.
A good way to keep and eye on test progress is to run either of the two
following commands:
```
watch tclsh $(TESTDIR)/testrunner.tcl status
tclsh $(TESTDIR)/testrunner.tcl status -d 2
```
Both of the commands above accomplish about the same thing, but the second
one has the advantage of not requiring "watch" to be installed on your
system.
Sometimes testrunner.tcl uses the `testfixture` binary that it is run with
to run tests (see "Binary Tests" below). Sometimes it builds testfixture and
other binaries in specific configurations to test (see "Source Tests").
<a name=binary_tests></a>
# 2. Binary Tests
The commands described in this section all run various combinations of the Tcl
test scripts using the `testfixture` binary used to run the testrunner.tcl
script (i.e. they do not invoke the compiler to build new binaries, or the
`make` command to run tests that are not Tcl scripts). The procedure to run
these tests is therefore:
1. Build the "testfixture" (or "testfixture.exe" for windows) binary using
whatever method seems convenient.
2. Test the binary built in step 1 by running testrunner.tcl with it,
perhaps with various options.
The following sub-sections describe the various options that can be
passed to testrunner.tcl to test binary testfixture builds.
<a name=organization_tests></a>
## 2.1. Organization of Tcl Tests
Tcl tests are stored in files that match the pattern *\*.test*. They are
found in both the $TOP/test/ directory, and in the various sub-directories
of the $TOP/ext/ directory of the source tree. Not all *\*.test* files
contain Tcl tests - a handful are Tcl scripts designed to invoke other
*\*.test* files.
The **veryquick** set of tests is a subset of all Tcl test scripts in the
source tree. In includes most tests, but excludes some that are very slow.
Almost all fault-injection tests (those that test the response of the library
to OOM or IO errors) are excluded. It is defined in source file
*test/permutations.test*.
The **full** set of tests includes all Tcl test scripts in the source tree.
To run a "full" test is to run all Tcl test scripts that can be found in the
source tree.
File *permutations.test* defines various test "permutations". A permutation
consists of:
* A subset of Tcl test scripts, and
* Runtime configuration to apply before running each test script
(e.g. enabling auto-vacuum, or disable lookaside).
Running **all** tests is to run all tests in the full test set, plus a dozen
or so permutations. The specific permutations that are run as part of "all"
are defined in file *testrunner_data.tcl*.
<a name=run_tests></a>
## 2.2. Commands to Run Tests
To run the "veryquick" test set, use either of the following:
```
./testfixture $TESTDIR/testrunner.tcl
./testfixture $TESTDIR/testrunner.tcl veryquick
```
To run the "full" test suite:
```
./testfixture $TESTDIR/testrunner.tcl full
```
To run the subset of the "full" test suite for which the test file name matches
a specified pattern (e.g. all tests that start with "fts5"), either of:
```
./testfixture $TESTDIR/testrunner.tcl fts5%
./testfixture $TESTDIR/testrunner.tcl 'fts5*'
```
Strictly speaking, for a test to be run the pattern must match the script
filename, not including the directory, using the rules of Tcl's
\[string match\] command. Except that before the matching is done, any "%"
characters specified as part of the pattern are transformed to "\*".
To run "all" tests (full + permutations):
```
./testfixture $TESTDIR/testrunner.tcl all
```
<a name=binary_test_failures></a>
## 2.3. Investigating Binary Test Failures
If a test fails, testrunner.tcl reports name of the Tcl test script and, if
applicable, the name of the permutation, to stdout. This information can also
be retrieved from either *testrunner.log* or *testrunner.db*.
If there is no permutation, the individual test script may be run with:
```
./testfixture $PATH_TO_SCRIPT
```
Or, if the failure occured as part of a permutation:
```
./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
```
TODO: An example instead of "$PERMUTATION" and $PATH\_TO\_SCRIPT?
<a name=source_code_tests></a>
# 3. Source Code Tests
The commands described in this section invoke the C compiler to build
binaries from the source tree, then use those binaries to run Tcl and
other tests. The advantages of this are that:
* it is possible to test multiple build configurations with a single
command, and
* it ensures that tests are always run using binaries created with the
same set of compiler options.
The testrunner.tcl commands described in this section may be run using
either a *testfixture* (or testfixture.exe) build, or with any other Tcl
shell that supports SQLite 3.31.1 or newer via "package require sqlite3".
TODO: ./configure + Makefile.msc build systems.
<a name=commands_to_run_tests></a>
## 3.1. Commands to Run SQLite Tests
The **mdevtest** command is equivalent to running the veryquick tests and
the `make fuzztest` target once for each of two --enable-all builds - one
with debugging enabled and one without:
```
tclsh $TESTDIR/testrunner.tcl mdevtest
```
In other words, it is equivalent to running:
```
$TOP/configure --enable-all --enable-debug
make fuzztest
make testfixture
./testfixture $TOP/test/testrunner.tcl veryquick
# Then, after removing files created by the tests above:
$TOP/configure --enable-all OPTS="-O0"
make fuzztest
make testfixture
./testfixture $TOP/test/testrunner.tcl veryquick
```
The **sdevtest** command is identical to the mdevtest command, except that the
second of the two builds is a sanitizer build. Specifically, this means that
OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":
```
tclsh $TESTDIR/testrunner.tcl sdevtest
```
The **release** command runs lots of tests under lots of builds. It runs
different combinations of builds and tests depending on whether it is run
on Linux, Windows or OSX. Refer to *testrunner\_data.tcl* for the details
of the specific tests run.
```
tclsh $TESTDIR/testrunner.tcl release
```
As with <a href=#source code tests>source code tests</a>, one or more patterns
may be appended to any of the above commands (mdevtest, sdevtest or release).
Pattern matching is used for both Tcl tests and fuzz tests.
```
tclsh $TESTDIR/testrunner.tcl release rtree%
```
<a name=zipvfs_tests></a>
## 3.2. Running ZipVFS Tests
testrunner.tcl can build a zipvfs-enabled testfixture and use it to run
tests from the Zipvfs project with the following command:
```
tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
```
This can be combined with any of "mdevtest", "sdevtest" or "release" to
test both SQLite and Zipvfs with a single command:
```
tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
```
<a name=source_code_test_failures></a>
## 3.3. Investigating Source Code Test Failures
Investigating a test failure that occurs during source code testing is a
two step process:
1. Recreating the build configuration in which the test failed, and
2. Re-running the actual test.
To recreate a build configuration, use the testrunner.tcl **script** command
to create a build script. A build script is a bash script on Linux or OSX, or
a dos \*.bat file on windows. For example:
```
# Create a script that recreates build configuration "Device-One" on
# Linux or OSX:
tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh
# Create a script that recreates build configuration "Have-Not" on Windows:
tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat
```
The generated bash or \*.bat file script accepts a single argument - a makefile
target to build. This may be used either to run a `make` command test directly,
or else to build a testfixture (or testfixture.exe) binary with which to
run a Tcl test script, as <a href=#binary_test_failures>described above</a>.
<a name=fuzzdb></a>
## 3.4 External Fuzzcheck Databases
Testrunner.tcl will also run fuzzcheck against an external (out of tree)
database, for example fuzzcheck databases generated by dbsqlfuzz. To do
this, simply add the "`--fuzzdb` *FILENAME*" command-line option or set
the FUZZDB environment variable to the name of the external
database. For large external databases, testrunner.tcl will automatically use
the "`--slice`" command-line option of fuzzcheck to divide the work up into
multiple jobs, to increase parallelism.
Thus, for example, to run a full releasetest including an external
dbsqlfuzz database, run a command like one of these:
```
tclsh test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
FUZZDB=../fuzz/20250415.db make releasetest
nmake /f Makefile.msc FUZZDB=../fuzz/20250415.db releasetest
```
The patternlist option to testrunner.tcl will match against fuzzcheck
databases. So if you want to run *only* tests involving the external
database, you can use a command something like this:
```
tclsh test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db
```
<a name=testrunner_options></a>
# 4. Extra testrunner.tcl Options
The testrunner.tcl script options in this section may be used with both source
code and binary tests.
The **--buildonly** option instructs testrunner.tcl just to build the binaries
required by a test, not to run any actual tests. For example:
```
# Build binaries required by release test.
tclsh $TESTDIR/testrunner.tcl --buildonly release"
```
The **--dryrun** option prevents testrunner.tcl from building any binaries
or running any tests. Instead, it just writes the shell commands that it
would normally execute into the testrunner.log file. Example:
```
# Log the shell commmands that make up the mdevtest test.
tclsh $TESTDIR/testrunner.tcl --dryrun mdevtest"
```
The **--explain** option is similar to --dryrun in that it prevents
testrunner.tcl from building any binaries or running any tests. The
difference is that --explain prints on standard output a human-readable
summary of all the builds and tests that would have been run.
```
# Show what builds and tests would have been run
tclsh $TESTDIR/testrunner.tcl --explain mdevtest
```
The **--status** option uses VT100 escape sequences to display the test
status full-screen. This is similar to running
"`watch test/testrunner status`" in a separate window, just more convenient.
Unfortunately, this option does not work correctly on Windows, due to the
sketchy implementation of VT100 escapes on the Windows console.
<a name=cpu_cores></a>
# 5. Controlling CPU Core Utilization
When running either binary or source code tests, testrunner.tcl reports the
number of jobs it intends to use to stdout. e.g.
```
$ ./testfixture $TESTDIR/testrunner.tcl
splitting work across 16 jobs
... more output ...
```
By default, testfixture.tcl attempts to set the number of jobs to the number
of real cores on the machine. This can be overridden using the "--jobs" (or -j)
switch:
```
$ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
splitting work across 8 jobs
... more output ...
```
The number of jobs may also be changed while an instance of testrunner.tcl is
running by exucuting the following command from the directory containing the
testrunner.log and testrunner.db files:
```
$ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
```
doc/trusted-schema.md
+142
View File
@@ -0,0 +1,142 @@
# The new-security-options branch
## The problem that the [new-security-options](/timeline?r=new-security-options) branch tries to solve
An attacker might modify the schema of an SQLite database by adding
structures that cause code to run when some other application opens and
reads the database. For example, the attacker might replace a table
definition with a view. Or the attacker might add triggers to tables
or views, or add new CHECK constraints or generated columns or indexes
with expressions in the index list or in the WHERE clause. If the
added features invoke SQL functions or virtual tables with side effects,
that might cause harm to the system if run by a high-privilege victim.
Or, the added features might exfiltrate information if the database is
read by a high-privilege victim.
The changes in this branch strive to make it easier for high-privilege
applications to safely read SQLite database files that might have been
maliciously corrupted by an attacker.
## Overview of changes in [new-security-options](/timeline?r=new-security-options)
The basic idea is to tag every SQL function and virtual table with one
of three risk levels:
1. Innocuous
2. Normal
3. Direct-Only
Innocuous functions/vtabs are safe and can be used at any time.
Direct-only elements, in contrast, might have cause side-effects and
should only be used from top-level SQL, not from within triggers or views nor
in elements of the schema such as CHECK constraint, DEFAULT values,
generated columns, index expressions, or in the WHERE clause of a
partial index that are potentially under the control of an attacker.
Normal elements behave like Innocuous if TRUSTED\_SCHEMA=on
and behave like direct-only if TRUSTED\_SCHEMA=off.
Application-defined functions and virtual tables go in as Normal unless
the application takes deliberate steps to change the risk level.
For backwards compatibility, the default is TRUSTED\_SCHEMA=on. Documentation
will be updated to recommend applications turn TRUSTED\_SCHEMA to off.
An innocuous function or virtual table is one that can only read content
from the database file in which it resides, and can only alter the database
in which it resides. Most SQL functions are innocuous. For example, there
is no harm in an attacker running the abs() function.
Direct-only elements that have side-effects that go outside the database file
in which it lives, or return information from outside of the database file.
Examples of direct-only elements include:
1. The fts3\_tokenizer() function
2. The writefile() function
3. The readfile() function
4. The zipvfs virtual table
5. The csv virtual table
We do not want an attacker to be able to add these kinds of things to
the database schema and possibly trick a high-privilege application
from performing any of these actions. Therefore, functions and vtabs
with side-effects are marked as Direct-Only.
Legacy applications might add other risky functions or vtabs. Those will
go in as "Normal" by default. For optimal security, we want those risky
app-defined functions and vtabs to be direct-only, but making that the
default might break some legacy applications. Hence, all app-defined
functions and vtabs go in as Normal, but the application can switch them
over to "Direct-Only" behavior using a single pragma.
The restrictions on the use of functions and virtual tables do not apply
to TEMP. A TEMP VIEW or a TEMP TRIGGER can use any valid SQL function
or virtual table. The idea is that TEMP views and triggers must be
directly created by the application and are thus under the control of the
application. TEMP views and triggers cannot be created by an attacker who
corrupts the schema of a persistent database file. Hence TEMP views and
triggers are safe.
## Specific changes
1. New sqlite3\_db\_config() option SQLITE\_DBCONFIG\_TRUSTED\_SCHEMA for
turning TRUSTED\_SCHEMA on and off. It defaults to ON.
2. Compile-time option -DSQLITE\_TRUSTED\_SCHEMA=0 causes the default
TRUSTED\_SCHEMA setting to be off.
3. New pragma "PRAGMA trusted\_schema=(ON\|OFF);". This provides access
to the TRUSTED_SCHEMA setting for application coded using scripting
languages or other secondary languages where they are unable to make
calls to sqlite3\_db\_config().
4. New options for the "enc" parameter to sqlite3\_create\_function() and
its kin:
<ol type="a">
<li> _SQLITE\_INNOCUOUS_ &rarr; tags the new functions as Innocuous
<li> _SQLITE\_DIRECTONLY_ &rarr; tags the new functions as Direct-Only
</ol>
5. New options to sqlite3\_vtab\_config():
<ol type="a">
<li> _SQLITE\_VTAB\_INNOCUOUS_ &rarr; tags the vtab as Innocuous
<li> _SQLITE\_VTAB\_DIRECTONLY_ &rarr; tags the vtab as Direct-Only
</ol>
6. Change many of the functions and virtual tables in the SQLite source
tree to use one of the tags above.
7. Enhanced PRAGMA function\_list and virtual-table "pragma\_function\_list"
with additional columns. The columns now are:
<ul>
<li> _name_ &rarr; Name of the function
<li> _builtin_ &rarr; 1 for built-in functions. 0 otherwise.
<li> _type_ &rarr; 's'=Scalar, 'a'=Aggregate, 'w'=Window
<li> _enc_ &rarr; 'utf8', 'utf16le', or 'utf16be'
<li> _narg_ &rarr; number of argument
<li> _flags_ &rarr; Bitmask of SQLITE\_INNOCUOUS, SQLITE\_DIRECTONLY,
SQLITE\_DETERMINISTIC, SQLITE\_SUBTYPE, and
SQLITE\_FUNC\_INTERNAL flags.
</ul>
<p>The last four columns are new.
8. The function\_list PRAGMA now also shows all entries for each function.
So, for example, if a function can take either 2 or 3 arguments,
there are separate rows for the 2-argument and 3-argument versions of
the function.
## Additional Notes
The function_list enhancements allow the application to query the set
of SQL functions that meet various criteria. For example, to see all
SQL functions that are never allowed to be used in the schema or in
trigger or views:
~~~
SELECT DISTINCT name FROM pragma_function_list
WHERE (flags & 0x80000)!=0
ORDER BY name;
~~~
Doing the same is not possible for virtual tables, as a virtual table
might be Innocuous, Normal, or Direct-Only depending on the arguments
passed into the xConnect method.
doc/vdbesort-memory.md
+49
View File
@@ -0,0 +1,49 @@
20-11-2020
# Memory Allocation In vdbesort.c
Memory allocation is slightly different depending on:
* whether or not SQLITE_CONFIG_SMALL_MALLOC is set, and
* whether or not worker threads are enabled.
## SQLITE_CONFIG_SMALL_MALLOC=0
Assuming SQLITE_CONFIG_SMALL_MALLOC is not set, keys passed to the sorter are
added to an in-memory buffer. This buffer is grown using sqlite3Realloc() as
required it reaches the size configured for the main pager cache using "PRAGMA
cache_size". i.e. if the user has executed "PRAGMA main.cache_size = -2048",
then this buffer is allowed to grow up to 2MB in size.
Once the buffer has grown to its threshold, keys are sorted and written to
a temp file. If worker threads are not enabled, this is the only significant
allocation the sorter module makes. After keys are sorted and flushed out to
the temp file, the buffer is reused to accumulate the next batch of keys.
If worker threads are available, then the buffer is passed to a worker thread
to sort and flush once it is full, and a new buffer allocated to allow the
main thread to continue to accumulate keys. Buffers are reused once they
have been flushed, so in this case at most (nWorker+1) buffers are allocated
and used, where nWorker is the number of configured worker threads.
There are no other significant users of heap memory in the sorter module.
Once sorted buffers of keys have been flushed to disk, they are read back
either by mapping the file (via sqlite3_file.xFetch()) or else read back
in one page at a time.
All buffers are allocated by the main thread. A sorter object is associated
with a single database connection, to which it holds a pointer.
## SQLITE_CONFIG_SMALL_MALLOC=1
This case is similar to the above, except that instead of accumulating
multiple keys in a single large buffer, sqlite3VdbeSorterWrite() stores
keys in a regular heap-memory linked list (one allocation per element).
List elements are freed as they are flushed to disk, either by the main
thread or by a worker thread.
Each time a key is added the sorter (and an allocation made),
sqlite3HeapNearlyFull() is called. If it returns true, the current
list of keys is flushed to a temporary file, even if it has not yet
reached the size threshold.
doc/vfs-shm.txt
+130
View File
@@ -0,0 +1,130 @@
The 5 states of an historical rollback lock as implemented by the
xLock, xUnlock, and xCheckReservedLock methods of the sqlite3_io_methods
object are:
UNLOCKED
SHARED
RESERVED
PENDING
EXCLUSIVE
The wal-index file has a similar locking hierarchy implemented using
the xShmLock method of the sqlite3_vfs object, but with 7
states. Each connection to a wal-index file must be in one of
the following 7 states:
UNLOCKED
READ
READ_FULL
WRITE
PENDING
CHECKPOINT
RECOVER
These roughly correspond to the 5 states of a rollback lock except
that SHARED is split out into 2 states: READ and READ_FULL and
there is an extra RECOVER state used for wal-index reconstruction.
The meanings of the various wal-index locking states is as follows:
UNLOCKED - The wal-index is not in use.
READ - Some prefix of the wal-index is being read. Additional
wal-index information can be appended at any time. The
newly appended content will be ignored by the holder of
the READ lock.
READ_FULL - The entire wal-index is being read. No new information
can be added to the wal-index. The holder of a READ_FULL
lock promises never to read pages from the database file
that are available anywhere in the wal-index.
WRITE - It is OK to append to the wal-index file and to adjust
the header to indicate the new "last valid frame".
PENDING - Waiting on all READ locks to clear so that a
CHECKPOINT lock can be acquired.
CHECKPOINT - It is OK to write any WAL data into the database file
and zero the last valid frame field of the wal-index
header. The wal-index file itself may not be changed
other than to zero the last valid frame field in the
header.
RECOVER - Held during wal-index recovery. Used to prevent a
race if multiple clients try to recover a wal-index at
the same time.
A particular lock manager implementation may coalesce one or more of
the wal-index locking states, though with a reduction in concurrency.
For example, an implementation might implement only exclusive locking,
in which case all states would be equivalent to CHECKPOINT, meaning that
only one reader or one writer or one checkpointer could be active at a
time. Or, an implementation might combine READ and READ_FULL into
a single state equivalent to READ, meaning that a writer could
coexist with a reader, but no reader or writers could coexist with a
checkpointer.
The lock manager must obey the following rules:
(1) A READ cannot coexist with CHECKPOINT.
(2) A READ_FULL cannot coexist with WRITE.
(3) None of WRITE, PENDING, CHECKPOINT, or RECOVER can coexist.
The SQLite core will obey the next set of rules. These rules are
assertions on the behavior of the SQLite core which might be verified
during testing using an instrumented lock manager.
(5) No part of the wal-index will be read without holding either some
kind of SHM lock or an EXCLUSIVE lock on the original database.
The original database is the file named in the 2nd parameter to
the xShmOpen method.
(6) A holder of a READ_FULL will never read any page of the database
file that is contained anywhere in the wal-index.
(7) No part of the wal-index other than the header will be written nor
will the size of the wal-index grow without holding a WRITE or
an EXCLUSIVE on the original database file.
(8) The wal-index header will not be written without holding one of
WRITE, CHECKPOINT, or RECOVER on the wal-index or an EXCLUSIVE on
the original database files.
(9) A CHECKPOINT or RECOVER must be held on the wal-index, or an
EXCLUSIVE on the original database file, in order to reset the
last valid frame counter in the header of the wal-index back to zero.
(10) A WRITE can only increase the last valid frame pointer in the header.
The SQLite core will only ever send requests for UNLOCK, READ, WRITE,
CHECKPOINT, or RECOVER to the lock manager. The SQLite core will never
request a READ_FULL or PENDING lock though the lock manager may deliver
those locking states in response to READ and CHECKPOINT requests,
respectively, if and only if the requested READ or CHECKPOINT cannot
be delivered.
The following are the allowed lock transitions:
Original-State Request New-State
-------------- ---------- ----------
(11a) UNLOCK READ READ
(11b) UNLOCK READ READ_FULL
(11c) UNLOCK CHECKPOINT PENDING
(11d) UNLOCK CHECKPOINT CHECKPOINT
(11e) READ UNLOCK UNLOCK
(11f) READ WRITE WRITE
(11g) READ RECOVER RECOVER
(11h) READ_FULL UNLOCK UNLOCK
(11i) READ_FULL WRITE WRITE
(11j) READ_FULL RECOVER RECOVER
(11k) WRITE READ READ
(11l) PENDING UNLOCK UNLOCK
(11m) PENDING CHECKPOINT CHECKPOINT
(11n) CHECKPOINT UNLOCK UNLOCK
(11o) RECOVER READ READ
These 15 transitions are all that needs to be supported. The lock
manager implementation can assert that fact. The other 27 possible
transitions among the 7 locking states will never occur.
doc/wal-lock.md
+88
View File
@@ -0,0 +1,88 @@
# Wal-Mode Blocking Locks
On some Unix-like systems, SQLite may be configured to use POSIX blocking locks
by:
* building the library with SQLITE\_ENABLE\_SETLK\_TIMEOUT defined, and
* configuring a timeout in ms using the sqlite3\_busy\_timeout() API.
Blocking locks may be advantageous as (a) waiting database clients do not
need to continuously poll the database lock, and (b) using blocking locks
facilitates transfer of OS priority between processes when a high priority
process is blocked by a lower priority one.
Only read/write clients use blocking locks. Clients that have read-only access
to the \*-shm file never use blocking locks.
Threads or processes that access a single database at a time never deadlock as
a result of blocking database locks. But it is of course possible for threads
that lock multiple databases simultaneously to do so. In most cases the OS will
detect the deadlock and return an error.
## Wal Recovery
Wal database "recovery" is a process required when the number of connected
database clients changes from zero to one. In this case, a client is
considered to connect to the database when it first reads data from it.
Before recovery commences, an exclusive WRITER lock is taken.
Without blocking locks, if two clients attempt recovery simultaneously, one
fails to obtain the WRITER lock and either invokes the busy-handler callback or
returns SQLITE\_BUSY to the user. With blocking locks configured, the second
client blocks on the WRITER lock.
## Database Readers
Usually, read-only are not blocked by any other database clients, so they
have no need of blocking locks.
If a read-only transaction is being opened on a snapshot, the CHECKPOINTER
lock is required briefly as part of opening the transaction (to check that a
checkpointer is not currently overwriting the snapshot being opened). A
blocking lock is used to obtain the CHECKPOINTER lock in this case. A snapshot
opener may therefore block on and transfer priority to a checkpointer in some
cases.
## Database Writers
A database writer must obtain the exclusive WRITER lock. It uses a blocking
lock to do so if any of the following are true:
* the transaction is an implicit one consisting of a single DML or DDL
statement, or
* the transaction is opened using BEGIN IMMEDIATE or BEGIN EXCLUSIVE, or
* the first SQL statement executed following the BEGIN command is a DML or
DDL statement (not a read-only statement like a SELECT).
In other words, in all cases except when an open read-transaction is upgraded
to a write-transaction. In that case a non-blocking lock is used.
## Database Checkpointers
Database checkpointers takes the following locks, in order:
* The exclusive CHECKPOINTER lock.
* The exclusive WRITER lock (FULL, RESTART and TRUNCATE only).
* Exclusive lock on read-mark slots 1-N. These are immediately released after being taken.
* Exclusive lock on read-mark 0.
* Exclusive lock on read-mark slots 1-N again. These are immediately released
after being taken (RESTART and TRUNCATE only).
All of the above use blocking locks.
## Summary
With blocking locks configured, the only cases in which clients should see an
SQLITE\_BUSY error are:
* if the OS does not grant a blocking lock before the configured timeout
expires, and
* when an open read-transaction is upgraded to a write-transaction.
In all other cases the blocking locks implementation should prevent clients
from having to handle SQLITE\_BUSY errors and facilitate appropriate transfer
of priorities between competing clients.
Clients that lock multiple databases simultaneously must be wary of deadlock.