admin/SQLite-fork
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Adding the sqlite sourcecode

Browse Source
This commit is contained in:
Jiří Stach committed 2025-07-17 11:54:48 +02:00
1 parent bdb54a1f8f
commit 406b584760
2208 files changed
+1172785 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

LICENSE.md
+91
View File
@@ -0,0 +1,91 @@
License Information
===================
SQLite Is Public Domain
-----------------------
The SQLite source code, including all of the files in the directories
listed in the bullets below are
[Public Domain](https://sqlite.org/copyright.html).
The authors have submitted written affidavits releasing their work to
the public for any use. Every byte of the public-domain code can be
traced back to the original authors. The files of this repository
that are public domain include the following:
* All of the primary SQLite source code files found in the
[src/ directory](https://sqlite.org/src/tree/src?type=tree&expand)
* All of the test cases and testing code in the
[test/ directory](https://sqlite.org/src/tree/test?type=tree&expand)
* All of the SQLite extension source code and test cases in the
[ext/ directory](https://sqlite.org/src/tree/ext?type=tree&expand)
* All code that ends up in the "sqlite3.c" and "sqlite3.h" build products
that actually implement the SQLite RDBMS.
* All of the code used to compile the
[command-line interface](https://sqlite.org/cli.html)
* All of the code used to build various utility programs such as
"sqldiff", "sqlite3_rsync", and "sqlite3_analyzer".
The public domain source files usually contain a header comment
similar to the following to make it clear that the software is
public domain.
> ~~~
The author disclaims copyright to this source code. In place of
a legal notice, here is a blessing:
* May you do good and not evil.
* May you find forgiveness for yourself and forgive others.
* May you share freely, never taking more than you give.
~~~
Almost every file you find in this source repository will be
public domain. But there are a small number of exceptions:
Non-Public-Domain Code Included With This Source Repository AS A Convenience
----------------------------------------------------------------------------
This repository contains a (relatively) small amount of non-public-domain
code used to help implement the configuration and build logic. In other
words, there are some non-public-domain files used to implement:
> ~~~
./configure && make
~~~
In all cases, the non-public-domain files included with this
repository have generous BSD-style licenses. So anyone is free to
use any of the code in this source repository for any purpose, though
attribution may be required to reuse or republish the configure and
build scripts. None of the non-public-domain code ever actually reaches
the build products, such as "sqlite3.c", however, so no attribution is
required to use SQLite itself. The non-public-domain code consists of
scripts used to help compile SQLite. The non-public-domain code is
technically not part of SQLite. The non-public-domain code is
included in this repository as a convenience to developers, so that those
who want to build SQLite do not need to go download a bunch of
third-party build scripts in order to compile SQLite.
Non-public-domain code included in this respository includes:
* The ["autosetup"](http://msteveb.github.io/autosetup/) configuration
system that is contained (mostly) the autosetup/ directory, but also
includes the "./configure" script at the top-level of this archive.
Autosetup has a separate BSD-style license. See the
[autosetup/LICENSE](http://msteveb.github.io/autosetup/license/)
for details.
* There are BSD-style licenses on some of the configuration
software found in the legacy autoconf/ directory and its
subdirectories.
The following unix shell command is can be run from the top-level
of this source repository in order to remove all non-public-domain
code:
> ~~~
rm -rf configure autosetup autoconf
~~~
If you unpack this source repository and then run the command above, what
is left will be 100% public domain.
Makefile.in
+342
View File
@@ -0,0 +1,342 @@
#!/usr/bin/make
# ^^^^ help out editors which guess this file's type.
#
# Makefile for SQLITE
#
# This makefile is intended to be configured automatically using the
# configure script.
#
# The docs for many of its variables are in the primary static
# makefile, main.mk (which this one includes at runtime).
#
all:
########################################################################
#
# Known TODOs/FIXMEs/TOIMPROVEs for the autosetup port, in no
# particular order...
#
# - TEA pieces.
#
# - Replace the autotools-specific distribution deliverable(s).
#
# - Confirm whether cross-compilation works and patch it
# appropriately.
#
# Maintenance reminders:
#
# - This makefile should remain as POSIX-make-compatible as possible:
# https://pubs.opengroup.org/onlinepubs/9799919799/utilities/make.html
#
# - The naming convention of some vars, using periods instead of
# underscores, though unconventional, was selected for a couple of
# reasons: 1) Personal taste (for which there is no accounting). 2)
# It is thought to help defend against inadvertent injection of
# those vars via environment variables (because X.Y is not a legal
# environment variable name). "Feature or bug?" is debatable and
# this naming convention may be reverted if it causes any grief.
#
#
# The top-most directory of the source tree. This is the directory
# that contains this "Makefile.in" and the "configure" script.
#
TOP = @abs_top_srcdir@
#
# Autotools-conventional vars which are used by package installation
# rules in main.mk. To get sane handling when a user overrides only
# a subset of these, we perform some acrobatics with these vars
# in the configure script: see [proj-remap-autoconf-dir-vars] for
# full details.
#
# For completeness's sake, the aforementioned conventional vars which
# are relevant to our installation rules are:
#
# datadir = $(prefix)/share
# mandir = $(datadir)/man
# includedir = $(prefix)/include
# exec_prefix = $(prefix)
# bindir = $(exec_prefix)/bin
# libdir = $(exec_prefix)/lib
#
# Our builds do not require any of their relatives:
#
# sbindir = $(exec_prefix)/sbin
# sysconfdir = /etc
# sharedstatedir = $(prefix)/com
# localstatedir = /var
# runstatedir = /run
# infodir = $(datadir)/info
# libexecdir = $(exec_prefix)/libexec
#
prefix = @prefix@
datadir = @datadir@
mandir = @mandir@
includedir = @includedir@
exec_prefix = @exec_prefix@
bindir = @bindir@
libdir = @libdir@
INSTALL = @BIN_INSTALL@
AR = @AR@
AR.flags = cr
CC = @CC@
B.cc = @CC_FOR_BUILD@ @BUILD_CFLAGS@
T.cc = $(CC)
#
# $(CFLAGS) is problematic because it is frequently overridden when
# invoking make, which loses things like -fPIC. So... we avoid using
# it directly and instead add a level of indirection. We combine
# $(CFLAGS) and $(CPPFLAGS) here because that's the way the legacy
# build did it and many builds rely on that. See main.mk for more
# details.
#
# Historical note: the pre-3.48 build only honored CPPFLAGS at
# configure-time, and expanded them into the generated Makefile. There
# are, in that build, no uses of CPPFLAGS in the configure-expanded
# Makefile. Ergo: if a client configures with CPPFLAGS=... and then
# explicitly passes CFLAGS=... to make, the CPPFLAGS will be
# lost. That behavior is retained in 3.48+.
#
CFLAGS = @CFLAGS@ @CPPFLAGS@
#
# $(LDFLAGS.configure) represents any LDFLAGS=... the client passes to
# configure. See main.mk.
#
LDFLAGS.configure = @LDFLAGS@
#
# CFLAGS.core is documented in main.mk.
#
CFLAGS.core = @SH_CFLAGS@
LDFLAGS.shlib = @SH_LDFLAGS@
LDFLAGS.zlib = @LDFLAGS_ZLIB@
LDFLAGS.math = @LDFLAGS_MATH@
LDFLAGS.rpath = @LDFLAGS_RPATH@
LDFLAGS.pthread = @LDFLAGS_PTHREAD@
LDFLAGS.dlopen = @LDFLAGS_DLOPEN@
LDFLAGS.readline = @LDFLAGS_READLINE@
CFLAGS.readline = @CFLAGS_READLINE@
LDFLAGS.icu = @LDFLAGS_ICU@
LDFLAGS.rt = @LDFLAGS_RT@
CFLAGS.icu = @CFLAGS_ICU@
LDFLAGS.libsqlite3.soname = @LDFLAGS_LIBSQLITE3_SONAME@
# soname: see https://sqlite.org/src/forumpost/5a3b44f510df8ded
LDFLAGS.libsqlite3.os-specific = \
@LDFLAGS_MAC_CVERSION@ @LDFLAGS_MAC_INSTALL_NAME@ @LDFLAGS_OUT_IMPLIB@
# os-specific: see
# - https://sqlite.org/forum/forumpost/9dfd5b8fd525a5d7
# - https://sqlite.org/forum/forumpost/0c7fc097b2
# - https://sqlite.org/forum/forumpost/5651662b8875ec0a
libsqlite3.DLL.basename = @SQLITE_DLL_BASENAME@
# DLL.basename: see https://sqlite.org/forum/forumpost/828fdfe904
libsqlite3.out.implib = @SQLITE_OUT_IMPLIB@
# libsqlite3.out.implib => the output filename part of LDFLAGS_OUT_IMPLIB.
ENABLE_LIB_SHARED = @ENABLE_LIB_SHARED@
ENABLE_LIB_STATIC = @ENABLE_LIB_STATIC@
HAVE_WASI_SDK = @HAVE_WASI_SDK@
libsqlite3.DLL.install-rules = @SQLITE_DLL_INSTALL_RULES@
# -fsanitize flags for the fuzzcheck-asap app
CFLAGS.fuzzcheck-asan.fsanitize = @CFLAGS_ASAN_FSANITIZE@
#
# Intended to either be empty or be set to -g -DSQLITE_DEBUG=1.
#
T.cc.TARGET_DEBUG = @TARGET_DEBUG@
#
# $(JIMSH) and $(CFLAGS.jimsh) are documented in main.mk. $(JIMSH)
# must start with a path component so that it can be invoked as a
# shell command.
#
CFLAGS.jimsh = @CFLAGS_JIMSH@
JIMSH = ./jimsh$(T.exe)
#
# $(B.tclsh) is documented in main.mk.
#
B.tclsh = @BTCLSH@
$(B.tclsh):
#
# $(OPT_FEATURE_FLAGS) is documented in main.mk.
#
# The appending of $(OPTIONS) to $(OPT_FEATURE_FLAGS) is historical
# and somewhat confusing because there's another var, $(OPTS), which
# has a similar (but not identical) role.
#
OPT_FEATURE_FLAGS = @OPT_FEATURE_FLAGS@ $(OPTIONS)
#
# Version (X.Y.Z) number for the SQLite being compiled.
#
PACKAGE_VERSION = @PACKAGE_VERSION@
#
# Filename extensions for binaries and libraries
#
B.exe = @BUILD_EXEEXT@
T.exe = @TARGET_EXEEXT@
B.dll = @BUILD_DLLEXT@
T.dll = @TARGET_DLLEXT@
B.lib = @BUILD_LIBEXT@
T.lib = @TARGET_LIBEXT@
#
# $(HAVE_TCL) is 1 if the configure script was able to locate the
# tclConfig.sh file, else it is 0. When this variable is 1, the TCL
# extension library (libtclsqlite3.so) and related testing apps are
# built.
#
HAVE_TCL = @HAVE_TCL@
#
# $(TCLSH_CMD) is the command to use for tclsh - normally just
# "tclsh", but we may know the specific version we want to use. This
# must point to a canonical TCL interpreter, not JimTCL.
#
TCLSH_CMD = @TCLSH_CMD@
TCL_CONFIG_SH = @TCL_CONFIG_SH@
#
# TCL config info from tclConfig.sh
#
# We have to inject this differently in main.mk to accommodate static
# makefiles, so we don't currently bother to export it here. This
# block is retained in case we decide that we do indeed need to export
# it at configure-time instead of calculate it at make-time.
#
#TCL_INCLUDE_SPEC = @TCL_INCLUDE_SPEC@
#TCL_LIB_SPEC = @TCL_LIB_SPEC@
#TCL_STUB_LIB_SPEC = @TCL_STUB_LIB_SPEC@
#TCL_EXEC_PREFIX = @TCL_EXEC_PREFIX@
#TCL_VERSION = @TCL_VERSION@
#
# $(TCLLIBDIR) = where to install the tcl plugin. If this is empty, it
# is calculated at make-time by the targets which need it but we
# export it here so that it can be set at configure-time, so that
# clients are not required to pass it at make-time, or may set it in
# their environment to override it.
#
TCLLIBDIR = @TCLLIBDIR@
#
# Additional options when running tests using testrunner.tcl
# This is usually either blank or --status.
#
TSTRNNR_OPTS = @TSTRNNR_OPTS@
#
# If gcov support was enabled by the configure script, add the appropriate
# flags here. It's not always as easy as just having the user add the right
# CFLAGS / LDFLAGS, because libtool wants to use CFLAGS when linking, which
# causes build errors with -fprofile-arcs -ftest-coverage with some GCCs.
# Supposedly GCC does the right thing if you use --coverage, but in
# practice it still fails. See:
#
# http://www.mail-archive.com/debian-gcc@lists.debian.org/msg26197.html
#
# for more info.
#
CFLAGS.gcov1 = -DSQLITE_COVERAGE_TEST=1 -fprofile-arcs -ftest-coverage
LDFLAGS.gcov1 = -lgcov
USE_GCOV = @USE_GCOV@
T.compile.gcov = $(CFLAGS.gcov$(USE_GCOV))
T.link.gcov = $(LDFLAGS.gcov$(USE_GCOV))
#
# Vars with the AS_ prefix are specifically related to AutoSetup.
#
# AS_AUTO_DEF is the main configure script.
#
AS_AUTO_DEF = $(TOP)/auto.def
#
# Shell commands to re-run $(TOP)/configure with the same args it was
# invoked with to produce this makefile.
#
AS_AUTORECONFIG = @SQLITE_AUTORECONFIG@
USE_AMALGAMATION ?= @USE_AMALGAMATION@
LINK_TOOLS_DYNAMICALLY ?= @LINK_TOOLS_DYNAMICALLY@
AMALGAMATION_GEN_FLAGS ?= --linemacros=@AMALGAMATION_LINE_MACROS@
EXTRA_SRC ?= @AMALGAMATION_EXTRA_SRC@
STATIC_TCLSQLITE3 = @STATIC_TCLSQLITE3@
STATIC_CLI_SHELL = @STATIC_CLI_SHELL@
#
# CFLAGS for sqlite3$(T.exe)
#
SHELL_OPT ?= @OPT_SHELL@
Makefile: $(TOP)/Makefile.in $(AS_AUTO_DEF)
$(AS_AUTORECONFIG)
@touch $@
sqlite3.pc: $(TOP)/sqlite3.pc.in $(AS_AUTO_DEF)
$(AS_AUTORECONFIG)
@touch $@
install: install-pc # defined in main.mk
sqlite_cfg.h: $(AS_AUTO_DEF)
$(AS_AUTORECONFIG)
@touch $@
#
# Fiddle app
#
# EMCC_WRAPPER must refer to the genuine emcc binary, or a
# call-compatible wrapper, e.g. $(TOP)/tool/emcc.sh. If it's empty,
# build components requiring Emscripten will not build.
#
# Achtung: though _this_ makefile is POSIX-make compatible, the fiddle
# build requires GNU make.
#
EMCC_WRAPPER = @EMCC_WRAPPER@
fiddle: sqlite3.c shell.c
@if [ x = "x$(EMCC_WRAPPER)" ]; then \
echo "Emscripten SDK not found by configure. Cannot build fiddle." 1&>2; \
exit 1; \
fi
$(MAKE) -C ext/wasm fiddle emcc_opt=-Os
#
# Spell-checking for source comments
# The sources checked are either C sources or C source templates.
# Their comments are extracted and processed through aspell using
# a custom dictionary that contains scads of odd identifiers that
# find their way into the comments.
#
# Currently, this target is setup to be "made" in-tree only.
# The output is ephemeral. Redirect it to guide spelling fixups,
# either to correct spelling or add words to tool/custom.txt.
#
./custom.rws: ./tool/custom.txt
@echo 'Updating custom dictionary from tool/custom.txt'
aspell --lang=en create master ./custom.rws < ./tool/custom.txt
# Note that jimsh does not work here:
# https://github.com/msteveb/jimtcl/issues/319
misspell: ./custom.rws has_tclsh84
$(TCLSH_CMD) ./tool/spellsift.tcl ./src/*.c ./src/*.h ./src/*.in
#
# clean/distclean are mostly defined in main.mk. In this makefile we
# perform cleanup known to be relevant to (only) the autosetup-driven
# build.
#
distclean-autosetup:
rm -f sqlite_cfg.h config.log config.status config.defines.* Makefile sqlite3.pc
rm -f $(TOP)/tool/emcc.sh
rm -f libsqlite3*$(T.dll)
rm -f jimsh0*
distclean: distclean-autosetup
#
# tool/version-info: a utility for emitting sqlite3 version info
# in various forms.
#
version-info$(T.exe): $(TOP)/tool/version-info.c Makefile sqlite3.h
$(T.link) $(ST_OPT) -o $@ $(TOP)/tool/version-info.c
include $(TOP)/main.mk
Makefile.linux-generic
+64
View File
@@ -0,0 +1,64 @@
#!/usr/make
all:
#
# Makefile for SQLITE
#
# This is a template makefile for SQLite. Most people prefer to
# use the autoconf generated "configure" script to generate the
# makefile automatically. But that does not work for everybody
# and in every situation. If you are having problems with the
# "configure" script, you might want to try this makefile as an
# alternative. Create a copy of this file, edit the parameters
# below and type "make".
#
# Maintenance note: because this is the template for Linux systems, it
# is assumed that the platform has GNU make and this file takes
# advantage of that.
#
####
#
# $(TOP) = The toplevel directory of the source tree. This is the
# directory that contains "Makefile.in" and "auto.def".
#
TOP ?= $(realpath $(dir $(lastword $(MAKEFILE_LIST))))
#
# $(CFLAGS) will be used when compiling the library and most
# utilities. It must normally contain -fPIC on Linux systems,
# but overriding CFLAGS is an easy way for users to inadvertently
# remove -fPIC from their builds, so we generally expect to see
# -fPIC in $(CFLAGS.core), which main.mk will integrate with
# the CFLAGS where needed.
#
CFLAGS =
CFLAGS.core = -fPIC
#
# $(SHELL_OPT) contains CFLAGS for building the sqlite3 CLI shell.
# See main.mk for other potentially-relevant vars which may need
# tweaking, like $(LDFLAGS_READLINE).
#
SHELL_OPT += -DHAVE_READLINE=1
SHELL_OPT += -DSQLITE_HAVE_ZLIB=1
LDFLAGS.readline = -lreadline # may need -lcurses etc, depending on the system
CFLAGS.readline = # needs -I... if readline.h is in an unusual place.
LDFLAGS.zlib = -lz
#
# Library's version number.
#
PACKAGE_VERSION ?= $(shell cat $(TOP)/VERSION 2>/dev/null)
# sqlite_cfg.h is typically created by the configure script. It's
# commonly not needed but main.mk does not know that so we have to
# create a dummy if we don't already have one.
sqlite_cfg.h:
touch $@
distclean-.:
rm -f sqlite_cfg.h
#
# With the above in place, we can now import the rules make use of
# it...
#
include $(TOP)/main.mk
Makefile.msc
+2869
View File
File diff suppressed because it is too large. Load diff
README.md
+1 -1
View File
@@ -429,4 +429,4 @@ with geographically distributed backups at
Contact the SQLite developers through the
[SQLite Forum](https://sqlite.org/forum/). In an emergency, you
can send private email to the lead developer at drh at sqlite dot org.
can send private email to the lead developer at drh at sqlite dot org.
VERSION
+1
View File
@@ -0,0 +1 @@
3.50.2
art/icon-243x273.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

art/icon-80x90.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.3 KiB

art/sqlite370.eps
+5333
View File
File diff suppressed because it is too large. Load diff
art/sqlite370.ico
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.2 KiB

art/sqlite370.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 79 KiB

art/sqlite370.svg
+104
View File
@@ -0,0 +1,104 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
width="103.73447mm"
height="46.723713mm"
viewBox="0 0 103.73447 46.723713"
version="1.1"
id="svg470"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg">
<defs
id="defs467">
<clipPath
clipPathUnits="userSpaceOnUse"
id="clipPath41">
<path
d="m 994.055,997.609 c 6.725,-11.629 35.115,-61.371 40.815,-77.398 6.42,-18.121 7.77,-23.313 7.77,-23.313 0,0 -15.57,80.114 -41.12,126.862 5.6,18.9 12.29,39.79 19.86,62.12 8.85,-15.53 28.96,-51.2 36.23,-68.1 0.27,3.19 0.54,6.38 0.82,9.54 -6.44,24.75 -16.22,57.15 -28.74,82.54 32.14,167.25 135.59,386.25 247.04,504.48 H 509.602 c -37.883,0 -68.711,-30.82 -68.711,-68.71 V 731.789 c 173.738,66.68 383.23,127.633 563.529,125.02 -6.693,25.812 -14.315,49.152 -22.318,62.679 -4.122,6.981 0.449,35.653 11.953,78.121 z"
id="path39" />
</clipPath>
<linearGradient
x1="0"
y1="0"
x2="1"
y2="0"
gradientUnits="userSpaceOnUse"
gradientTransform="matrix(-4.02e-5,-918.907,-918.907,4.02e-5,858.809,1614.34)"
spreadMethod="pad"
id="linearGradient51">
<stop
style="stop-opacity:1;stop-color:#97d9f6"
offset="0"
id="stop43" />
<stop
style="stop-opacity:1;stop-color:#0f80cc"
offset="0.66199914"
id="stop45" />
<stop
style="stop-opacity:1;stop-color:#0f80cc"
offset="0.920245"
id="stop47" />
<stop
style="stop-opacity:1;stop-color:#0f80cc"
offset="1"
id="stop49" />
</linearGradient>
</defs>
<g
id="g337">
<rect
style="fill:#ffffff;stroke:#ffffff;stroke-width:0.239562;stroke-linecap:round;stroke-linejoin:round"
id="rect366"
width="103.4949"
height="46.48415"
x="0.119781"
y="0.119781"
ry="0" />
<g
id="g465"
transform="translate(-56.57816,-61.353828)">
<path
d="m 79.198545,96.729705 c -0.02011,-0.25467 -0.0321,-0.420158 -0.0321,-0.420158 0,0 -0.77223,-5.206224 -1.691499,-6.759822 -0.145415,-0.246275 0.01584,-1.257759 0.421676,-2.755936 0.237243,0.410245 1.238779,2.165033 1.439862,2.73043 0.226484,0.639268 0.274109,0.822431 0.274109,0.822431 0,0 -0.549275,-2.826244 -1.450623,-4.47541 0.197556,-0.66675 0.433564,-1.403703 0.700617,-2.191456 0.341489,0.599017 1.158875,2.049286 1.338792,2.552348 0.03634,0.103364 0.06703,0.191205 0.09454,0.271639 0.0088,-0.04833 0.01764,-0.09666 0.02646,-0.144992 -0.206375,-0.875595 -0.611716,-2.399242 -1.166636,-3.524956 1.231195,-6.408209 5.430309,-14.973654 9.735256,-18.795296 H 60.806192 c -1.88214,0 -3.422086,1.540228 -3.422086,3.422298 v 30.970081 c 0,1.881964 1.539946,3.421944 3.422086,3.421944 h 18.49607 c -0.13335,-1.61438 -0.1778,-3.40046 -0.103717,-5.123145"
style="fill:#0f80cc;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352778"
id="path21" />
<path
d="m 99.934471,84.584659 h 0.474839 v 3.954957 h -0.503061 c -0.115005,-1.093188 -0.510469,-1.946593 -1.186392,-2.560426 -0.676627,-0.612775 -1.526822,-0.920045 -2.5527,-0.920045 -0.911578,0 -1.64465,0.230364 -2.200628,0.690387 -0.555977,0.459669 -0.833966,1.073467 -0.833966,1.840759 0,0.709401 0.230011,1.356536 0.690386,1.940948 0.459669,0.585258 1.457325,1.337134 2.991556,2.258907 1.782939,1.063978 2.981325,1.992912 3.594805,2.789414 0.61348,0.795549 0.92004,1.706421 0.92004,2.732229 0,1.322916 -0.46813,2.454275 -1.405815,3.393691 -0.939094,0.93983 -2.069042,1.40945 -3.389489,1.40945 -0.766233,0 -1.742369,-0.21096 -2.929467,-0.63334 -0.487891,-0.17237 -0.799394,-0.25781 -0.933097,-0.25781 -0.277989,0 -0.492831,0.21523 -0.646289,0.64699 h -0.503414 v -4.558981 h 0.503414 c 0.0092,1.256206 0.418747,2.262752 1.227314,3.020521 0.808567,0.75667 1.753306,1.13524 2.834217,1.13524 1.024819,0 1.844322,-0.27231 2.462389,-0.81979 0.617008,-0.54723 0.925689,-1.276066 0.925689,-2.188068 0,-0.681143 -0.254,-1.333394 -0.760942,-1.956823 -0.506589,-0.623958 -1.567392,-1.427621 -3.180292,-2.412259 -1.31445,-0.812765 -2.192161,-1.452422 -2.63278,-1.923027 -0.44062,-0.470182 -0.766234,-0.985591 -0.976842,-1.547284 -0.210608,-0.560564 -0.316089,-1.177537 -0.316089,-1.848485 0,-1.294659 0.424744,-2.345302 1.273175,-3.151399 0.848431,-0.805392 1.953331,-1.208264 3.314347,-1.208264 0.747889,0 1.563159,0.148519 2.445103,0.445558 0.411692,0.134409 0.699559,0.201436 0.862189,0.201436 0.135114,0 0.225778,-0.02858 0.27305,-0.08608 0.04868,-0.05786 0.1016,-0.186973 0.15875,-0.388409"
style="fill:#003b57;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352778"
id="path23" />
<path
d="m 119.14851,93.38576 c 0,-2.579547 -0.59019,-4.611476 -1.76953,-6.097835 -1.17969,-1.48583 -2.70369,-2.22878 -4.57412,-2.22878 -1.98437,0 -3.56799,0.746125 -4.75191,2.235977 -1.18463,1.491438 -1.77553,3.482834 -1.77553,5.975562 0,2.42577 0.58702,4.398857 1.76106,5.918271 1.17511,1.519415 2.70616,2.278595 4.59458,2.278595 1.96568,0 3.5426,-0.73562 4.7311,-2.207334 1.18851,-1.472177 1.78435,-3.430094 1.78435,-5.874456 z m 2.93265,12.9716 0.18662,0.46041 c -0.82409,0.37402 -1.7586,0.56085 -2.80353,0.56085 -0.86289,0 -1.66405,-0.11314 -2.40206,-0.33789 -0.73731,-0.22571 -1.43122,-0.57051 -2.07751,-1.03572 -0.647,-0.46496 -1.43616,-1.23941 -2.36608,-2.32198 -0.82515,-0.95928 -1.45768,-1.55445 -1.8983,-1.78404 -1.92723,-0.39232 -3.58246,-1.40973 -4.96852,-3.048067 -1.38536,-1.639323 -2.07822,-3.509716 -2.07822,-5.609167 0,-1.5627 0.39794,-3.033889 1.19345,-4.414556 0.79587,-1.380772 1.87466,-2.461718 3.23709,-3.242416 1.36031,-0.782108 2.87055,-1.172633 4.52896,-1.172633 2.56011,0 4.71699,0.851253 6.47206,2.55263 1.75366,1.7012 2.63067,3.79416 2.63067,6.276975 0,2.09102 -0.67522,3.951923 -2.02706,5.586695 -1.3522,1.635479 -3.08293,2.658779 -5.19148,3.070539 0.20179,0.16151 0.56233,0.5697 1.0795,1.22234 1.10031,1.36204 2.02071,2.28604 2.75837,2.76998 0.73871,0.48274 1.47214,0.72542 2.20133,0.72542 0.53728,0 1.04634,-0.0868 1.52471,-0.25937"
style="fill:#003b57;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352778"
id="path25" />
<path
d="m 134.47177,99.116318 -0.28116,2.694482 h -11.00385 l 0.0233,-0.53619 c 0.59654,-0.061 1.01458,-0.21413 1.25835,-0.45861 0.24342,-0.24624 0.36654,-0.81111 0.37571,-1.699682 l 0.0127,-11.904593 c -0.004,-0.621877 -0.0346,-1.050502 -0.0861,-1.284393 -0.13828,-0.653698 -0.75565,-0.980017 -1.85984,-0.980017 l -0.0236,-0.535164 h 5.74393 l 0.0219,0.535164 c -0.61242,0.04163 -1.03152,0.206728 -1.26612,0.496711 -0.23212,0.291042 -0.34713,0.880533 -0.34678,1.767699 l -0.0441,12.029053 c 0.002,0.204646 0.0191,0.391901 0.0572,0.565538 0.1845,0.877674 0.88053,1.315474 2.08774,1.315474 h 1.03681 c 1.34161,0 2.27259,-0.15088 2.79788,-0.45226 0.52388,-0.29919 0.8569,-0.818023 1.00048,-1.553212 h 0.49565"
style="fill:#003b57;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352778"
id="path27" />
<path
d="m 138.12055,87.618936 c 0.25965,0 0.49883,0.09095 0.71967,0.273685 0.21837,0.182034 0.36054,0.407071 0.41522,0.674829 0.0533,0.25975 0.009,0.48507 -0.133,0.676769 -0.14217,0.191523 -0.34114,0.28702 -0.60078,0.28702 -0.26846,0 -0.51858,-0.0955 -0.75001,-0.28702 -0.23283,-0.191699 -0.37535,-0.417019 -0.42827,-0.676769 -0.0547,-0.267758 -0.0106,-0.492795 0.14605,-0.674829 0.15311,-0.182738 0.36266,-0.273685 0.63112,-0.273685 z m 1.75754,4.127783 -0.0176,7.535404 c 0.003,0.526662 0.03,0.900957 0.0769,1.122117 0.0607,0.30688 0.19685,0.5147 0.40111,0.6248 0.20778,0.11081 0.67133,0.1662 1.39065,0.1662 l 0.0166,0.50299 h -5.17702 l 0.0123,-0.50299 c 0.70979,0 1.14582,-0.0656 1.31198,-0.1947 0.16475,-0.12887 0.24941,-0.56307 0.25859,-1.301433 l 0.0325,-5.335094 c -0.006,-0.291042 -0.0243,-0.518548 -0.0586,-0.683613 -0.1016,-0.493219 -0.42686,-0.740587 -0.97367,-0.740587 l -1.12324,0.143616 -0.005,-0.589527 3.26319,-0.747183 h 0.59161"
style="fill:#003b57;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352778"
id="path29" />
<path
d="m 148.56207,92.004952 0.0247,0.805074 h -2.70545 l -0.0194,6.728955 c 0,0.183409 0.0109,0.341065 0.0423,0.476109 0.11465,0.55605 0.41945,0.83467 0.90946,0.83467 0.79728,0 1.70709,-0.52239 2.73967,-1.567637 l 0.43427,0.366148 c -1.43016,1.587219 -2.79682,2.380119 -4.10033,2.380119 -0.901,0 -1.43828,-0.41229 -1.6129,-1.23677 -0.072,-0.34438 -0.10795,-1.025556 -0.11077,-2.041838 l 0.037,-5.939756 h -1.76954 l -0.0205,-0.502708 c 1.51836,-0.153105 2.50472,-1.106981 2.96087,-2.861769 h 0.51752 l -0.005,2.559403 h 2.67758"
style="fill:#003b57;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352778"
id="path31" />
<path
d="m 152.6931,96.836244 c 2.82998,-0.546417 4.09434,-1.527845 3.79906,-2.9464 -0.0603,-0.288819 -0.24483,-0.55245 -0.54716,-0.791527 -0.30656,-0.239925 -0.61418,-0.359975 -0.92921,-0.359975 -0.71932,0 -1.3081,0.424462 -1.76178,1.272505 -0.4572,0.848854 -0.6417,1.791017 -0.56091,2.825397 z m -0.0162,0.533436 c 0.024,0.248708 0.0575,0.48394 0.10442,0.704179 0.36089,1.725437 1.3328,2.588371 2.91395,2.588371 1.15005,0 2.67264,-0.694412 3.42829,-2.08471 l 0.38312,0.395217 c -0.47378,1.071703 -0.9899,1.848523 -1.54658,2.331513 -0.55527,0.48327 -1.87678,0.72414 -2.82434,0.72414 -1.01776,0 -1.86161,-0.28497 -2.53259,-0.85548 -0.67204,-0.57076 -1.11407,-1.378306 -1.33456,-2.423128 -0.37465,-1.783751 -0.0843,-3.405012 0.86854,-4.868334 0.95779,-1.460853 2.19145,-2.19329 3.70593,-2.19329 0.59584,0 1.66511,0.152118 2.07574,0.453637 0.40535,0.30166 0.66393,0.707072 0.77118,1.214614 0.18027,0.85411 -0.78352,1.660948 -1.74484,2.423689 -0.96203,0.762318 -2.38725,1.291767 -4.26826,1.589582"
style="fill:#003b57;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352778"
id="path33" />
<g
id="g37"
clip-path="url(#clipPath41)"
transform="matrix(0.03527778,0,0,-0.03527778,42.828565,121.98722)">
<path
d="m 994.055,997.609 c 6.725,-11.629 35.115,-61.371 40.815,-77.398 6.42,-18.121 7.77,-23.313 7.77,-23.313 0,0 -15.57,80.114 -41.12,126.862 5.6,18.9 12.29,39.79 19.86,62.12 8.85,-15.53 28.96,-51.2 36.23,-68.1 0.27,3.19 0.54,6.38 0.82,9.54 -6.44,24.75 -16.22,57.15 -28.74,82.54 32.14,167.25 135.59,386.25 247.04,504.48 H 509.602 c -37.883,0 -68.711,-30.82 -68.711,-68.71 V 731.789 c 173.738,66.68 383.23,127.633 563.529,125.02 -6.693,25.812 -14.315,49.152 -22.318,62.679 -4.122,6.981 0.449,35.653 11.953,78.121"
style="fill:url(#linearGradient51);fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path53" />
</g>
<path
d="m 95.439377,63.024997 c -1.923345,-1.715206 -4.252031,-1.026231 -6.550379,1.01353 -0.341136,0.303037 -0.681566,0.639234 -1.020233,0.998362 -3.931709,4.170892 -7.581195,11.896725 -8.715023,17.796934 0.441678,0.895703 0.786695,2.038703 1.013884,2.911828 0.05821,0.224014 0.110772,0.43427 0.152752,0.613128 0.09984,0.423298 0.153459,0.697794 0.153459,0.697794 0,0 -0.03528,-0.133385 -0.179917,-0.552802 -0.02752,-0.08043 -0.05821,-0.168275 -0.09454,-0.271639 -0.01552,-0.04269 -0.03704,-0.09454 -0.06068,-0.149931 -0.25647,-0.596194 -0.965906,-1.854553 -1.278114,-2.402417 -0.267053,0.787753 -0.503061,1.524706 -0.700617,2.191456 0.901348,1.649166 1.450623,4.47541 1.450623,4.47541 0,0 -0.04762,-0.183163 -0.274109,-0.822431 -0.201083,-0.565397 -1.202619,-2.320185 -1.439862,-2.73043 -0.405836,1.498177 -0.567091,2.509661 -0.421676,2.755936 0.282328,0.477202 0.551216,1.300586 0.78733,2.211176 0.5334,2.051367 0.904169,4.548646 0.904169,4.548646 0,0 0.01199,0.165488 0.0321,0.420158 -0.07408,1.722685 -0.02963,3.508765 0.103717,5.123145 0.176741,2.13706 0.509411,3.97288 0.93345,4.9554 l 0.287866,-0.15695 c -0.622652,-1.93573 -0.875594,-4.47255 -0.764822,-7.39814 0.16757,-4.471846 1.196623,-9.86469 3.098095,-15.485534 3.212395,-8.485012 7.669389,-15.292918 11.748559,-18.544118 -3.717925,3.357739 -8.749948,14.22647 -10.256309,18.251312 -1.686631,4.50709 -2.881842,8.736578 -3.602214,12.788831 1.242836,-3.798959 5.261328,-5.431932 5.261328,-5.431932 0,0 1.97097,-2.430745 4.274256,-5.903454 -1.379714,0.314678 -3.645253,0.853369 -4.404078,1.172281 -1.119364,0.469547 -1.420989,0.629708 -1.420989,0.629708 0,0 3.62585,-2.208036 6.736645,-3.207809 4.278136,-6.738056 8.939042,-16.310328 4.245328,-20.497448"
style="fill:#003b57;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352778"
id="path55" />
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

auto.def
+64
View File
@@ -0,0 +1,64 @@
#!/do/not/tclsh
# ^^^ help out editors which guess this file's content type.
#
# This is the main autosetup-compatible configure script for the
# SQLite project.
#
# This script and all of its dependencies must be kept compatible with
# JimTCL, a copy of which is included in this source tree as
# ./autosetup/jimsh0.c. The number of incompatibilities between
# canonical TCL and JimTCL is very low and alternative formulations of
# incompatible constructs have, so far, been easy to find.
#
# JimTCL: https://jim.tcl.tk
#
use sqlite-config
sqlite-configure canonical {
proj-if-opt-truthy dev {
# --enable-dev needs to come early so that the downstream tests
# which check for the following flags use their updated state.
proj-opt-set all 1
proj-opt-set debug 1
proj-opt-set amalgamation 0
define CFLAGS [get-env CFLAGS {-O0 -g}]
# -------------^^^^^^^ intentionally using [get-env] instead of
# [proj-get-env] here because [sqlite-setup-default-cflags] uses
# [proj-get-env] and we want this to supercede that.
sqlite-munge-cflags; # straighten out -DSQLITE_ENABLE/OMIT flags
}
sqlite-handle-debug ;# must come after --dev flag check
sqlite-check-common-bins ;# must come before [sqlite-handle-wasi-sdk]
sqlite-handle-wasi-sdk ;# must run relatively early, as it changes the environment
sqlite-check-common-system-deps
proj-define-for-opt amalgamation USE_AMALGAMATION "Use amalgamation for builds?"
proj-define-for-opt gcov USE_GCOV "Use gcov?"
proj-define-for-opt test-status TSTRNNR_OPTS \
"test-runner flags:" {--status} {}
proj-define-for-opt linemacros AMALGAMATION_LINE_MACROS \
"Use #line macros in the amalgamation:"
define AMALGAMATION_EXTRA_SRC \
[join [opt-val amalgamation-extra-src ""] " "]
define LINK_TOOLS_DYNAMICALLY [proj-opt-was-provided dynlink-tools]
if {[set fsan [join [opt-val asan-fsanitize] ","]] in {auto ""}} {
set fsan address,bounds-strict
}
define CFLAGS_ASAN_FSANITIZE [proj-check-fsanitize [split $fsan ", "]]
sqlite-handle-tcl
sqlite-handle-emsdk
proj-if-opt-truthy static-shells {
proj-opt-set static-tclsqlite3 1
proj-opt-set static-cli-shell 1
}
proj-define-for-opt static-tclsqlite3 STATIC_TCLSQLITE3 "Statically link tclsqlite3?"
proj-define-for-opt static-cli-shell STATIC_CLI_SHELL "Statically link CLI shell?"
}; # sqlite-configure
autoconf/Makefile.fallback
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/make
#
# If the configure script does not work, then this Makefile is available
# as a backup. Manually configure the variables below.
#
# Note: This makefile works out-of-the-box on MacOS 10.2 (Jaguar)
#
CC = gcc
CFLAGS = -O0 -I.
LIBS = -lz
COPTS += -D_BSD_SOURCE
COPTS += -DSQLITE_ENABLE_LOCKING_STYLE=0
COPTS += -DSQLITE_THREADSAFE=0
COPTS += -DSQLITE_OMIT_LOAD_EXTENSION
COPTS += -DSQLITE_WITHOUT_ZONEMALLOC
COPTS += -DSQLITE_ENABLE_RTREE
sqlite3: shell.c sqlite3.c
$(CC) $(CFLAGS) $(COPTS) -o sqlite3 shell.c sqlite3.c $(LIBS)
autoconf/Makefile.in
+298
View File
@@ -0,0 +1,298 @@
########################################################################
# This is a main makefile for the "autoconf" bundle of SQLite. This is
# a trimmed-down version of the canonical makefile, devoid of most
# documentation. For the full docs, see /main.mk in the canonical
# source tree.
#
# Maintenance reminders:
#
# - To keep this working with an out-of-tree build, be sure to prefix
# input file names with $(TOP)/ where appropriate (which is most
# places).
#
# - The original/canonical recipes can be found in /main.mk in the
# canonical source tree.
all:
TOP = @abs_top_srcdir@
PACKAGE_VERSION = @PACKAGE_VERSION@
#
# Filename extensions for binaries and libraries
#
B.exe = @BUILD_EXEEXT@
T.exe = @TARGET_EXEEXT@
B.dll = @BUILD_DLLEXT@
T.dll = @TARGET_DLLEXT@
B.lib = @BUILD_LIBEXT@
T.lib = @TARGET_LIBEXT@
#
# Autotools-compatibility dirs
#
prefix = @prefix@
datadir = @datadir@
mandir = @mandir@
includedir = @includedir@
exec_prefix = @exec_prefix@
bindir = @bindir@
libdir = @libdir@
#
# Required binaries
#
INSTALL = @BIN_INSTALL@
AR = @AR@
AR.flags = cr
CC = @CC@
ENABLE_LIB_SHARED = @ENABLE_LIB_SHARED@
ENABLE_LIB_STATIC = @ENABLE_LIB_STATIC@
HAVE_WASI_SDK = @HAVE_WASI_SDK@
CFLAGS = @CFLAGS@ @CPPFLAGS@
#
# $(LDFLAGS.configure) represents any LDFLAGS=... the client passes to
# configure. See main.mk.
#
LDFLAGS.configure = @LDFLAGS@
CFLAGS.core = @SH_CFLAGS@
LDFLAGS.shlib = @SH_LDFLAGS@
LDFLAGS.zlib = @LDFLAGS_ZLIB@
LDFLAGS.math = @LDFLAGS_MATH@
LDFLAGS.rpath = @LDFLAGS_RPATH@
LDFLAGS.pthread = @LDFLAGS_PTHREAD@
LDFLAGS.dlopen = @LDFLAGS_DLOPEN@
LDFLAGS.readline = @LDFLAGS_READLINE@
CFLAGS.readline = @CFLAGS_READLINE@
LDFLAGS.rt = @LDFLAGS_RT@
LDFLAGS.icu = @LDFLAGS_ICU@
CFLAGS.icu = @CFLAGS_ICU@
# INSTALL reminder: we specifically do not strip binaries,
# as discussed in https://sqlite.org/forum/forumpost/9a67df63eda9925c.
INSTALL.noexec = $(INSTALL) -m 0644
install-dir.bin = $(DESTDIR)$(bindir)
install-dir.lib = $(DESTDIR)$(libdir)
install-dir.include = $(DESTDIR)$(includedir)
install-dir.pkgconfig = $(DESTDIR)$(libdir)/pkgconfig
install-dir.man1 = $(DESTDIR)$(mandir)/man1
install-dir.all = $(install-dir.bin) $(install-dir.include) \
$(install-dir.lib) $(install-dir.man1) \
$(install-dir.pkgconfig)
$(install-dir.all):
@if [ ! -d "$@" ]; then set -x; $(INSTALL) -d "$@"; fi
# ^^^^ on some platforms, install -d fails if the target already exists.
#
# Vars with the AS_ prefix are specifically related to AutoSetup.
#
# AS_AUTO_DEF is the main configure script.
#
AS_AUTO_DEF = $(TOP)/auto.def
#
# Shell commands to re-run $(TOP)/configure with the same args it was
# invoked with to produce this makefile.
#
AS_AUTORECONFIG = @SQLITE_AUTORECONFIG@
Makefile: $(TOP)/Makefile.in $(AS_AUTO_DEF)
$(AS_AUTORECONFIG)
@touch $@
sqlite3.pc: $(TOP)/sqlite3.pc.in $(AS_AUTO_DEF)
$(AS_AUTORECONFIG)
@touch $@
sqlite_cfg.h: $(AS_AUTO_DEF)
$(AS_AUTORECONFIG)
@touch $@
#
# CFLAGS for sqlite3$(T.exe)
#
SHELL_OPT ?= @OPT_SHELL@
#
# Library-level feature flags
#
OPT_FEATURE_FLAGS = @OPT_FEATURE_FLAGS@
LDFLAGS.libsqlite3.soname = @LDFLAGS_LIBSQLITE3_SONAME@
# soname: see https://sqlite.org/src/forumpost/5a3b44f510df8ded
LDFLAGS.libsqlite3.os-specific = \
@LDFLAGS_MAC_CVERSION@ @LDFLAGS_MAC_INSTALL_NAME@ @LDFLAGS_OUT_IMPLIB@
LDFLAGS.libsqlite3 = \
$(LDFLAGS.rpath) $(LDFLAGS.pthread) \
$(LDFLAGS.math) $(LDFLAGS.dlopen) \
$(LDFLAGS.zlib) $(LDFLAGS.icu) \
$(LDFLAGS.rt) $(LDFLAGS.configure)
CFLAGS.libsqlite3 = -I. $(CFLAGS.core) $(CFLAGS.icu) $(OPT_FEATURE_FLAGS)
sqlite3.o: $(TOP)/sqlite3.h $(TOP)/sqlite3.c
$(CC) -c $(TOP)/sqlite3.c -o $@ $(CFLAGS) $(CFLAGS.libsqlite3)
libsqlite3.LIB = libsqlite3$(T.lib)
libsqlite3.DLL.basename = @SQLITE_DLL_BASENAME@
libsqlite3.out.implib = @SQLITE_OUT_IMPLIB@
libsqlite3.DLL = $(libsqlite3.DLL.basename)$(T.dll)
libsqlite3.DLL.install-rules = @SQLITE_DLL_INSTALL_RULES@
$(libsqlite3.DLL): sqlite3.o
$(CC) -o $@ sqlite3.o $(LDFLAGS.shlib) \
$(LDFLAGS) $(LDFLAGS.libsqlite3) \
$(LDFLAGS.libsqlite3.os-specific) $(LDFLAGS.libsqlite3.soname)
$(libsqlite3.DLL)-1: $(libsqlite3.DLL)
$(libsqlite3.DLL)-0:
all: $(libsqlite3.DLL)-$(ENABLE_LIB_SHARED)
$(libsqlite3.LIB): sqlite3.o
$(AR) $(AR.flags) $@ sqlite3.o
$(libsqlite3.LIB)-1: $(libsqlite3.LIB)
$(libsqlite3.LIB)-0:
all: $(libsqlite3.LIB)-$(ENABLE_LIB_STATIC)
#
# Maintenance reminder: the install-dll-... rules must be kept in sync
# with the main copies rom /main.mk.
#
install-dll-out-implib: $(install-dir.lib) $(libsqlite3.DLL)
if [ x != "x$(libsqlite3.out.implib)" ] && [ -f "$(libsqlite3.out.implib)" ]; then \
$(INSTALL) $(libsqlite3.out.implib) "$(install-dir.lib)"; \
fi
install-dll-unix-generic: install-dll-out-implib
$(INSTALL) $(libsqlite3.DLL) "$(install-dir.lib)"
@echo "Setting up $(libsqlite3.DLL) version symlinks..."; \
cd "$(install-dir.lib)" || exit $$?; \
rm -f $(libsqlite3.DLL).0 $(libsqlite3.DLL).$(PACKAGE_VERSION) || exit $$?; \
mv $(libsqlite3.DLL) $(libsqlite3.DLL).$(PACKAGE_VERSION) || exit $$?; \
ln -s $(libsqlite3.DLL).$(PACKAGE_VERSION) $(libsqlite3.DLL) || exit $$?; \
ln -s $(libsqlite3.DLL).$(PACKAGE_VERSION) $(libsqlite3.DLL).0 || exit $$?; \
ls -la $(libsqlite3.DLL) $(libsqlite3.DLL).[a03]*; \
if [ -e $(libsqlite3.DLL).0.8.6 ]; then \
echo "ACHTUNG: legacy libtool-compatible install found. Re-linking it..."; \
rm -f libsqlite3.la $(libsqlite3.DLL).0.8.6 || exit $$?; \
ln -s $(libsqlite3.DLL).$(PACKAGE_VERSION) $(libsqlite3.DLL).0.8.6 || exit $$?; \
ls -la $(libsqlite3.DLL).0.8.6; \
elif [ x1 = "x$(INSTALL_SO_086_LINK)" ]; then \
echo "ACHTUNG: installing legacy libtool-style links because INSTALL_SO_086_LINK=1"; \
rm -f libsqlite3.la $(libsqlite3.DLL).0.8.6 || exit $$?; \
ln -s $(libsqlite3.DLL).$(PACKAGE_VERSION) $(libsqlite3.DLL).0.8.6 || exit $$?; \
ls -la $(libsqlite3.DLL).0.8.6; \
fi
install-dll-msys: install-dll-out-implib $(install-dir.bin)
$(INSTALL) $(libsqlite3.DLL) "$(install-dir.bin)"
# ----------------------------------------------^^^ yes, bin
# Each of {msys,mingw,cygwin} uses a different name for the DLL, but
# that is already accounted for via $(libsqlite3.DLL).
install-dll-mingw: install-dll-msys
install-dll-cygwin: install-dll-msys
install-dll-darwin: $(install-dir.lib) $(libsqlite3.DLL)
$(INSTALL) $(libsqlite3.DLL) "$(install-dir.lib)"
@echo "Setting up $(libsqlite3.DLL) version symlinks..."; \
cd "$(install-dir.lib)" || exit $$?; \
rm -f libsqlite3.0$(T.dll) libsqlite3.$(PACKAGE_VERSION)$(T.dll) || exit $$?; \
dllname=libsqlite3.$(PACKAGE_VERSION)$(T.dll); \
mv $(libsqlite3.DLL) $$dllname || exit $$?; \
ln -s $$dllname $(libsqlite3.DLL) || exit $$?; \
ln -s $$dllname libsqlite3.0$(T.dll) || exit $$?; \
ls -la $$dllname $(libsqlite3.DLL) libsqlite3.0$(T.dll)
install-dll-1: install-dll-$(libsqlite3.DLL.install-rules)
install-dll-0 install-dll-:
install-dll: install-dll-$(ENABLE_LIB_SHARED)
install: install-dll
install-lib-1: $(install-dir.lib) $(libsqlite3.LIB)
$(INSTALL.noexec) $(libsqlite3.LIB) "$(install-dir.lib)"
install-lib-0 install-lib-:
install-lib: install-lib-$(ENABLE_LIB_STATIC)
install: install-lib
#
# Flags to link the shell app either directly against sqlite3.c
# (ENABLE_STATIC_SHELL==1) or libsqlite3.so (ENABLE_STATIC_SHELL==0).
#
ENABLE_STATIC_SHELL = @ENABLE_STATIC_SHELL@
sqlite3-shell-link-flags.1 = $(TOP)/sqlite3.c $(LDFLAGS.libsqlite3)
sqlite3-shell-link-flags.0 = -L. -lsqlite3 $(LDFLAGS.zlib) $(LDFLAGS.math)
sqlite3-shell-deps.1 = $(TOP)/sqlite3.c
sqlite3-shell-deps.0 = $(libsqlite3.DLL)
#
# STATIC_CLI_SHELL = 1 to statically link sqlite3$(T.exe), else
# 0. Requires static versions of all requisite libraries. Primarily
# intended for use with static-friendly environments like Alpine
# Linux.
#
STATIC_CLI_SHELL = @STATIC_CLI_SHELL@
#
# sqlite3-shell-static.flags.N = N is $(STATIC_CLI_SHELL)
#
sqlite3-shell-static.flags.1 = -static
sqlite3-shell-static.flags.0 =
sqlite3$(T.exe): $(TOP)/shell.c $(sqlite3-shell-deps.$(ENABLE_STATIC_SHELL))
$(CC) -o $@ \
$(TOP)/shell.c $(sqlite3-shell-link-flags.$(ENABLE_STATIC_SHELL)) \
$(sqlite3-shell-static.flags.$(STATIC_CLI_SHELL)) \
-I. $(OPT_FEATURE_FLAGS) $(SHELL_OPT) \
$(CFLAGS) $(CFLAGS.readline) $(CFLAGS.icu) \
$(LDFLAGS) $(LDFLAGS.readline)
sqlite3$(T.exe)-1:
sqlite3$(T.exe)-0: sqlite3$(T.exe)
all: sqlite3$(T.exe)-$(HAVE_WASI_SDK)
install-shell-0: sqlite3$(T.exe) $(install-dir.bin)
$(INSTALL) sqlite3$(T.exe) "$(install-dir.bin)"
install-shell-1:
install: install-shell-$(HAVE_WASI_SDK)
install-headers: $(TOP)/sqlite3.h $(install-dir.include)
$(INSTALL.noexec) $(TOP)/sqlite3.h $(TOP)/sqlite3ext.h "$(install-dir.include)"
install: install-headers
install-pc: sqlite3.pc $(install-dir.pkgconfig)
$(INSTALL.noexec) sqlite3.pc "$(install-dir.pkgconfig)"
install: install-pc
install-man1: $(TOP)/sqlite3.1 $(install-dir.man1)
$(INSTALL.noexec) $(TOP)/sqlite3.1 "$(install-dir.man1)"
install: install-man1
clean:
rm -f *.o sqlite3$(T.exe)
rm -f $(libsqlite3.LIB) $(libsqlite3.DLL) libsqlite3$(T.dll).a
distclean: clean
rm -f jimsh0$(T.exe) config.* sqlite3.pc sqlite_cfg.h Makefile
DIST_FILES := \
README.txt VERSION \
auto.def autosetup configure tea \
sqlite3.h sqlite3.c shell.c sqlite3ext.h \
Makefile.in Makefile.msc Makefile.fallback \
sqlite3.rc sqlite3rc.h Replace.cs \
sqlite3.pc.in sqlite3.1
#
# Maintenance note: dist_name must be sqlite-$(PACKAGE_VERSION) so
# that tool/mkautoconfamal.sh knows how to find it.
#
dist_name = sqlite-$(PACKAGE_VERSION)
dist_tarball = $(dist_name).tar.gz
dist:
rm -fr $(dist_name)
mkdir -p $(dist_name)
cp -rp $(DIST_FILES) $(dist_name)/.
tar czf $(dist_tarball) $(dist_name)
rm -fr $(dist_name)
ls -l $(dist_tarball)
autoconf/Makefile.msc
+1102
View File
File diff suppressed because it is too large. Load diff
autoconf/README.first
+12
View File
@@ -0,0 +1,12 @@
This directory contains components used to build an autoconf-like
package of the SQLite amalgamation: sqlite-autoconf-30XXXXXX.tar.gz
To build the autoconf amalgamation, run from the top of the canonical
source tree:
./configure
make amalgamation-tarball
The amalgamation-tarball target (available in "main.mk") runs the
script tool/mkautoconfamal.sh which does the work. Refer to that
script for details.
autoconf/README.txt
+101
View File
@@ -0,0 +1,101 @@
This package contains:
* the SQLite library amalgamation source code file: sqlite3.c
* the sqlite3.h and sqlite3ext.h header files that define the C-language
interface to the sqlite3.c library file
* the shell.c file used to build the sqlite3 command-line shell program
* autoconf-like installation infrastucture for building on POSIX
compliant systems
* a Makefile.msc, sqlite3.rc, and Replace.cs for building with Microsoft
Visual C++ on Windows
WHY USE THIS PACKAGE?
=====================
The canonical make system for SQLite requires TCL as part of the build
process. Various TCL scripts are used to generate parts of the code and
TCL is used to run tests. But some people would prefer to build SQLite
using only generic tools and without having to install TCL. The purpose
of this package is to provide that capability.
This package contains a pre-build SQLite amalgamation file "sqlite3.c"
(and its associated header file "sqlite3.h"). Because the
amalgamation has been pre-built, no TCL is required for the code
generate (the configure script itself is written in TCL but it can use
the embedded copy of JimTCL).
REASONS TO USE THE CANONICAL BUILD SYSTEM RATHER THAN THIS PACKAGE
==================================================================
* the canonical build system allows you to run tests to verify that
the build worked
* the canonical build system supports more compile-time options
* the canonical build system works for any arbitrary check-in to
the SQLite source tree
Step-by-step instructions on how to build using the canonical make
system for SQLite can be found at:
https://sqlite.org/src/doc/trunk/doc/compile-for-unix.md
https://sqlite.org/src/doc/trunk/doc/compile-for-windows.md
SUMMARY OF HOW TO BUILD USING THIS PACKAGE
==========================================
Unix: ./configure; make
Windows: nmake /f Makefile.msc
BUILDING ON POSIX
=================
The configure script follows common conventions, making it easy
to use for anyone who has configured a software tree before.
It supports a number of build-time flags, the full list of which
can be seen by running:
./configure --help
The default value for the CFLAGS variable (options passed to the C
compiler) includes debugging symbols in the build, resulting in larger
binaries than are necessary. Override it on the configure command
line like this:
$ CFLAGS="-Os" ./configure
to produce a smaller installation footprint.
Many SQLite compilation parameters can be defined by passing flags
to the configure script. Others may be passed on in the CFLAGS. For
example:
$ CFLAGS="-Os -DSQLITE_OMIT_DEPRECATED" ./configure
BUILDING WITH MICROSOFT VISUAL C++
==================================
To compile for Windows using Microsoft Visual C++:
$ nmake /f Makefile.msc
Using Microsoft Visual C++ 2005 (or later) is recommended. Several Windows
platform variants may be built by adding additional macros to the NMAKE
command line.
Other preprocessor defines
--------------------------
Additionally, preprocessor defines may be specified by using the OPTS macro
on the NMAKE command line. However, not all possible preprocessor defines
may be specified in this manner as some require the amalgamation to be built
with them enabled (see http://sqlite.org/compile.html). For example, the
following will work:
"OPTS=-DSQLITE_ENABLE_STAT4=1 -DSQLITE_OMIT_JSON=1"
However, the following will not compile unless the amalgamation was built
with it enabled:
"OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1"
autoconf/auto.def
+25
View File
@@ -0,0 +1,25 @@
#!/do/not/tclsh
# ^^^ help out editors which guess this file's content type.
#
# This is the main autosetup-compatible configure script for the
# "autoconf" bundle of the SQLite project.
use sqlite-config
sqlite-configure autoconf {
sqlite-handle-debug
sqlite-check-common-bins ;# must come before [sqlite-handle-wasi-sdk]
sqlite-handle-wasi-sdk ;# must run relatively early, as it changes the environment
sqlite-check-common-system-deps
proj-define-for-opt static-shell ENABLE_STATIC_SHELL \
"Link library statically into the CLI shell?"
proj-define-for-opt static-cli-shell STATIC_CLI_SHELL "Statically link CLI shell?"
if {![opt-bool static-shell] && [opt-bool static-cli-shell]} {
proj-fatal "--disable-static-shell and --static-cli-shell are mutualy exclusive"
}
if {![opt-bool shared] && ![opt-bool static-shell]} {
proj-opt-set shared 1
proj-indented-notice {
NOTICE: ignoring --disable-shared because --disable-static-shell
was specified.
}
}
}
autoconf/tea/Makefile.in
+535
View File
@@ -0,0 +1,535 @@
all:
#
# Unless this file is named Makefile.in, you are probably looking
# at an automatically generated/filtered copy and should probably not
# edit it.
#
# This makefile is part of the teaish framework, a tool for building
# Tcl extensions, conceptually related to TEA/tclconfig but using the
# Autosetup configuration system instead of the GNU Autotools.
#
# A copy of this makefile gets processed for each extension separately
# and populated with info about how to build, test, and install the
# extension.
#
# Maintenance reminder: this file needs to stay portable with POSIX
# Make, not just GNU Make. Yes, that's unfortunate because it makes
# some things impossible (like skipping over swathes of rules when
# 'make distclean' is invoked).
#
CC = @CC@
INSTALL = @BIN_INSTALL@
INSTALL.noexec = $(INSTALL) -m 0644
#
# Var name prefixes:
#
# teaish. => teaish core
# tx. => teaish extension
#
# Vars with a "tx." or "teaish." prefix are all "public" for purposes
# of the extension makefile, but the extension must not any "teaish."
# vars and must only modify "tx." vars where that allowance is
# specifically noted.
#
# Vars with a "teaish__" prefix are "private" and must not be used by
# the extension makefile. They may change semantics or be removed in
# any given teaish build.
#
tx.name = @TEAISH_NAME@
tx.version = @TEAISH_VERSION@
tx.name.pkg = @TEAISH_PKGNAME@
tx.libdir = @TEAISH_LIBDIR_NAME@
tx.loadPrefix = @TEAISH_LOAD_PREFIX@
tx.tcl = @TEAISH_TCL@
tx.makefile = @TEAISH_MAKEFILE@
tx.makefile.in = @TEAISH_MAKEFILE_IN@
tx.dll8.basename = @TEAISH_DLL8_BASENAME@
tx.dll9.basename = @TEAISH_DLL9_BASENAME@
tx.dll8 = @TEAISH_DLL8@
tx.dll9 = @TEAISH_DLL9@
tx.dll = $(tx.dll$(TCL_MAJOR_VERSION))
tx.dir = @TEAISH_EXT_DIR@
@if TEAISH_TM_TCL
# Input filename for tcl::tm-style module
tx.tm = @TEAISH_TM_TCL@
# Target filename for tcl::tm-style installation
tx.tm.tgt = $(tx.name.pkg)-$(tx.version).tm
@endif
@if TEAISH_DIST_NAME
tx.name.dist = @TEAISH_DIST_NAME@
@else
tx.name.dist = $(teaish.name)
@endif
teaish.dir = @abs_top_srcdir@
#teaish.dir.autosetup = @TEAISH_AUTOSETUP_DIR@
teaish.makefile = Makefile
teaish.makefile.in = $(teaish.dir)/Makefile.in
teaish__auto.def = $(teaish.dir)/auto.def
#
# Autotools-conventional vars. We don't actually use these in this
# makefile but some may be referenced by vars imported via
# tclConfig.sh. They are part of the public API and may be reliably
# depended on from teaish.make.in.
#
bindir = @bindir@
datadir = @datadir@
exec_prefix = @exec_prefix@
includedir = @includedir@
infodir = @infodir@
libdir = @libdir@
libexecdir = @libexecdir@
localstatedir = @localstatedir@
mandir = @mandir@
prefix = @prefix@
runstatedir = @runstatedir@
sbindir = @sbindir@
sharedstatedir = @sharedstatedir@
sysconfdir = @sysconfdir@
#
# Vars derived (mostly) from tclConfig.sh. These may be reliably
# used from the extension makefile.
#
TCLSH = @TCLSH_CMD@
TCL_CONFIG_SH = @TCL_CONFIG_SH@
TCL_EXEC_PREFIX = @TCL_EXEC_PREFIX@
TCL_INCLUDE_SPEC = @TCL_INCLUDE_SPEC@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
TCL_MAJOR_VERSION = @TCL_MAJOR_VERSION@
TCL_MINOR_VERSION = @TCL_MINOR_VERSION@
TCL_PATCH_LEVEL = @TCL_PATCH_LEVEL@
TCL_PREFIX = @TCL_PREFIX@
TCL_SHLIB_SUFFIX = @TCL_SHLIB_SUFFIX@
TCL_STUB_LIB_SPEC = @TCL_STUB_LIB_SPEC@
TCL_VERSION = @TCL_VERSION@
TCLLIBDIR = @TCLLIBDIR@
#
# CFLAGS.configure = CFLAGS as known at configure-time.
#
# This ordering is deliberate: flags populated via tcl's
# [teaish-cflags-add] should preceed CFLAGS and CPPFLAGS (which
# typically come from the ./configure command-line invocation).
#
CFLAGS.configure = @SH_CFLAGS@ @TEAISH_CFLAGS@ @CFLAGS@ @CPPFLAGS@ $(TCL_INCLUDE_SPEC)
#CFLAGS.configure += -DUSE_TCL_STUBS=1
#
# LDFLAGS.configure = LDFLAGS as known at configure-time.
#
# This ordering is deliberate: flags populated via tcl's
# [teaish-ldflags-add] should precede LDFLAGS (which typically
# comes from the ./configure command-line invocation).
#
LDFLAGS.configure = @TEAISH_LDFLAGS@ @LDFLAGS@
#
# Linker flags for linkhing a shared library.
#
LDFLAGS.shlib = @SH_LDFLAGS@
#
# The following tx.XYZ vars may be populated/modified by teaish.tcl
# and/or teaish.make.
#
#
# tx.src is the list of source or object files to include in the
# (single) compiler/linker invocation. This will initially contain any
# sources passed to [teaish-src-add], but may also be appended to by
# teaish.make.
#
tx.src =@TEAISH_EXT_SRC@
#
# tx.CFLAGS is typically set by teaish.make, whereas TEAISH_CFLAGS
# gets set up via the configure script.
#
tx.CFLAGS =
#
# tx.LDFLAGS is typically set by teaish.make, whereas TEAISH_LDFLAGS
# gets set up via the configure script.
#
tx.LDFLAGS =
#
# The list of 'dist' files may be appended to from teaish.make.in.
# It can also be set up from teaish.tcl using [teaish-dist-add]
# and/or [teaish-src-add -dist ...].
#
tx.dist.files = @TEAISH_DIST_FILES@
# List of deps which may trigger an auto-reconfigure.
#
teaish__autogen.deps = \
$(tx.makefile.in) $(teaish.makefile.in) \
$(tx.tcl) \
@TEAISH_PKGINDEX_TCL_IN@ @TEAISH_TM_TCL_IN@ \
@AUTODEPS@
@if TEAISH_MAKEFILE_IN
$(tx.makefile): $(tx.makefile.in)
@endif
teaish.autoreconfig = \
@TEAISH_AUTORECONFIG@
#
# Problem: when more than one target can invoke TEAISH_AUTORECONFIG,
# we can get parallel reconfigures running. Thus, targets which
# may require reconfigure should depend on...
#
config.log: $(teaish__autogen.deps)
$(teaish.autoreconfig)
# ^^^ We would love to skip this when running [dist]clean, but there's
# no POSIX Make-portable way to do that. GNU Make can.
.PHONY: reconfigure
reconfigure:
$(teaish.autoreconfig)
$(teaish.makefile): $(teaish__auto.def) $(teaish.makefile.in) \
@AUTODEPS@
@if TEAISH_TESTER_TCL_IN
@TEAISH_TESTER_TCL_IN@:
@TEAISH_TESTER_TCL@: @TEAISH_TESTER_TCL_IN@
config.log: @TEAISH_TESTER_TCL@
@endif
#
# CC variant for compiling Tcl-using sources.
#
CC.tcl = \
$(CC) -o $@ $(CFLAGS.configure) $(CFLAGS) $(tx.CFLAGS)
#
# CC variant for linking $(tx.src) into an extension DLL. Note that
# $(tx.src) must come before $(LDFLAGS...) for linking to third-party
# libs to work.
#
CC.dll = \
$(CC.tcl) $(tx.src) $(LDFLAGS.shlib) \
$(LDFLAGS.configure) $(LDFLAGS) $(tx.LDFLAGS) $(TCL_STUB_LIB_SPEC)
@if TEAISH_ENABLE_DLL
#
# The rest of this makefile exists solely to support this brief
# target: the extension shared lib.
#
$(tx.dll): $(tx.src) config.log
@if [ "x" = "x$(tx.src)" ]; then \
echo "Makefile var tx.src (source/object files) is empty" 1>&2; \
exit 1; \
fi
$(CC.dll)
all: $(tx.dll)
@endif # TEAISH_ENABLE_DLL
tclsh: $(teaish.makefile) config.log
@{ echo "#!/bin/sh"; echo 'exec $(TCLSH) "$$@"'; } > $@
@chmod +x $@
@echo "Created $@"
#
# Run the generated test script.
#
.PHONY: test-pre test-prepre test-core test test-post test-extension
test-extension: # this name is reserved for use by teaish.make[.in]
@if TEAISH_ENABLE_DLL
test-prepre: $(tx.dll)
@endif
@if TEAISH_TESTER_TCL
test-core.args = @TEAISH_TESTER_TCL@
@if TEAISH_ENABLE_DLL
test-core.args += '$(tx.dll)' '$(tx.loadPrefix)'
@else
test-core.args += '' ''
@endif
test-core.args += @TEAISH_TESTUTIL_TCL@
test-core: test-pre
$(TCLSH) $(test-core.args)
test-prepre: @TEAISH_TESTER_TCL@
@else # !TEAISH_TESTER_TCL
test-prepre:
@endif # TEAISH_TESTER_TCL
test-pre: test-prepre
test-core: test-pre
test-post: test-core
test: test-post
#
# Cleanup rules...
#
#.PHONY: clean-pre clean-core clean-post clean-extension
#
clean-pre:
clean-core: clean-pre
rm -f $(tx.dll8) $(tx.dll9) tclsh
clean-post: clean-core
clean: clean-post
.PHONY: distclean-pre distclean-core distclean-post clean-extension
distclean-pre: clean
distclean-core: distclean-pre
rm -f Makefile
rm -f config.log config.defines.txt
@if TEAISH_MAKEFILE_IN
@if TEAISH_MAKEFILE
rm -f @TEAISH_MAKEFILE@
@endif
@endif
@if TEAISH_TESTER_TCL_IN
rm -f @TEAISH_TESTER_TCL@
@endif
@if TEAISH_PKGINDEX_TCL_IN
rm -f @TEAISH_PKGINDEX_TCL@
@endif
@if TEAISH_PKGINIT_TCL_IN
rm -f @TEAISH_PKGINIT_TCL@
@endif
@if TEAISH_TEST_TCL_IN
rm -f @TEAISH_TEST_TCL@
@endif
distclean-post: distclean-core
distclean: distclean-post
#
# The (dist)clean-extension targets are reserved for use by
# client-side teaish.make.
#
# Client code which wants to clean up extra stuff should do so by
# adding their cleanup target (e.g. clean-extension) as a dependency
# to the 'clean' target, like so:
#
# clean: distclean-extension
# distclean: distclean-extension
#
distclean-extension:
clean-extension:
#
# Installation rules...
#
@if TEAISH_ENABLE_INSTALL
.PHONY: install-pre install-core install-post install-test install-prepre install-extension
install-extension: # this name is reserved for use by teaish.make
@if TEAISH_ENABLE_DLL
install-prepre: $(tx.dll)
@else
install-prepre:
@endif
@if TEAISH_TM_TCL
install-core.tmdir = $(DESTDIR)@TEAISH_TCL_TM_DIR@
@endif
install-pre: install-prepre
install-core: install-pre
@if [ ! -d "$(DESTDIR)$(TCLLIBDIR)" ]; then \
set -x; $(INSTALL) -d "$(DESTDIR)$(TCLLIBDIR)"; \
fi
# ^^^^ on some platforms, install -d fails if the target already exists.
@if TEAISH_ENABLE_DLL
$(INSTALL) $(tx.dll) "$(DESTDIR)$(TCLLIBDIR)"
@endif
@if TEAISH_PKGINDEX_TCL
$(INSTALL.noexec) "@TEAISH_PKGINDEX_TCL@" "$(DESTDIR)$(TCLLIBDIR)"
@endif
@if TEAISH_PKGINIT_TCL
$(INSTALL.noexec) "@TEAISH_PKGINIT_TCL@" "$(DESTDIR)$(TCLLIBDIR)"
@endif
@if TEAISH_TM_TCL
@if [ ! -d "$(install-core.tmdir)" ]; then \
set -x; $(INSTALL) -d "$(install-core.tmdir)"; \
fi
$(INSTALL.noexec) "@TEAISH_TM_TCL@" "$(install-core.tmdir)/$(tx.tm.tgt)"
@endif
install-test: install-core
@echo "Post-install test of [package require $(tx.name.pkg) $(tx.version)]..."; \
if echo \
'set c 0; ' \
'@TEAISH_POSTINST_PREREQUIRE@' \
'if {[catch {package require $(tx.name.pkg) $(tx.version)}]} {incr c};' \
'exit $$c' \
| $(TCLSH) ; then \
echo "passed"; \
else \
echo "FAILED"; \
exit 1; \
fi
install-post: install-test
install: install-post
#
# Uninstall rules...
#
.PHONY: uninstall uninstall-pre uninstall-core uninstall-post uninstall-extension
uninstall-extension: # this name is reserved for use by teaish.make
uninstall-pre:
uninstall-core: uninstall-pre
@if TEAISH_ENABLE_DLL
rm -fr "$(DESTDIR)$(TCLLIBDIR)"
@endif
@if TEAISH_TM_TCL
rm -f "$(DESTDIR)$(install-core.tmdir)/$(tx.tm.tgt)"
@endif
uninstall-post: uninstall-core
@echo "Uninstalled Tcl extension $(tx.name) $(tx.version)"
uninstall: uninstall-post
@endif # TEAISH_ENABLE_INSTALL
@if TEAISH_MAKEFILE_IN
Makefile: $(tx.makefile.in)
config.log: $(teaish.makefile.in)
@endif
#
# Package archive generation ("dist") rules...
#
@if TEAISH_ENABLE_DIST
@if BIN_TAR
@if BIN_ZIP
# When installing teaish as part of "make dist", we need to run
# configure with similar flags to what we last configured with but we
# must not pass on any extension-specific flags, as those won't be
# recognized when running in --teaish-install mode, causing
# the sub-configure to fail.
dist.flags = --with-tclsh=$(TCLSH)
dist.reconfig = $(teaish.dir)/configure $(dist.flags)
# Temp dir for dist.zip. Must be different than dist.tgz or else
# parallel builds may hose the dist.
teaish__dist.tmp.zip = teaish__dist_zip
#
# Make a distribution zip file...
#
dist.basename = $(tx.name.dist)-$(tx.version)
dist.zip = $(dist.basename).zip
.PHONY: dist.zip dist.zip-core dist.zip-post
#dist.zip-pre:
# We apparently can't add a pre-hook here, else "make dist" rebuilds
# the archive each time it's run.
$(dist.zip): $(tx.dist.files)
@rm -fr $(teaish__dist.tmp.zip)
@mkdir -p $(teaish__dist.tmp.zip)/$(dist.basename)
@tar cf $(teaish__dist.tmp.zip)/tmp.tar $(tx.dist.files)
@tar xf $(teaish__dist.tmp.zip)/tmp.tar -C $(teaish__dist.tmp.zip)/$(dist.basename)
@if TEAISH_DIST_FULL
@$(dist.reconfig) \
--teaish-install=$(teaish__dist.tmp.zip)/$(dist.basename) \
--t-e-d=$(teaish__dist.tmp.zip)/$(dist.basename) >/dev/null
@endif
@rm -f $(dist.basename)/tmp.tar $(dist.zip)
@cd $(teaish__dist.tmp.zip) && zip -q -r ../$(dist.zip) $(dist.basename)
@rm -fr $(teaish__dist.tmp.zip)
@ls -la $(dist.zip)
dist.zip-core: $(dist.zip)
dist.zip-post: dist.zip-core
dist.zip: dist.zip-post
dist: dist.zip
undist-zip:
rm -f $(dist.zip)
undist: undist-zip
@endif #BIN_ZIP
#
# Make a distribution tarball...
#
teaish__dist.tmp.tgz = teaish__dist_tgz
dist.tgz = $(dist.basename).tar.gz
.PHONY: dist.tgz dist.tgz-core dist.tgz-post
# dist.tgz-pre:
# see notes in dist.zip
$(dist.tgz): $(tx.dist.files)
@rm -fr $(teaish__dist.tmp.tgz)
@mkdir -p $(teaish__dist.tmp.tgz)/$(dist.basename)
@tar cf $(teaish__dist.tmp.tgz)/tmp.tar $(tx.dist.files)
@tar xf $(teaish__dist.tmp.tgz)/tmp.tar -C $(teaish__dist.tmp.tgz)/$(dist.basename)
@if TEAISH_DIST_FULL
@rm -f $(teaish__dist.tmp.tgz)/$(dist.basename)/pkgIndex.tcl.in; # kludge
@$(dist.reconfig) \
--teaish-install=$(teaish__dist.tmp.tgz)/$(dist.basename) \
--t-e-d=$(teaish__dist.tmp.zip)/$(dist.basename) >/dev/null
@endif
@rm -f $(dist.basename)/tmp.tar $(dist.tgz)
@cd $(teaish__dist.tmp.tgz) && tar czf ../$(dist.tgz) $(dist.basename)
@rm -fr $(teaish__dist.tmp.tgz)
@ls -la $(dist.tgz)
dist.tgz-core: $(dist.tgz)
dist.tgz-post: dist.tgz-core
dist.tgz: dist.tgz-post
dist: dist.tgz
undist-tgz:
rm -f $(dist.tgz)
undist: undist-tgz
@else #!BIN_TAR
dist:
@echo "The dist rules require tar, which configure did not find." 1>&2; exit 1
@endif #BIN_TAR
@else #!TEAISH_ENABLE_DIST
undist:
dist:
@if TEAISH_OUT_OF_EXT_TREE
@echo "'dist' can only be used from an extension's home dir" 1>&2; \
echo "In this case: @TEAISH_EXT_DIR@" 1>&2; exit 1
@endif
@endif #TEAISH_ENABLE_DIST
Makefile: @TEAISH_TCL@
@if TEAISH_MAKEFILE_CODE
#
# TEAISH_MAKEFILE_CODE may contain literal makefile code, which
# gets pasted verbatim here. Either [define TEAISH_MAKEFILE_CODE
# ...] or use [teaish-make-add] to incrementally build up this
# content.
#
# <TEAISH_MAKEFILE_CODE>
@TEAISH_MAKEFILE_CODE@
# </TEAISH_MAKEFILE_CODE>
@endif
@if TEAISH_MAKEFILE
#
# TEAISH_MAKEFILE[_IN] defines any extension-specific state this file
# needs.
#
# It must set the following vars if they're not already accounted for
# via teaish.tcl.
#
# - tx.src = list of the extension's source files, being sure to
# prefix each with $(tx.dir) (if it's in the same dir as the
# extension) so that out-of-tree builds can find them. Optionally,
# [define] TEAISH_EXT_SRC or pass them to [teaish-src-add].
#
# It may optionally set the following vars:
#
# - tx.CFLAGS = CFLAGS/CPPFLAGS. Optionally, [define] TEAISH_CFLAGS
# or pass them to [teaish-cflags-add].
#
# - tx.LDFLAGS = LDFLAGS. Optionally, [define] TEAISH_LDFLAGS or
# pass them to [teaish-ldflags-add].
#
# It may optionally hook into various targets as documented in
# /doc/extensions.md in the canonical teaish source tree.
#
# Interestingly, we don't have to pre-filter teaish.makefile.in - we
# can just @include it here. That skips its teaish-specific validation
# though. Hmm.
#
# <TEAISH_MAKEFILE>
Makefile: @TEAISH_MAKEFILE@
@include @TEAISH_MAKEFILE@
# </TEAISH_MAKEFILE>
@endif
autoconf/tea/README.txt
+104
View File
@@ -0,0 +1,104 @@
This is the SQLite extension for Tcl using something akin to
the Tcl Extension Architecture (TEA). To build it:
./configure ...flags...
e.g.:
./configure --with-tcl=/path/to/tcl/install/root
or:
./configure --with-tclsh=/path/to/tcl/install/root
Run ./configure --help for the full list of flags.
The configuration process will fail if tclConfig.sh cannot be found.
The makefile will only honor CFLAGS and CPPFLAGS passed to the
configure script, not those directly passed to the makefile.
Then:
make test install
----------------------- THE PREFERRED WAY ---------------------------
The preferred way to build the TCL extension for SQLite is to use the
canonical source code tarball. For Unix:
./configure --with-tclsh=$(TCLSH)
make tclextension-install
For Windows:
nmake /f Makefile.msc tclextension-install TCLSH_CMD=$(TCLSH)
In both of the above, replace $(TCLSH) with the full pathname of
of the tclsh that you want the SQLite extension to work with. See
step-by-step instructions at the links below for more information:
https://sqlite.org/src/doc/trunk/doc/compile-for-unix.md
https://sqlite.org/src/doc/trunk/doc/compile-for-windows.md
And info about the extension's Tcl interface can be found at:
https://sqlite.org/tclsqlite.html
The whole point of the amalgamation-autoconf tarball (in which this
README.txt file is embedded) is to provide a means of compiling SQLite
that does not require first installing TCL and/or "tclsh". The
canonical Makefile in the SQLite source tree provides more
capabilities (such as the the ability to run test cases to ensure that
the build worked) and is better maintained. The only downside of the
canonical Makefile is that it requires a TCL installation. But if you
are wanting to build the TCL extension for SQLite, then presumably you
already have a TCL installation. So why not just use the more-capable
and better-maintained canoncal Makefile?
As of version 3.50.0, this build process uses "teaish":
https://fossil.wanderinghorse.net/r/teaish
which is conceptually derived from the pre-3.50 toolchain, TEA:
http://core.tcl-lang.org/tclconfig
http://core.tcl-lang.org/sampleextension
It to works for us. It might also work for you. But we cannot
promise that.
If you want to use this TEA builder and it works for you, that's fine.
But if you have trouble, the first thing you should do is go back
to using the canonical Makefile in the SQLite source tree.
------------------------------------------------------------------
UNIX BUILD
==========
Building under most UNIX systems is easy, just run the configure
script and then run make. For example:
$ cd sqlite-*-tea
$ ./configure --with-tcl=/path/to/tcl/install/root
$ make
$ make install
WINDOWS BUILD
=============
The recommended method to build extensions under windows is to use the
Msys + Mingw build process. This provides a Unix-style build while
generating native Windows binaries. Using the Msys + Mingw build tools
means that you can use the same configure script as per the Unix build
to create a Makefile. See the tcl/win/README file for the URL of
the Msys + Mingw download.
If you have VC++ then you may wish to use the files in the win
subdirectory and build the extension using just VC++. These files have
been designed to be as generic as possible but will require some
additional maintenance by the project developer to synchronise with
the TEA configure.in and Makefile.in files. Instructions for using the
VC++ makefile are written in the first part of the Makefile.vc
file.
autoconf/tea/_teaish.tester.tcl.in
+49
View File
@@ -0,0 +1,49 @@
# -*- tcl -*-
#
# Unless this file is named _teaish.tester.tcl.in, you are probably
# looking at an automatically generated/filtered copy and should
# probably not edit it.
#
# This is the wrapper script invoked by teaish's "make test" recipe.
# It gets passed 3 args:
#
# $1 = the DLL name, or "" if the extension has no DLL
#
# $2 = the "load prefix" for Tcl's [load] or empty if $1 is empty
#
# $3 = the /path/to/teaish/tester.tcl (test utility code)
#
@if TEAISH_VSATISFIES_CODE
@TEAISH_VSATISFIES_CODE@
@endif
if {[llength [lindex $::argv 0]] > 0} {
load [file normalize [lindex $::argv 0]] [lindex $::argv 1];
# ----^^^^^^^ needed on Haiku when argv 0 is just a filename, else
# load cannot find the file.
}
source -encoding utf-8 [lindex $::argv 2]; # teaish/tester.tcl
@if TEAISH_PKGINIT_TCL
apply {{file} {
set dir [file dirname $::argv0]
source -encoding utf-8 $file
}} [join {@TEAISH_PKGINIT_TCL@}]
@endif
@if TEAISH_TM_TCL
apply {{file} {
set dir [file dirname $::argv0]
source -encoding utf-8 $file
}} [join {@TEAISH_TM_TCL@}]
@endif
@if TEAISH_TEST_TCL
apply {{file} {
# Populate state for [tester.tcl::teaish-build-flag*]
array set ::teaish__BuildFlags @TEAISH__DEFINES_MAP@
set dir [file normalize [file dirname $file]]
#test-fail "Just testing"
source -encoding utf-8 $file
}} [join {@TEAISH_TEST_TCL@}]
@else # TEAISH_TEST_TCL
# No $TEAISH_TEST_TCL provided, so here's a default test which simply
# loads the extension.
puts {Extension @TEAISH_NAME@ @TEAISH_VERSION@ successfully loaded from @TEAISH_TESTER_TCL@}
@endif
autoconf/tea/auto.def
+8
View File
@@ -0,0 +1,8 @@
#/do/not/tclsh
# ^^^ help out editors which guess this file's content type.
#
# Main configure script entry point for the TEA(ish) framework. All
# extension-specific customization goes in teaish.tcl.in or
# teaish.tcl.
use teaish/core
teaish-configure-core
autoconf/tea/configure
Vendored Executable
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
dir0="`dirname "$0"`"
dirA="$dir0/../autosetup"
# This is the case ^^^^^^^^^^^^ in the SQLite "autoconf" bundle.
WRAPPER="$0"; export WRAPPER; exec "`"$dirA/autosetup-find-tclsh"`" \
"$dirA/autosetup" --teaish-extension-dir="$dir0" \
"$@"
autoconf/tea/doc/sqlite3.n
+15
View File
@@ -0,0 +1,15 @@
.TH sqlite3 n 4.1 "Tcl-Extensions"
.HS sqlite3 tcl
.BS
.SH NAME
sqlite3 \- an interface to the SQLite3 database engine
.SH SYNOPSIS
\fBsqlite3\fI command_name ?filename?\fR
.br
.SH DESCRIPTION
SQLite3 is a self-contains, zero-configuration, transactional SQL database
engine. This extension provides an easy to use interface for accessing
SQLite database files from Tcl.
.PP
For full documentation see \fIhttps://sqlite.org/\fR and
in particular \fIhttps://sqlite.org/tclsqlite.html\fR.
autoconf/tea/license.terms
+6
View File
@@ -0,0 +1,6 @@
The author disclaims copyright to this source code. In place of
a legal notice, here is a blessing:
May you do good and not evil.
May you find forgiveness for yourself and forgive others.
May you share freely, never taking more than you give.
autoconf/tea/pkgIndex.tcl.in
+40
View File
@@ -0,0 +1,40 @@
# -*- tcl -*-
# Tcl package index file
#
# Unless this file is named pkgIndex.tcl.in, you are probably looking
# at an automatically generated/filtered copy and should probably not
# edit it.
#
# Adapted from https://core.tcl-lang.org/tcltls
@if TEAISH_VSATISFIES_CODE
@TEAISH_VSATISFIES_CODE@
@endif
if {[package vsatisfies [package provide Tcl] 9.0-]} {
package ifneeded {@TEAISH_PKGNAME@} {@TEAISH_VERSION@} [list apply {{dir} {
@if TEAISH_ENABLE_DLL
load [file join $dir {@TEAISH_DLL9@}] @TEAISH_LOAD_PREFIX@
@endif
@if TEAISH_PKGINIT_TCL_TAIL
set initScript [file join $dir {@TEAISH_PKGINIT_TCL_TAIL@}]
if {[file exists $initScript]} {
source -encoding utf-8 $initScript
}
@endif
}} $dir]
} else {
package ifneeded {@TEAISH_PKGNAME@} {@TEAISH_VERSION@} [list apply {{dir} {
@if TEAISH_ENABLE_DLL
if {[string tolower [file extension {@TEAISH_DLL8@}]] in [list .dll .dylib .so]} {
load [file join $dir {@TEAISH_DLL8@}] @TEAISH_LOAD_PREFIX@
} else {
load {} @TEAISH_LOAD_PREFIX@
}
@endif
@if TEAISH_PKGINIT_TCL_TAIL
set initScript [file join $dir {@TEAISH_PKGINIT_TCL_TAIL@}]
if {[file exists $initScript]} {
source -encoding utf-8 $initScript
}
@endif
}} $dir]
}
autoconf/tea/teaish.tcl
+565
View File
@@ -0,0 +1,565 @@
# Teaish configure script for the SQLite Tcl extension
#
# State for disparate config-time pieces.
#
array set sqlite__Config [proj-strip-hash-comments {
#
# The list of feature --flags which the --all flag implies. This
# requires special handling in a few places.
#
all-flag-enables {fts3 fts4 fts5 rtree geopoly}
# >0 if building in the canonical tree. -1=undetermined
is-canonical -1
}]
#
# Set up the package info for teaish...
#
apply {{dir} {
# Figure out the version number...
set version ""
if {[file exists $dir/../VERSION]} {
# The canonical SQLite TEA(ish) build
set version [proj-file-content -trim $dir/../VERSION]
set ::sqlite__Config(is-canonical) 1
set distname sqlite-tcl
} elseif {[file exists $dir/generic/tclsqlite3.c]} {
# The copy from the teaish tree, used as a dev/test bed before
# updating SQLite's tree.
set ::sqlite__Config(is-canonical) 0
set fd [open $dir/generic/tclsqlite3.c rb]
while {[gets $fd line] >=0} {
if {[regexp {^#define[ ]+SQLITE_VERSION[ ]+"(3.+)"} \
$line - version]} {
set distname sqlite-teaish
break
}
}
close $fd
}
if {"" eq $version} {
proj-fatal "Cannot determine the SQLite version number"
}
proj-assert {$::sqlite__Config(is-canonical) > -1}
proj-assert {[string match 3.*.* $version]} \
"Unexpected SQLite version: $version"
set pragmas {}
if {$::sqlite__Config(is-canonical)} {
# Disable "make dist" in the canonical tree. That tree is
# generated from several pieces and creating/testing working
# "dist" rules for that sub-build currently feels unnecessary. The
# copy in the teaish tree, though, should be able to "make dist".
lappend pragmas no-dist
} else {
lappend pragmas full-dist
}
teaish-pkginfo-set -vars {
-name sqlite
-name.pkg sqlite3
-version $version
-name.dist $distname
-vsatisfies 8.6-
-libDir sqlite$version
-pragmas $pragmas
}
}} [teaish-get -dir]
#
# Must return either an empty string or a list in the form accepted by
# autosetup's [options] function.
#
proc teaish-options {} {
# These flags and defaults mostly derive from the historical TEA
# build. Some, like ICU, are taken from the canonical SQLite tree.
return [subst -nocommands -nobackslashes {
with-system-sqlite=0
=> {Use the system-level SQLite instead of the copy in this tree.
Also requires use of --override-sqlite-version so that the build
knows what version number to associate with the system-level SQLite.}
override-sqlite-version:VERSION
=> {For use with --with-system-sqlite to set the version number.}
threadsafe=1 => {Disable mutexing}
with-tempstore:=no => {Use an in-RAM database for temporary tables: never,no,yes,always}
load-extension=0 => {Enable loading of external extensions}
math=1 => {Disable math functions}
json=1 => {Disable JSON functions}
fts3 => {Enable the FTS3 extension}
fts4 => {Enable the FTS4 extension}
fts5 => {Enable the FTS5 extension}
update-limit => {Enable the UPDATE/DELETE LIMIT clause}
geopoly => {Enable the GEOPOLY extension}
rtree => {Enable the RTREE extension}
session => {Enable the SESSION extension}
all=1 => {Disable $::sqlite__Config(all-flag-enables)}
with-icu-ldflags:LDFLAGS
=> {Enable SQLITE_ENABLE_ICU and add the given linker flags for the
ICU libraries. e.g. on Ubuntu systems, try '-licui18n -licuuc -licudata'.}
with-icu-cflags:CFLAGS
=> {Apply extra CFLAGS/CPPFLAGS necessary for building with ICU.
e.g. -I/usr/local/include}
with-icu-config:=auto
=> {Enable SQLITE_ENABLE_ICU. Value must be one of: auto, pkg-config,
/path/to/icu-config}
icu-collations=0
=> {Enable SQLITE_ENABLE_ICU_COLLATIONS. Requires --with-icu-ldflags=...
or --with-icu-config}
}]
}
#
# Gets called by tea-configure-core. Must perform any configuration
# work needed for this extension.
#
proc teaish-configure {} {
use teaish/feature
teaish-src-add -dist -dir generic/tclsqlite3.c
if {[proj-opt-was-provided override-sqlite-version]} {
teaish-pkginfo-set -version [opt-val override-sqlite-version]
proj-warn "overriding sqlite version number:" [teaish-pkginfo-get -version]
} elseif {[proj-opt-was-provided with-system-sqlite]
&& [opt-val with-system-sqlite] ne "0"} {
proj-fatal "when using --with-system-sqlite also use" \
"--override-sqlite-version to specify a library version number."
}
define CFLAGS [proj-get-env CFLAGS {-O2}]
sqlite-munge-cflags
#
# Add feature flags from legacy configure.ac which are not covered by
# --flags.
#
sqlite-add-feature-flag {
-DSQLITE_3_SUFFIX_ONLY=1
-DSQLITE_ENABLE_DESERIALIZE=1
-DSQLITE_ENABLE_DBPAGE_VTAB=1
-DSQLITE_ENABLE_BYTECODE_VTAB=1
-DSQLITE_ENABLE_DBSTAT_VTAB=1
}
if {[opt-bool with-system-sqlite]} {
msg-result "Using system-level sqlite3."
teaish-cflags-add -DUSE_SYSTEM_SQLITE
teaish-ldflags-add -lsqlite3
} elseif {$::sqlite__Config(is-canonical)} {
teaish-cflags-add -I[teaish-get -dir]/..
}
teaish-check-librt
teaish-check-libz
sqlite-handle-threadsafe
sqlite-handle-tempstore
sqlite-handle-load-extension
sqlite-handle-math
sqlite-handle-icu
sqlite-handle-common-feature-flags; # must be late in the process
}; # teaish-configure
define OPT_FEATURE_FLAGS {} ; # -DSQLITE_OMIT/ENABLE flags.
#
# Adds $args, if not empty, to OPT_FEATURE_FLAGS. This is intended only for holding
# -DSQLITE_ENABLE/OMIT/... flags, but that is not enforced here.
proc sqlite-add-feature-flag {args} {
if {"" ne $args} {
define-append OPT_FEATURE_FLAGS {*}$args
}
}
#
# Check for log(3) in libm and die with an error if it is not
# found. $featureName should be the feature name which requires that
# function (it's used only in error messages). defines LDFLAGS_MATH to
# the required linker flags (which may be empty even if the math APIs
# are found, depending on the OS).
proc sqlite-affirm-have-math {featureName} {
if {"" eq [get-define LDFLAGS_MATH ""]} {
if {![msg-quiet proj-check-function-in-lib log m]} {
user-error "Missing math APIs for $featureName"
}
set lfl [get-define lib_log ""]
undefine lib_log
if {"" ne $lfl} {
user-notice "Forcing requirement of $lfl for $featureName"
}
define LDFLAGS_MATH $lfl
teaish-ldflags-prepend $lfl
}
}
#
# Handle various SQLITE_ENABLE/OMIT_... feature flags.
proc sqlite-handle-common-feature-flags {} {
msg-result "Feature flags..."
if {![opt-bool all]} {
# Special handling for --disable-all
foreach flag $::sqlite__Config(all-flag-enables) {
if {![proj-opt-was-provided $flag]} {
proj-opt-set $flag 0
}
}
}
foreach {boolFlag featureFlag ifSetEvalThis} [proj-strip-hash-comments {
all {} {
# The 'all' option must be first in this list. This impl makes
# an effort to only apply flags which the user did not already
# apply, so that combinations like (--all --disable-geopoly)
# will indeed disable geopoly. There are corner cases where
# flags which depend on each other will behave in non-intuitive
# ways:
#
# --all --disable-rtree
#
# Will NOT disable geopoly, though geopoly depends on rtree.
# The --geopoly flag, though, will automatically re-enable
# --rtree, so --disable-rtree won't actually disable anything in
# that case.
foreach k $::sqlite__Config(all-flag-enables) {
if {![proj-opt-was-provided $k]} {
proj-opt-set $k 1
}
}
}
fts3 -DSQLITE_ENABLE_FTS3 {sqlite-affirm-have-math fts3}
fts4 -DSQLITE_ENABLE_FTS4 {sqlite-affirm-have-math fts4}
fts5 -DSQLITE_ENABLE_FTS5 {sqlite-affirm-have-math fts5}
geopoly -DSQLITE_ENABLE_GEOPOLY {proj-opt-set rtree}
rtree -DSQLITE_ENABLE_RTREE {}
session {-DSQLITE_ENABLE_SESSION -DSQLITE_ENABLE_PREUPDATE_HOOK} {}
update-limit -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT {}
scanstatus -DSQLITE_ENABLE_STMT_SCANSTATUS {}
}] {
if {$boolFlag ni $::autosetup(options)} {
# Skip flags which are in the canonical build but not
# the autoconf bundle.
continue
}
proj-if-opt-truthy $boolFlag {
sqlite-add-feature-flag $featureFlag
if {0 != [eval $ifSetEvalThis] && "all" ne $boolFlag} {
msg-result " + $boolFlag"
}
} {
if {"all" ne $boolFlag} {
msg-result " - $boolFlag"
}
}
}
#
# Invert the above loop's logic for some SQLITE_OMIT_... cases. If
# config option $boolFlag is false, [sqlite-add-feature-flag
# $featureFlag], where $featureFlag is intended to be
# -DSQLITE_OMIT_...
foreach {boolFlag featureFlag} {
json -DSQLITE_OMIT_JSON
} {
if {[proj-opt-truthy $boolFlag]} {
msg-result " + $boolFlag"
} else {
sqlite-add-feature-flag $featureFlag
msg-result " - $boolFlag"
}
}
##
# Remove duplicates from the final feature flag sets and show them
# to the user.
set oFF [get-define OPT_FEATURE_FLAGS]
if {"" ne $oFF} {
define OPT_FEATURE_FLAGS [lsort -unique $oFF]
msg-result "Library feature flags: [get-define OPT_FEATURE_FLAGS]"
}
if {[lsearch [get-define TARGET_DEBUG ""] -DSQLITE_DEBUG=1] > -1} {
msg-result "Note: this is a debug build, so performance will suffer."
}
teaish-cflags-add -define OPT_FEATURE_FLAGS
}; # sqlite-handle-common-feature-flags
#
# If --enable-threadsafe is set, this adds -DSQLITE_THREADSAFE=1 to
# OPT_FEATURE_FLAGS and sets LDFLAGS_PTHREAD to the linker flags
# needed for linking pthread (possibly an empty string). If
# --enable-threadsafe is not set, adds -DSQLITE_THREADSAFE=0 to
# OPT_FEATURE_FLAGS and sets LDFLAGS_PTHREAD to an empty string.
#
# It prepends the flags to the global LDFLAGS.
proc sqlite-handle-threadsafe {} {
msg-checking "Support threadsafe operation? "
define LDFLAGS_PTHREAD ""
set enable 0
if {[proj-opt-was-provided threadsafe]} {
proj-if-opt-truthy threadsafe {
if {[proj-check-function-in-lib pthread_create pthread]
&& [proj-check-function-in-lib pthread_mutexattr_init pthread]} {
incr enable
set ldf [get-define lib_pthread_create]
define LDFLAGS_PTHREAD $ldf
teaish-ldflags-prepend $ldf
undefine lib_pthread_create
undefine lib_pthread_mutexattr_init
} else {
user-error "Missing required pthread libraries. Use --disable-threadsafe to disable this check."
}
# Recall that LDFLAGS_PTHREAD might be empty even if pthreads if
# found because it's in -lc on some platforms.
} {
msg-result "Disabled using --disable-threadsafe"
}
} else {
#
# If user does not specify --[disable-]threadsafe then select a
# default based on whether it looks like Tcl has threading
# support.
#
catch {
scan [exec echo {puts [tcl::pkgconfig get threaded]} | [get-define TCLSH_CMD]] \
%d enable
}
if {$enable} {
set flagName "--threadsafe"
set lblAbled "enabled"
msg-result yes
} else {
set flagName "--disable-threadsafe"
set lblAbled "disabled"
msg-result no
}
msg-result "Defaulting to ${flagName} because Tcl has threading ${lblAbled}."
# ^^^ We (probably) don't need to link against -lpthread in the
# is-enabled case. We might in the case of static linking. Unsure.
}
sqlite-add-feature-flag -DSQLITE_THREADSAFE=${enable}
return $enable
}
#
# Handles the --enable-load-extension flag. Returns 1 if the support
# is enabled, else 0. If support for that feature is not found, a
# fatal error is triggered if --enable-load-extension is explicitly
# provided, else a loud warning is instead emitted. If
# --disable-load-extension is used, no check is performed.
#
# Makes the following environment changes:
#
# - defines LDFLAGS_DLOPEN to any linker flags needed for this
# feature. It may legally be empty on some systems where dlopen()
# is in libc.
#
# - If the feature is not available, adds
# -DSQLITE_OMIT_LOAD_EXTENSION=1 to the feature flags list.
proc sqlite-handle-load-extension {} {
define LDFLAGS_DLOPEN ""
set found 0
proj-if-opt-truthy load-extension {
set found [proj-check-function-in-lib dlopen dl]
if {$found} {
set ldf [get-define lib_dlopen]
define LDFLAGS_DLOPEN $ldf
teaish-ldflags-prepend $ldf
undefine lib_dlopen
} else {
if {[proj-opt-was-provided load-extension]} {
# Explicit --enable-load-extension: fail if not found
proj-indented-notice -error {
--enable-load-extension was provided but dlopen()
not found. Use --disable-load-extension to bypass this
check.
}
} else {
# It was implicitly enabled: warn if not found
proj-indented-notice {
WARNING: dlopen() not found, so loadable module support will
be disabled. Use --disable-load-extension to bypass this
check.
}
}
}
}
if {$found} {
msg-result "Loadable extension support enabled."
} else {
msg-result "Disabling loadable extension support. Use --enable-load-extension to enable them."
sqlite-add-feature-flag -DSQLITE_OMIT_LOAD_EXTENSION=1
}
return $found
}
#
# ICU - International Components for Unicode
#
# Handles these flags:
#
# --with-icu-ldflags=LDFLAGS
# --with-icu-cflags=CFLAGS
# --with-icu-config[=auto | pkg-config | /path/to/icu-config]
# --enable-icu-collations
#
# --with-icu-config values:
#
# - auto: use the first one of (pkg-config, icu-config) found on the
# system.
# - pkg-config: use only pkg-config to determine flags
# - /path/to/icu-config: use that to determine flags
#
# If --with-icu-config is used as neither pkg-config nor icu-config
# are found, fail fatally.
#
# If both --with-icu-ldflags and --with-icu-config are provided, they
# are cumulative. If neither are provided, icu-collations is not
# honored and a warning is emitted if it is provided.
#
# Design note: though we could automatically enable ICU if the
# icu-config binary or (pkg-config icu-io) are found, we specifically
# do not. ICU is always an opt-in feature.
proc sqlite-handle-icu {} {
define LDFLAGS_LIBICU [join [opt-val with-icu-ldflags ""]]
define CFLAGS_LIBICU [join [opt-val with-icu-cflags ""]]
if {[proj-opt-was-provided with-icu-config]} {
msg-result "Checking for ICU support..."
set icuConfigBin [opt-val with-icu-config]
set tryIcuConfigBin 1; # set to 0 if we end up using pkg-config
if {$icuConfigBin in {auto pkg-config}} {
uplevel 3 { use pkg-config }
if {[pkg-config-init 0] && [pkg-config icu-io]} {
# Maintenance reminder: historical docs say to use both of
# (icu-io, icu-uc). icu-uc lacks a required lib and icu-io has
# all of them on tested OSes.
set tryIcuConfigBin 0
define LDFLAGS_LIBICU [get-define PKG_ICU_IO_LDFLAGS]
define-append LDFLAGS_LIBICU [get-define PKG_ICU_IO_LIBS]
define CFLAGS_LIBICU [get-define PKG_ICU_IO_CFLAGS]
} elseif {"pkg-config" eq $icuConfigBin} {
proj-fatal "pkg-config cannot find package icu-io"
} else {
proj-assert {"auto" eq $icuConfigBin}
}
}
if {$tryIcuConfigBin} {
if {"auto" eq $icuConfigBin} {
set icuConfigBin [proj-first-bin-of \
/usr/local/bin/icu-config \
/usr/bin/icu-config]
if {"" eq $icuConfigBin} {
proj-indented-notice -error {
--with-icu-config=auto cannot find (pkg-config icu-io) or icu-config binary.
On Ubuntu-like systems try:
--with-icu-ldflags='-licui18n -licuuc -licudata'
}
}
}
if {[file-isexec $icuConfigBin]} {
set x [exec $icuConfigBin --ldflags]
if {"" eq $x} {
proj-indented-notice -error \
[subst {
$icuConfigBin --ldflags returned no data.
On Ubuntu-like systems try:
--with-icu-ldflags='-licui18n -licuuc -licudata'
}]
}
define-append LDFLAGS_LIBICU $x
set x [exec $icuConfigBin --cppflags]
define-append CFLAGS_LIBICU $x
} else {
proj-fatal "--with-icu-config=$icuConfigBin does not refer to an executable"
}
}
}
set ldflags [define LDFLAGS_LIBICU [string trim [get-define LDFLAGS_LIBICU]]]
set cflags [define CFLAGS_LIBICU [string trim [get-define CFLAGS_LIBICU]]]
if {"" ne $ldflags} {
sqlite-add-feature-flag -DSQLITE_ENABLE_ICU
msg-result "Enabling ICU support with flags: $ldflags $cflags"
if {[opt-bool icu-collations]} {
msg-result "Enabling ICU collations."
sqlite-add-feature-flag -DSQLITE_ENABLE_ICU_COLLATIONS
}
teaish-ldflags-prepend $ldflags
teaish-cflags-add $cflags
} elseif {[opt-bool icu-collations]} {
proj-warn "ignoring --enable-icu-collations because neither --with-icu-ldflags nor --with-icu-config provided any linker flags"
} else {
msg-result "ICU support is disabled."
}
}; # sqlite-handle-icu
#
# Handles the --with-tempstore flag.
#
# The test fixture likes to set SQLITE_TEMP_STORE on its own, so do
# not set that feature flag unless it was explicitly provided to the
# configure script.
proc sqlite-handle-tempstore {} {
if {[proj-opt-was-provided with-tempstore]} {
set ts [opt-val with-tempstore no]
set tsn 1
msg-checking "Use an in-RAM database for temporary tables? "
switch -exact -- $ts {
never { set tsn 0 }
no { set tsn 1 }
yes { set tsn 2 }
always { set tsn 3 }
default {
user-error "Invalid --with-tempstore value '$ts'. Use one of: never, no, yes, always"
}
}
msg-result $ts
sqlite-add-feature-flag -DSQLITE_TEMP_STORE=$tsn
}
}
#
# Handles the --enable-math flag.
proc sqlite-handle-math {} {
proj-if-opt-truthy math {
if {![proj-check-function-in-lib ceil m]} {
user-error "Cannot find libm functions. Use --disable-math to bypass this."
}
set lfl [get-define lib_ceil]
undefine lib_ceil
define LDFLAGS_MATH $lfl
teaish-ldflags-prepend $lfl
sqlite-add-feature-flag -DSQLITE_ENABLE_MATH_FUNCTIONS
msg-result "Enabling math SQL functions"
} {
define LDFLAGS_MATH ""
msg-result "Disabling math SQL functions"
}
}
#
# Move -DSQLITE_OMIT... and -DSQLITE_ENABLE... flags from CFLAGS and
# CPPFLAGS to OPT_FEATURE_FLAGS and remove them from BUILD_CFLAGS.
proc sqlite-munge-cflags {} {
# Move CFLAGS and CPPFLAGS entries matching -DSQLITE_OMIT* and
# -DSQLITE_ENABLE* to OPT_FEATURE_FLAGS. This behavior is derived
# from the pre-3.48 build.
#
# If any configure flags for features are in conflict with
# CFLAGS/CPPFLAGS-specified feature flags, all bets are off. There
# are no guarantees about which one will take precedence.
foreach flagDef {CFLAGS CPPFLAGS} {
set tmp ""
foreach cf [get-define $flagDef ""] {
switch -glob -- $cf {
-DSQLITE_OMIT* -
-DSQLITE_ENABLE* {
sqlite-add-feature-flag $cf
}
default {
lappend tmp $cf
}
}
}
define $flagDef $tmp
}
}
autoconf/tea/teaish.test.tcl
+14
View File
@@ -0,0 +1,14 @@
test-expect 1.0-open {
sqlite3 db :memory:
} {}
test-assert 1.1-version-3.x {
[string match 3.* [db eval {select sqlite_version()}]]
}
test-expect 1.2-select {
db eval {select 'hi, world',1,2,3}
} {{hi, world} 1 2 3}
test-expect 99.0-db-close {db close} {}
autosetup/LICENSE
+35
View File
@@ -0,0 +1,35 @@
Unless explicitly stated, all files which form part of autosetup
are released under the following license:
---------------------------------------------------------------------
autosetup - A build environment "autoconfigurator"
Copyright (c) 2010-2011, WorkWare Systems <http://workware.net.au/>
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials
provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE WORKWARE SYSTEMS ``AS IS'' AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WORKWARE
SYSTEMS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The views and conclusions contained in the software and documentation
are those of the authors and should not be interpreted as representing
official policies, either expressed or implied, of WorkWare Systems.
autosetup/README.autosetup
+11
View File
@@ -0,0 +1,11 @@
README.autosetup created by autosetup v0.7.2
This is the autosetup directory for a local install of autosetup.
It contains autosetup, support files and loadable modules.
*.tcl files in this directory are optional modules which
can be loaded with the 'use' directive.
*.auto files in this directory are auto-loaded.
For more information, see https://msteveb.github.io/autosetup/
autosetup/README.md
+453
View File
@@ -0,0 +1,453 @@
Maintaining Autosetup in the SQLite Tree
========================================================================
This document provides some tips and reminders for the SQLite
developers regarding using and maintaining the [Autosetup][]-based
build infrastructure. It is not an [Autosetup][] reference.
**Table of Contents**:
- [Autosetup API Reference](#apiref)
- [API Tips](#apitips)
- [Ensuring TCL Compatibility](#tclcompat)
- [Design Conventions](#conventions)
- Symbolic Names of Feature Flags
- Do Not Update Global Shared State
- [Updating Autosetup](#updating)
- ***[Patching Autosetup for Project-local changes](#patching)***
- [Branch-specific Customization](#branch-customization)
------------------------------------------------------------------------
<a name="apiref"></a>
Autosetup API Reference
========================================================================
The Autosetup API is quite extensive and can be read either in
the [files in the `autosetup` dir](/dir/autosetup) or using:
>
```
$ ./configure --reference | less
```
That will include any docs from any TCL files in the `./autosetup` dir
which contain certain (simple) markup defined by autosetup.
This project's own configuration-related TCL code is spread across the
following files:
- [proj.tcl][]: project-agnostic utility code for autosetup-driven
projects. This file is designed to be shared between this project,
other projects managed under the SQLite/Hwaci umbrella
(e.g. Fossil), and personal projects of SQLite's developers. It is
essentially an amalgamation of a decade's worth of autosetup-related
utility code.
- [sqlite-config.tcl][]: utility code which is too project-specific
for `proj.tcl`. We split this out of `auto.def` so that it can be
used by both `auto.def` and...
- [auto.def][]: the primary driver for the `./configure` process.
When we talk about "the configure script," we're technically
referring to this file, though it actually contains very little
of the TCL code.
- [autoconf/auto.def][]: the main driver script for the "autoconf"
bundle's configure script. It is essentially a slightly trimmed-down
version of the main `auto.def` file. The `autoconf` dir was ported
from the Autotools to Autosetup in the 3.49.0 dev cycle but retains
the "autoconf" name to minimize downstream disruption.
<a name="apitips"></a>
Autosetup API Tips
========================================================================
This section briefly covers only APIs which are frequently useful in
day-to-day maintenance and might not be immediately recognized as such
from a casual perusal of the relevant TCL files. The complete docs of
those with `proj-` prefix can be found in [proj.tcl][] and those with
an `sqlite-` prefix are in [sqlite-config.tcl][]. The others are part
of Autosetup's core packages and are scattered around [the TCL files
in ./autosetup](/dir/autosetup).
In (mostly) alphabetical order:
- **`file-isexec filename`**\
Should be used in place of `[file executable]`, as it will also
check for `${filename}.exe` on Windows platforms. However, on such
platforms it also assumes that _any_ existing file is executable.
- **`get-env VAR ?default?`**\
Will fetch an "environment variable" from the first of either: (1) a
KEY=VALUE passed to the configure script or (2) the system's
environment variables. Not to be confused with `getenv`, which only
does the latter and is rarely, if ever, useful in this tree.
- **`proj-get-env VAR ?default?`**\
Works like `get-env` but will, if that function finds no match,
look for a file named `./.env-$VAR` and, if found, return its
trimmed contents. This can be used, e.g., to set a developer's
local preferences for the default `CFLAGS`.\
Tip: adding `-O0` to `.env-CFLAGS` reduces rebuild times
considerably at the cost of performance in `make devtest` and the
like.
- **`proj-fatal msg`**\
Emits `$msg` to stderr and exits with non-zero. Its differences from
autosetup's `user-error` are purely cosmetic.
- **`proj-if-opt-truthy flag thenScript ?elseScript?`**\
Evals `thenScript` if the given `--flag` is truthy, else it
evals the optional `elseScript`.
- **`proj-indented-notice ?-error? ?-notice? msg`**\
Breaks its `msg` argument into lines, trims them, and emits them
with consistent indentation. Exactly how it emits depends on the
flags passed to it (or not), as covered in its docs. This will stick
out starkly from normal output and is intended to be used only for
important notices.
- **`proj-opt-truthy flag`**\
Returns 1 if `--flag`'s value is "truthy," i.e. one of (1, on,
enabled, yes, true).
- **`proj-opt-was-provided FLAG`**\
Returns 1 if `--FLAG` was explicitly provided to configure,
else 0. This distinction can be used to determine, e.g., whether
`--with-readline` was provided or whether we're searching for
readline by default. In the former case, failure to find it should
be treated as fatal, where in the latter case it's not.\
Unlike most functions which deal with `--flags`, this one does not
validate that `$FLAG` is a registered flag so will not fail fatally
if `$FLAG` is not registered as an Autosetup option.
- **`proj-val-truthy value`**\
Returns 1 if `$value` is "truthy," See `proj-opt-truthy` for the definition
of "truthy."
- **`proj-warn msg`**\
Emits `$msg` to stderr. Closely-related is autosetup's `user-notice`
(described below).
- **`sqlite-add-feature-flag ?-shell? FLAG...`**\
Adds the given feature flag to the CFLAGS which are specific to
building libsqlite3. It's intended to be passed one or more
`-DSQLITE_ENABLE_...`, or similar, flags. If the `-shell` flag is
used then it also passes its arguments to
`sqlite-add-shell-opt`. This is a no-op if `FLAG` is not provided or
is empty.
- **`sqlite-add-shell-opt FLAG...`**\
The shell-specific counterpart of `sqlite-add-feature-flag` which
only adds the given flag(s) to the CLI-shell-specific CFLAGS.
- **`sqlite-configure BUILD-NAME {script}`**\
This is where all configure `--flags` are defined for all known
build modes ("canonical" or "autoconf"). After processing all flags,
this function runs `$script`, which contains the build-mode-specific
configuration bits, and then runs any finalization bits which are
common to all build modes. The `auto.def` files are intended to contain
exactly two commands:
`use sqlite-config; sqlite-configure BUILD-NAME {script}`
- **`user-notice msg`**\
Queues `$msg` to be sent to stderr, but does not emit it until
either `show-notices` is called or the next time autosetup would
output something (it internally calls `show-notices`). This can be
used to generate warnings between a "checking for..." message and
its resulting "yes/no/whatever" message in such a way as to not
spoil the layout of such messages.
<a name="tclcompat"></a>
Ensuring TCL Compatibility
========================================================================
One of the significant benefits of using Autosetup is that (A) this
project uses many TCL scripts in the build process and (B) Autosetup
comes with a TCL interpreter named [JimTCL][].
It is important that any TCL files used by the configure process and
makefiles remain compatible with both [JimTCL][] and the canonical
TCL. Though JimTCL has outstanding compatibility with canonical TCL,
it does have a few corners with incompatibilities, e.g. regular
expressions. If a script runs in JimTCL without using any
JimTCL-specific features, then it's a certainty that it will run in
canonical TCL as well. The opposite, however, is not _always_ the
case.
When [`./configure`](/file/configure) is run, it goes through a
bootstrapping process to find a suitable TCL with which to run the
autosetup framework. The first step involves [finding or building a
TCL shell](/file/autosetup/autosetup-find-tclsh). That will first
search for an available `tclsh` (under several common names,
e.g. `tclsh8.6`) before falling back to compiling the copy of
`jimsh0.c` included in the source tree. i.e. it will prefer to use a
system-installed TCL for running the configure script. Once it finds
(or builds) a TCL shell, it then runs [a sanity test to ensure that
the shell is suitable](/file/autosetup/autosetup-test-tclsh) before
using it to run the main autosetup app.
There are two simple ways to ensure that running of the configure
process uses JimTCL instead of the canonical `tclsh`, and either
approach provides equally high assurances about configure script
compatibility across TCL implementations:
1. Build on a system with no `tclsh` installed in the `$PATH`. In that
case, the configure process will fall back to building the in-tree
copy of JimTCL.
2. Manually build `./jimsh0` in the top of the checkout with:\
`cc -o jimsh0 autosetup/jimsh0.c`\
With that in place, the configure script will prefer to use that
before looking for a system-level `tclsh`. Be aware, though, that
`make distclean` will remove that file.
**Note that `./jimsh0` is distinctly different from the `./jimsh`**
which gets built for code-generation purposes. The latter requires
non-default build flags to enable features which are
platform-dependent, most notably to make its `[file normalize]` work.
This means, for example, that the configure script and its utility
APIs must not use `[file normalize]`, but autosetup provides a
TCL-only implementation of `[file-normalize]` (note the dash) for
portable use in the configure script. Contrariwise, code-generation
scripts invoked via `make` may use `[file normalize]`, as they'll use
`./jimsh` or `tclsh` instead of `./jimsh0`.
Known TCL Incompatibilities
------------------------------------------------------------------------
A summary of known incompatibilities in JimTCL
- **CRNL line endings**: prior to 2025-02-05 `fconfigure -translation ...`
was a no-op in JimTCL, and it emits CRNL line endings by default on
Windows. Since then, it supports `-translation binary`, which is
close enough to `-translation lf` for our purposes. When working
with files using the `open` command, it is important to use mode
`"rb"` or `"wb"`, as appropriate, so that the output does not get
CRNL-mangled on Windows.
- **`file copy`** does not support multiple source files. See
[](/info/61f18c96183867fe) for a workaround.
- **Regular expressions**:
- Patterns treat `\nnn` octal values as back-references (which it
does not support). Those can be reformulated as demonstrated in
[](/info/aeac23359bb681c0).
- `regsub` does not support the `\y` flag. A workaround is demonstrated
in [](/info/c2e5dd791cce3ec4).
<a name="conventions"></a>
Design Conventions
========================================================================
This section describes the motivations for the most glaring of the
build's design decisions, in particular how they deviate from
historical, or even widely-conventional, practices.
Symbolic Names of Feature Flags
------------------------------------------------------------------------
Historically, the project's makefile has exclusively used
`UPPER_UNDERSCORE` form for makefile variables. This build, however,
primarily uses `X.y` format, where `X` is often a category label,
e.g. `CFLAGS`, and `y` is the specific instance of that category,
e.g. `CFLAGS.readline`.
When the configure script exports flags for consumption by filtered
files, e.g. [Makefile.in][] and the generated
`sqlite_cfg.h`, it does so in the more conventional `X_Y` form because
those flags get exported as as C `#define`s to `sqlite_cfg.h`, where
dots are not permitted.
The `X.y` convention is used in the makefiles primarily because the
person who did the initial port finds that considerably easier on the
eyes and fingers. In practice, the `X_Y` form of such exports is used
exactly once in [Makefile.in][], where it's translated from `@X_Y@`
into into `X.y` form for consumption by [Makefile.in][] and
[main.mk][]. For example:
>
```
LDFLAGS.shobj = @SHOBJ_LDFLAGS@
LDFLAGS.zlib = @LDFLAGS_ZLIB@
LDFLAGS.math = @LDFLAGS_MATH@
```
(That first one is defined by autosetup, and thus applies "LDFLAGS" as
the suffix rather than the prefix. Which is more legible is a matter
of taste, for which there is no accounting.)
Do Not Update Global Shared State
------------------------------------------------------------------------
In both the legacy Autotools-driven build and common Autosetup usage,
feature tests performed by the configure script may amend global flags
such as `LIBS`, `LDFLAGS`, and `CFLAGS`[^as-cflags]. That's
appropriate for a makefile which builds a single deliverable, but less
so for makefiles which produce multiple deliverables. Drawbacks of
that approach include:
- It's unlikely that every single deliverable will require the same
core set of those flags.
- It can be difficult to determine the origin of any given change to
that global state because those changes are hidden behind voodoo
performed outside the immediate visibility of the configure script's
maintainer.
- It can force the maintainers of the configure script to place tests
in a specific order so that the resulting flags get applied at
the correct time and/or in the correct order.\
(A real-life example: before the approach described below was taken
to collecting build-time flags, the test for `-rpath` had to come
_after_ the test for zlib because the results of the `-rpath` test
implicitly modified global state which broke the zlib feature
test. Because the feature tests no longer (intentionally) modify
shared global state, that is not an issue.)
In this build, cases where feature tests modify global state in such a
way that it may impact later feature tests are either (A) very
intentionally defined to do so (e.g. the `--with-wasi-sdk` flag has
invasive side-effects) or (B) are oversights (i.e. bugs).
This tree's [configure script][auto.def], [utility APIs][proj.tcl],
[Makefile.in][], and [main.mk][] therefore strive to separate the
results of any given feature test into its own well-defined
variables. For example:
- The linker flags for zlib are exported from the configure script as
`LDFLAGS_ZLIB`, which [Makefile.in][] and [main.mk][] then expose as
`LDFLAGS.zlib`.
- `CFLAGS_READLINE` (a.k.a. `CFLAGS.readline`) contains the `CFLAGS`
needed for including `libreadline`, `libedit`, or `linenoise`, and
`LDFLAGS_READLINE` (a.k.a. `LDFLAGS.readline`) is its link-time
counterpart.
It is then up to the Makefile to apply and order the flags however is
appropriate.
At the end of the configure script, the global `CFLAGS` _ideally_
holds only flags which are either relevant to all targets or, failing
that, will have no unintended side-effects on any targets. That said:
clients frequently pass custom `CFLAGS` to `./configure` or `make` to
set library-level feature toggles, e.g. `-DSQLITE_OMIT_FOO`, in which
case there is no practical way to avoid "polluting" the builds of
arbitrary makefile targets with those. _C'est la vie._
[^as-cflags]: But see this article for a detailed discussion of how
autosetup currently deals specifically with CFLAGS:
<https://msteveb.github.io/autosetup/articles/handling-cflags/>
<a name="updating"></a>
Updating Autosetup
========================================================================
Updating autosetup is, more often than not, painless. It requires having
a checked-out copy of [the autosetup git repository][autosetup-git]:
>
```
$ git clone https://github.com/msteveb/autosetup
$ cd autosetup
# Or, if it's already checked out:
$ git pull
```
Then, from the top-most directory of an SQLite checkout:
>
```
$ /path/to/autosetup-checkout/autosetup --install .
$ fossil status # show the modified files
```
Unless the upgrade made any incompatible changes (which is exceedingly
rare), that's all there is to it. After that's done, **apply a patch
for the change described in the following section**, test the
configure process, and check it in.
<a name="patching"></a>
Patching Autosetup for Project-local Changes
------------------------------------------------------------------------
Autosetup reserves the flag name **`--debug`** for its own purposes,
and its own special handling of `--enable-...` flags makes `--debug`
an alias for `--enable-debug`. As this project has a long history of
using `--enable-debug`, we patch autosetup to use the name
`--autosetup-debug` in place of `--debug`. That requires (as of this
writing) four small edits in [](/file/autosetup/autosetup), as
demonstrated in [check-in 3296c8d3](/info/3296c8d3).
If autosetup is upgraded and this patch is _not_ applied the invoking
`./configure` will fail loudly because of the declaration of the
`debug` flag in `auto.def` - duplicated flags are not permitted.
<a name="branch-customization"></a>
Branch-specific Customization
========================================================================
Certain vendor-specific branches require slight configure script
customization. Rather than editing `sqlite-config.tcl` for this,
which frequently leads to merge conflicts, the following approach
is recommended:
In the vendor-specific branch, create a file named
`autosetup/sqlite-custom.tcl`.
That file should contain the following content...
If flag customization is required, add:
>
```
proc sqlite-custom-flags {} {
# If any existing --flags require different default values
# then call:
options-defaults {
flag-name new-default-value
...
}
# ^^^ That will replace the default value but will not update
# the --help text, which may lead to some confusion:
# https://github.com/msteveb/autosetup/issues/77
return {
{*} {
new-flag-name => {Help text}
...
}
}; #see below
}
```
That function must return either an empty string or a list in the form
used internally by `sqlite-config.tcl:sqlite-configure`.
Next, define:
>
```
proc sqlite-custom-handle-flags {} {
... do any custom flag handling here ...
}
```
That function, if defined, will be called relatively late in the
configure process, before any filtered files are generated but after
all other significant processing.
[Autosetup]: https://msteveb.github.io/autosetup/
[auto.def]: /file/auto.def
[autoconf/auto.def]: /file/autoconf/auto.def
[autosetup-git]: https://github.com/msteveb/autosetup
[proj.tcl]: /file/autosetup/proj.tcl
[sqlite-config.tcl]: /file/autosetup/sqlite-config.tcl
[Makefile.in]: /file/Makefile.in
[main.mk]: /file/main.mk
[JimTCL]: https://jim.tcl.tk
autosetup/autosetup
Executable
+2540
View File
File diff suppressed because it is too large. Load diff
autosetup/autosetup-config.guess
Executable
+1815
View File
File diff suppressed because it is too large. Load diff
autosetup/autosetup-config.sub
Executable
+2354
View File
File diff suppressed because it is too large. Load diff
autosetup/autosetup-find-tclsh
Executable
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
# Looks for a suitable tclsh or jimsh in the PATH
# If not found, builds a bootstrap jimsh in current dir from source
# Prefer $autosetup_tclsh if is set in the environment (unless ./jimsh0 works)
# If an argument is given, use that as the test instead of autosetup-test-tclsh
d="`dirname "$0"`"
for tclsh in ./jimsh0 $autosetup_tclsh jimsh tclsh tclsh8.5 tclsh8.6 tclsh8.7; do
{ $tclsh "$d/${1-autosetup-test-tclsh}"; } 2>/dev/null && exit 0
done
echo 1>&2 "No installed jimsh or tclsh, building local bootstrap jimsh0"
for cc in ${CC_FOR_BUILD:-cc} gcc; do
{ $cc -o jimsh0 "$d/jimsh0.c"; } 2>/dev/null >/dev/null || continue
./jimsh0 "$d/${1-autosetup-test-tclsh}" && exit 0
done
echo 1>&2 "No working C compiler found. Tried ${CC_FOR_BUILD:-cc} and gcc."
echo false
autosetup/autosetup-test-tclsh
+20
View File
@@ -0,0 +1,20 @@
# A small Tcl script to verify that the chosen
# interpreter works. Sometimes we might e.g. pick up
# an interpreter for a different arch.
# Outputs the full path to the interpreter
if {[catch {info version} version] == 0} {
# This is Jim Tcl
if {$version >= 0.72} {
# Ensure that regexp works
regexp (a.*?) a
puts [info nameofexecutable]
exit 0
}
} elseif {[catch {info tclversion} version] == 0} {
if {$version >= 8.5 && ![string match 8.5a* [info patchlevel]]} {
puts [info nameofexecutable]
exit 0
}
}
exit 1
autosetup/cc-db.tcl
+15
View File
@@ -0,0 +1,15 @@
# Copyright (c) 2011 WorkWare Systems http://www.workware.net.au/
# All rights reserved
# @synopsis:
#
# The 'cc-db' module provides a knowledge-base of system idiosyncrasies.
# In general, this module can always be included.
use cc
options {}
# openbsd needs sys/types.h to detect some system headers
cc-include-needs sys/socket.h sys/types.h
cc-include-needs netinet/in.h sys/types.h
autosetup/cc-lib.tcl
+187
View File
@@ -0,0 +1,187 @@
# Copyright (c) 2011 WorkWare Systems http://www.workware.net.au/
# All rights reserved
# @synopsis:
#
# Provides a library of common tests on top of the 'cc' module.
use cc
# @cc-check-lfs
#
# The equivalent of the 'AC_SYS_LARGEFILE' macro.
#
# defines 'HAVE_LFS' if LFS is available,
# and defines '_FILE_OFFSET_BITS=64' if necessary
#
# Returns 1 if 'LFS' is available or 0 otherwise
#
proc cc-check-lfs {} {
cc-check-includes sys/types.h
msg-checking "Checking if -D_FILE_OFFSET_BITS=64 is needed..."
set lfs 1
if {[msg-quiet cc-with {-includes sys/types.h} {cc-check-sizeof off_t}] == 8} {
msg-result no
} elseif {[msg-quiet cc-with {-includes sys/types.h -cflags -D_FILE_OFFSET_BITS=64} {cc-check-sizeof off_t}] == 8} {
define _FILE_OFFSET_BITS 64
msg-result yes
} else {
set lfs 0
msg-result none
}
define-feature lfs $lfs
return $lfs
}
# @cc-check-endian
#
# The equivalent of the 'AC_C_BIGENDIAN' macro.
#
# defines 'HAVE_BIG_ENDIAN' if endian is known to be big,
# or 'HAVE_LITTLE_ENDIAN' if endian is known to be little.
#
# Returns 1 if determined, or 0 if not.
#
proc cc-check-endian {} {
cc-check-includes sys/types.h sys/param.h
set rc 0
msg-checking "Checking endian..."
cc-with {-includes {sys/types.h sys/param.h}} {
if {[cctest -code {
#if !defined(BIG_ENDIAN) || !defined(BYTE_ORDER)
#error unknown
#elif BYTE_ORDER != BIG_ENDIAN
#error little
#endif
}]} {
define-feature big-endian
msg-result "big"
set rc 1
} elseif {[cctest -code {
#if !defined(LITTLE_ENDIAN) || !defined(BYTE_ORDER)
#error unknown
#elif BYTE_ORDER != LITTLE_ENDIAN
#error big
#endif
}]} {
define-feature little-endian
msg-result "little"
set rc 1
} else {
msg-result "unknown"
}
}
return $rc
}
# @cc-check-flags flag ?...?
#
# Checks whether the given C/C++ compiler flags can be used. Defines feature
# names prefixed with 'HAVE_CFLAG' and 'HAVE_CXXFLAG' respectively, and
# appends working flags to '-cflags' and 'AS_CFLAGS' or 'AS_CXXFLAGS'.
proc cc-check-flags {args} {
set result 1
array set opts [cc-get-settings]
switch -exact -- $opts(-lang) {
c++ {
set lang C++
set prefix CXXFLAG
}
c {
set lang C
set prefix CFLAG
}
default {
autosetup-error "cc-check-flags failed with unknown language: $opts(-lang)"
}
}
foreach flag $args {
msg-checking "Checking whether the $lang compiler accepts $flag..."
if {[cctest -cflags $flag]} {
msg-result yes
define-feature $prefix$flag
cc-with [list -cflags [list $flag]]
define-append AS_${prefix}S $flag
} else {
msg-result no
set result 0
}
}
return $result
}
# @cc-check-standards ver ?...?
#
# Checks whether the C/C++ compiler accepts one of the specified '-std=$ver'
# options, and appends the first working one to '-cflags' and 'AS_CFLAGS' or
# 'AS_CXXFLAGS'.
proc cc-check-standards {args} {
array set opts [cc-get-settings]
foreach std $args {
if {[cc-check-flags -std=$std]} {
return $std
}
}
return ""
}
# Checks whether $keyword is usable as alignof
proc cctest_alignof {keyword} {
msg-checking "Checking for $keyword..."
if {[cctest -code "int x = ${keyword}(char), y = ${keyword}('x');"]} then {
msg-result ok
define-feature $keyword
} else {
msg-result "not found"
}
}
# @cc-check-c11
#
# Checks for several C11/C++11 extensions and their alternatives. Currently
# checks for '_Static_assert', '_Alignof', '__alignof__', '__alignof'.
proc cc-check-c11 {} {
msg-checking "Checking for _Static_assert..."
if {[cctest -code {
_Static_assert(1, "static assertions are available");
}]} then {
msg-result ok
define-feature _Static_assert
} else {
msg-result "not found"
}
cctest_alignof _Alignof
cctest_alignof __alignof__
cctest_alignof __alignof
}
# @cc-check-alloca
#
# The equivalent of the 'AC_FUNC_ALLOCA' macro.
#
# Checks for the existence of 'alloca'
# defines 'HAVE_ALLOCA' and returns 1 if it exists.
proc cc-check-alloca {} {
cc-check-some-feature alloca {
cctest -includes alloca.h -code { alloca (2 * sizeof (int)); }
}
}
# @cc-signal-return-type
#
# The equivalent of the 'AC_TYPE_SIGNAL' macro.
#
# defines 'RETSIGTYPE' to 'int' or 'void'.
proc cc-signal-return-type {} {
msg-checking "Checking return type of signal handlers..."
cc-with {-includes {sys/types.h signal.h}} {
if {[cctest -code {return *(signal (0, 0)) (0) == 1;}]} {
set type int
} else {
set type void
}
define RETSIGTYPE $type
msg-result $type
}
}
autosetup/cc-shared.tcl
+113
View File
@@ -0,0 +1,113 @@
# Copyright (c) 2010 WorkWare Systems http://www.workware.net.au/
# All rights reserved
# @synopsis:
#
# The 'cc-shared' module provides support for shared libraries and shared objects.
# It defines the following variables:
#
## SH_CFLAGS Flags to use compiling sources destined for a shared library
## SH_LDFLAGS Flags to use linking (creating) a shared library
## SH_SOPREFIX Prefix to use to set the soname when creating a shared library
## SH_SOFULLPATH Set to 1 if the shared library soname should include the full install path
## SH_SOEXT Extension for shared libs
## SH_SOEXTVER Format for versioned shared libs - %s = version
## SHOBJ_CFLAGS Flags to use compiling sources destined for a shared object
## SHOBJ_LDFLAGS Flags to use linking a shared object, undefined symbols allowed
## SHOBJ_LDFLAGS_R - as above, but all symbols must be resolved
## SH_LINKRPATH Format for setting the rpath when linking an executable, %s = path
## SH_LINKFLAGS Flags to use linking an executable which will load shared objects
## LD_LIBRARY_PATH Environment variable which specifies path to shared libraries
## STRIPLIBFLAGS Arguments to strip a dynamic library
options {}
# Defaults: gcc on unix
define SHOBJ_CFLAGS -fPIC
define SHOBJ_LDFLAGS -shared
define SH_CFLAGS -fPIC
define SH_LDFLAGS -shared
define SH_LINKFLAGS -rdynamic
define SH_LINKRPATH "-Wl,-rpath -Wl,%s"
define SH_SOEXT .so
define SH_SOEXTVER .so.%s
define SH_SOPREFIX -Wl,-soname,
define LD_LIBRARY_PATH LD_LIBRARY_PATH
define STRIPLIBFLAGS --strip-unneeded
# Note: This is a helpful reference for identifying the toolchain
# http://sourceforge.net/apps/mediawiki/predef/index.php?title=Compilers
switch -glob -- [get-define host] {
*-*-darwin* {
define SHOBJ_CFLAGS "-dynamic -fno-common"
define SHOBJ_LDFLAGS "-bundle -undefined dynamic_lookup"
define SHOBJ_LDFLAGS_R -bundle
define SH_CFLAGS -dynamic
define SH_LDFLAGS -dynamiclib
define SH_LINKFLAGS ""
define SH_SOEXT .dylib
define SH_SOEXTVER .%s.dylib
define SH_SOPREFIX -Wl,-install_name,
define SH_SOFULLPATH
define LD_LIBRARY_PATH DYLD_LIBRARY_PATH
define STRIPLIBFLAGS -x
}
*-*-ming* - *-*-cygwin - *-*-msys {
define SHOBJ_CFLAGS ""
define SHOBJ_LDFLAGS -shared
define SH_CFLAGS ""
define SH_LDFLAGS -shared
define SH_LINKRPATH ""
define SH_LINKFLAGS ""
define SH_SOEXT .dll
define SH_SOEXTVER .dll
define SH_SOPREFIX ""
define LD_LIBRARY_PATH PATH
}
sparc* {
if {[msg-quiet cc-check-decls __SUNPRO_C]} {
msg-result "Found sun stdio compiler"
# sun stdio compiler
# XXX: These haven't been fully tested.
define SHOBJ_CFLAGS -KPIC
define SHOBJ_LDFLAGS "-G"
define SH_CFLAGS -KPIC
define SH_LINKFLAGS -Wl,-export-dynamic
define SH_SOPREFIX -Wl,-h,
}
}
*-*-solaris* {
if {[msg-quiet cc-check-decls __SUNPRO_C]} {
msg-result "Found sun stdio compiler"
# sun stdio compiler
# XXX: These haven't been fully tested.
define SHOBJ_CFLAGS -KPIC
define SHOBJ_LDFLAGS "-G"
define SH_CFLAGS -KPIC
define SH_LINKFLAGS -Wl,-export-dynamic
define SH_SOPREFIX -Wl,-h,
}
}
*-*-hpux {
# XXX: These haven't been tested
define SHOBJ_CFLAGS "+O3 +z"
define SHOBJ_LDFLAGS -b
define SH_CFLAGS +z
define SH_LINKFLAGS -Wl,+s
define LD_LIBRARY_PATH SHLIB_PATH
}
*-*-haiku {
define SHOBJ_CFLAGS ""
define SHOBJ_LDFLAGS -shared
define SH_CFLAGS ""
define SH_LDFLAGS -shared
define SH_LINKFLAGS ""
define SH_SOPREFIX ""
define LD_LIBRARY_PATH LIBRARY_PATH
}
}
if {![is-defined SHOBJ_LDFLAGS_R]} {
define SHOBJ_LDFLAGS_R [get-define SHOBJ_LDFLAGS]
}
autosetup/cc.tcl
+758
View File
@@ -0,0 +1,758 @@
# Copyright (c) 2010 WorkWare Systems http://www.workware.net.au/
# All rights reserved
# @synopsis:
#
# The 'cc' module supports checking various 'features' of the C or C++
# compiler/linker environment. Common commands are 'cc-check-includes',
# 'cc-check-types', 'cc-check-functions', 'cc-with' and 'make-config-header'
#
# The following environment variables are used if set:
#
## CC - C compiler
## CXX - C++ compiler
## CPP - C preprocessor
## CCACHE - Set to "none" to disable automatic use of ccache
## CPPFLAGS - Additional C preprocessor compiler flags (C and C++), before CFLAGS, CXXFLAGS
## CFLAGS - Additional C compiler flags
## CXXFLAGS - Additional C++ compiler flags
## LDFLAGS - Additional compiler flags during linking
## LINKFLAGS - ?How is this different from LDFLAGS?
## LIBS - Additional libraries to use (for all tests)
## CROSS - Tool prefix for cross compilation
#
# The following variables are defined from the corresponding
# environment variables if set.
#
## CC_FOR_BUILD
## LD
use system
options {}
# Checks for the existence of the given function by linking
#
proc cctest_function {function} {
cctest -link 1 -declare "extern void $function\(void);" -code "$function\();"
}
# Checks for the existence of the given type by compiling
proc cctest_type {type} {
cctest -code "$type _x;"
}
# Checks for the existence of the given type/structure member.
# e.g. "struct stat.st_mtime"
proc cctest_member {struct_member} {
# split at the first dot
regexp {^([^.]+)[.](.*)$} $struct_member -> struct member
cctest -code "static $struct _s; return sizeof(_s.$member);"
}
# Checks for the existence of the given define by compiling
#
proc cctest_define {name} {
cctest -code "#ifndef $name\n#error not defined\n#endif"
}
# Checks for the existence of the given name either as
# a macro (#define) or an rvalue (such as an enum)
#
proc cctest_decl {name} {
cctest -code "#ifndef $name\n(void)$name;\n#endif"
}
# @cc-check-sizeof type ...
#
# Checks the size of the given types (between 1 and 32, inclusive).
# Defines a variable with the size determined, or 'unknown' otherwise.
# e.g. for type 'long long', defines 'SIZEOF_LONG_LONG'.
# Returns the size of the last type.
#
proc cc-check-sizeof {args} {
foreach type $args {
msg-checking "Checking for sizeof $type..."
set size unknown
# Try the most common sizes first
foreach i {4 8 1 2 16 32} {
if {[cctest -code "static int _x\[sizeof($type) == $i ? 1 : -1\] = { 1 };"]} {
set size $i
break
}
}
msg-result $size
set define [feature-define-name $type SIZEOF_]
define $define $size
}
# Return the last result
get-define $define
}
# Checks for each feature in $list by using the given script.
#
# When the script is evaluated, $each is set to the feature
# being checked, and $extra is set to any additional cctest args.
#
# Returns 1 if all features were found, or 0 otherwise.
proc cc-check-some-feature {list script} {
set ret 1
foreach each $list {
if {![check-feature $each $script]} {
set ret 0
}
}
return $ret
}
# @cc-check-includes includes ...
#
# Checks that the given include files can be used.
proc cc-check-includes {args} {
cc-check-some-feature $args {
set with {}
if {[dict exists $::autosetup(cc-include-deps) $each]} {
set deps [dict keys [dict get $::autosetup(cc-include-deps) $each]]
msg-quiet cc-check-includes {*}$deps
foreach i $deps {
if {[have-feature $i]} {
lappend with $i
}
}
}
if {[llength $with]} {
cc-with [list -includes $with] {
cctest -includes $each
}
} else {
cctest -includes $each
}
}
}
# @cc-include-needs include required ...
#
# Ensures that when checking for '$include', a check is first
# made for each '$required' file, and if found, it is included with '#include'.
proc cc-include-needs {file args} {
foreach depfile $args {
dict set ::autosetup(cc-include-deps) $file $depfile 1
}
}
# @cc-check-types type ...
#
# Checks that the types exist.
proc cc-check-types {args} {
cc-check-some-feature $args {
cctest_type $each
}
}
# @cc-check-defines define ...
#
# Checks that the given preprocessor symbols are defined.
proc cc-check-defines {args} {
cc-check-some-feature $args {
cctest_define $each
}
}
# @cc-check-decls name ...
#
# Checks that each given name is either a preprocessor symbol or rvalue
# such as an enum. Note that the define used is 'HAVE_DECL_xxx'
# rather than 'HAVE_xxx'.
proc cc-check-decls {args} {
set ret 1
foreach name $args {
msg-checking "Checking for $name..."
set r [cctest_decl $name]
define-feature "decl $name" $r
if {$r} {
msg-result "ok"
} else {
msg-result "not found"
set ret 0
}
}
return $ret
}
# @cc-check-functions function ...
#
# Checks that the given functions exist (can be linked).
proc cc-check-functions {args} {
cc-check-some-feature $args {
cctest_function $each
}
}
# @cc-check-members type.member ...
#
# Checks that the given type/structure members exist.
# A structure member is of the form 'struct stat.st_mtime'.
proc cc-check-members {args} {
cc-check-some-feature $args {
cctest_member $each
}
}
# @cc-check-function-in-lib function libs ?otherlibs?
#
# Checks that the given function can be found in one of the libs.
#
# First checks for no library required, then checks each of the libraries
# in turn.
#
# If the function is found, the feature is defined and 'lib_$function' is defined
# to '-l$lib' where the function was found, or "" if no library required.
# In addition, '-l$lib' is prepended to the 'LIBS' define.
#
# If additional libraries may be needed for linking, they should be specified
# with '$extralibs' as '-lotherlib1 -lotherlib2'.
# These libraries are not automatically added to 'LIBS'.
#
# Returns 1 if found or 0 if not.
#
proc cc-check-function-in-lib {function libs {otherlibs {}}} {
msg-checking "Checking libs for $function..."
set found 0
cc-with [list -libs $otherlibs] {
if {[cctest_function $function]} {
msg-result "none needed"
define lib_$function ""
incr found
} else {
foreach lib $libs {
cc-with [list -libs -l$lib] {
if {[cctest_function $function]} {
msg-result -l$lib
define lib_$function -l$lib
# prepend to LIBS
define LIBS "-l$lib [get-define LIBS]"
incr found
break
}
}
}
}
}
define-feature $function $found
if {!$found} {
msg-result "no"
}
return $found
}
# @cc-check-tools tool ...
#
# Checks for existence of the given compiler tools, taking
# into account any cross compilation prefix.
#
# For example, when checking for 'ar', first 'AR' is checked on the command
# line and then in the environment. If not found, '${host}-ar' or
# simply 'ar' is assumed depending upon whether cross compiling.
# The path is searched for this executable, and if found 'AR' is defined
# to the executable name.
# Note that even when cross compiling, the simple 'ar' is used as a fallback,
# but a warning is generated. This is necessary for some toolchains.
#
# It is an error if the executable is not found.
#
proc cc-check-tools {args} {
foreach tool $args {
set TOOL [string toupper $tool]
set exe [get-env $TOOL [get-define cross]$tool]
if {[find-executable $exe]} {
define $TOOL $exe
continue
}
if {[find-executable $tool]} {
msg-result "Warning: Failed to find $exe, falling back to $tool which may be incorrect"
define $TOOL $tool
continue
}
user-error "Failed to find $exe"
}
}
# @cc-check-progs prog ...
#
# Checks for existence of the given executables on the path.
#
# For example, when checking for 'grep', the path is searched for
# the executable, 'grep', and if found 'GREP' is defined as 'grep'.
#
# If the executable is not found, the variable is defined as 'false'.
# Returns 1 if all programs were found, or 0 otherwise.
#
proc cc-check-progs {args} {
set failed 0
foreach prog $args {
set PROG [string toupper $prog]
msg-checking "Checking for $prog..."
if {![find-executable $prog]} {
msg-result no
define $PROG false
incr failed
} else {
msg-result ok
define $PROG $prog
}
}
expr {!$failed}
}
# @cc-path-progs prog ...
#
# Like cc-check-progs, but sets the define to the full path rather
# than just the program name.
#
proc cc-path-progs {args} {
set failed 0
foreach prog $args {
set PROG [string toupper $prog]
msg-checking "Checking for $prog..."
set path [find-executable-path $prog]
if {$path eq ""} {
msg-result no
define $PROG false
incr failed
} else {
msg-result $path
define $PROG $path
}
}
expr {!$failed}
}
# Adds the given settings to $::autosetup(ccsettings) and
# returns the old settings.
#
proc cc-add-settings {settings} {
if {[llength $settings] % 2} {
autosetup-error "settings list is missing a value: $settings"
}
set prev [cc-get-settings]
# workaround a bug in some versions of jimsh by forcing
# conversion of $prev to a list
llength $prev
array set new $prev
foreach {name value} $settings {
switch -exact -- $name {
-cflags - -includes {
# These are given as lists
lappend new($name) {*}[list-non-empty $value]
}
-declare {
lappend new($name) $value
}
-libs {
# Note that new libraries are added before previous libraries
set new($name) [list {*}[list-non-empty $value] {*}$new($name)]
}
-link - -lang - -nooutput {
set new($name) $value
}
-source - -sourcefile - -code {
# XXX: These probably are only valid directly from cctest
set new($name) $value
}
default {
autosetup-error "unknown cctest setting: $name"
}
}
}
cc-store-settings [array get new]
return $prev
}
proc cc-store-settings {new} {
set ::autosetup(ccsettings) $new
}
proc cc-get-settings {} {
return $::autosetup(ccsettings)
}
# Similar to cc-add-settings, but each given setting
# simply replaces the existing value.
#
# Returns the previous settings
proc cc-update-settings {args} {
set prev [cc-get-settings]
cc-store-settings [dict merge $prev $args]
return $prev
}
# @cc-with settings ?{ script }?
#
# Sets the given 'cctest' settings and then runs the tests in '$script'.
# Note that settings such as '-lang' replace the current setting, while
# those such as '-includes' are appended to the existing setting.
#
# If no script is given, the settings become the default for the remainder
# of the 'auto.def' file.
#
## cc-with {-lang c++} {
## # This will check with the C++ compiler
## cc-check-types bool
## cc-with {-includes signal.h} {
## # This will check with the C++ compiler, signal.h and any existing includes.
## ...
## }
## # back to just the C++ compiler
## }
#
# The '-libs' setting is special in that newer values are added *before* earlier ones.
#
## cc-with {-libs {-lc -lm}} {
## cc-with {-libs -ldl} {
## cctest -libs -lsocket ...
## # libs will be in this order: -lsocket -ldl -lc -lm
## }
## }
#
# If you wish to invoke something like cc-check-flags but not have -cflags updated,
# use the following idiom:
#
## cc-with {} {
## cc-check-flags ...
## }
proc cc-with {settings args} {
if {[llength $args] == 0} {
cc-add-settings $settings
} elseif {[llength $args] > 1} {
autosetup-error "usage: cc-with settings ?script?"
} else {
set save [cc-add-settings $settings]
set rc [catch {uplevel 1 [lindex $args 0]} result info]
cc-store-settings $save
if {$rc != 0} {
return -code [dict get $info -code] $result
}
return $result
}
}
# @cctest ?settings?
#
# Low level C/C++ compiler checker. Compiles and or links a small C program
# according to the arguments and returns 1 if OK, or 0 if not.
#
# Supported settings are:
#
## -cflags cflags A list of flags to pass to the compiler
## -includes list A list of includes, e.g. {stdlib.h stdio.h}
## -declare code Code to declare before main()
## -link 1 Don't just compile, link too
## -lang c|c++ Use the C (default) or C++ compiler
## -libs liblist List of libraries to link, e.g. {-ldl -lm}
## -code code Code to compile in the body of main()
## -source code Compile a complete program. Ignore -includes, -declare and -code
## -sourcefile file Shorthand for -source [readfile [get-define srcdir]/$file]
## -nooutput 1 Treat any compiler output (e.g. a warning) as an error
#
# Unless '-source' or '-sourcefile' is specified, the C program looks like:
#
## #include <firstinclude> /* same for remaining includes in the list */
## declare-code /* any code in -declare, verbatim */
## int main(void) {
## code /* any code in -code, verbatim */
## return 0;
## }
#
# And the command line looks like:
#
## CC -cflags CFLAGS CPPFLAGS conftest.c -o conftest.o
## CXX -cflags CXXFLAGS CPPFLAGS conftest.cpp -o conftest.o
#
# And if linking:
#
## CC LDFLAGS -cflags CFLAGS conftest.c -o conftest -libs LIBS
## CXX LDFLAGS -cflags CXXFLAGS conftest.c -o conftest -libs LIBS
#
# Any failures are recorded in 'config.log'
#
proc cctest {args} {
set tmp conftest__
# Easiest way to merge in the settings
cc-with $args {
array set opts [cc-get-settings]
}
if {[info exists opts(-sourcefile)]} {
set opts(-source) [readfile [get-define srcdir]/$opts(-sourcefile) "#error can't find $opts(-sourcefile)"]
}
if {[info exists opts(-source)]} {
set lines $opts(-source)
} else {
foreach i $opts(-includes) {
if {$opts(-code) ne "" && ![feature-checked $i]} {
# Compiling real code with an unchecked header file
# Quickly (and silently) check for it now
# Remove all -includes from settings before checking
set saveopts [cc-update-settings -includes {}]
msg-quiet cc-check-includes $i
cc-store-settings $saveopts
}
if {$opts(-code) eq "" || [have-feature $i]} {
lappend source "#include <$i>"
}
}
lappend source {*}$opts(-declare)
lappend source "int main(void) {"
lappend source $opts(-code)
lappend source "return 0;"
lappend source "}"
set lines [join $source \n]
}
# Build the command line
set cmdline {}
lappend cmdline {*}[get-define CCACHE]
switch -exact -- $opts(-lang) {
c++ {
set src conftest__.cpp
lappend cmdline {*}[get-define CXX]
set cflags [get-define CXXFLAGS]
}
c {
set src conftest__.c
lappend cmdline {*}[get-define CC]
set cflags [get-define CFLAGS]
}
default {
autosetup-error "cctest called with unknown language: $opts(-lang)"
}
}
if {$opts(-link)} {
lappend cmdline {*}[get-define LDFLAGS]
} else {
lappend cflags {*}[get-define CPPFLAGS]
set tmp conftest__.o
lappend cmdline -c
}
lappend cmdline {*}$opts(-cflags) {*}[get-define cc-default-debug ""] {*}$cflags
lappend cmdline $src -o $tmp
if {$opts(-link)} {
lappend cmdline {*}$opts(-libs) {*}[get-define LIBS]
}
# At this point we have the complete command line and the
# complete source to be compiled. Get the result from cache if
# we can
if {[info exists ::cc_cache($cmdline,$lines)]} {
msg-checking "(cached) "
set ok $::cc_cache($cmdline,$lines)
if {$::autosetup(debug)} {
configlog "From cache (ok=$ok): [join $cmdline]"
configlog "============"
configlog $lines
configlog "============"
}
return $ok
}
writefile $src $lines\n
set ok 1
set err [catch {exec-with-stderr {*}$cmdline} result errinfo]
if {$err || ($opts(-nooutput) && [string length $result])} {
configlog "Failed: [join $cmdline]"
configlog $result
configlog "============"
configlog "The failed code was:"
configlog $lines
configlog "============"
set ok 0
} elseif {$::autosetup(debug)} {
configlog "Compiled OK: [join $cmdline]"
configlog "============"
configlog $lines
configlog "============"
}
file delete $src
file delete $tmp
# cache it
set ::cc_cache($cmdline,$lines) $ok
return $ok
}
# @make-autoconf-h outfile ?auto-patterns=HAVE_*? ?bare-patterns=SIZEOF_*?
#
# Deprecated - see 'make-config-header'
proc make-autoconf-h {file {autopatterns {HAVE_*}} {barepatterns {SIZEOF_* HAVE_DECL_*}}} {
user-notice "*** make-autoconf-h is deprecated -- use make-config-header instead"
make-config-header $file -auto $autopatterns -bare $barepatterns
}
# @make-config-header outfile ?-auto patternlist? ?-bare patternlist? ?-none patternlist? ?-str patternlist? ...
#
# Examines all defined variables which match the given patterns
# and writes an include file, '$file', which defines each of these.
# Variables which match '-auto' are output as follows:
# - defines which have the value '0' are ignored.
# - defines which have integer values are defined as the integer value.
# - any other value is defined as a string, e.g. '"value"'
# Variables which match '-bare' are defined as-is.
# Variables which match '-str' are defined as a string, e.g. '"value"'
# Variables which match '-none' are omitted.
#
# Note that order is important. The first pattern that matches is selected.
# Default behaviour is:
#
## -bare {SIZEOF_* HAVE_DECL_*} -auto HAVE_* -none *
#
# If the file would be unchanged, it is not written.
proc make-config-header {file args} {
set guard _[string toupper [regsub -all {[^a-zA-Z0-9]} [file tail $file] _]]
file mkdir [file dirname $file]
set lines {}
lappend lines "#ifndef $guard"
lappend lines "#define $guard"
# Add some defaults
lappend args -bare {SIZEOF_* HAVE_DECL_*} -auto HAVE_*
foreach n [lsort [dict keys [all-defines]]] {
set value [get-define $n]
set type [calc-define-output-type $n $args]
switch -exact -- $type {
-bare {
# Just output the value unchanged
}
-none {
continue
}
-str {
set value \"[string map [list \\ \\\\ \" \\\"] $value]\"
}
-auto {
# Automatically determine the type
if {$value eq "0"} {
lappend lines "/* #undef $n */"
continue
}
if {![string is integer -strict $value]} {
set value \"[string map [list \\ \\\\ \" \\\"] $value]\"
}
}
"" {
continue
}
default {
autosetup-error "Unknown type in make-config-header: $type"
}
}
lappend lines "#define $n $value"
}
lappend lines "#endif"
set buf [join $lines \n]
write-if-changed $file $buf {
msg-result "Created $file"
}
}
proc calc-define-output-type {name spec} {
foreach {type patterns} $spec {
foreach pattern $patterns {
if {[string match $pattern $name]} {
return $type
}
}
}
return ""
}
proc cc-init {} {
global autosetup
# Initialise some values from the environment or commandline or default settings
foreach i {LDFLAGS LIBS CPPFLAGS LINKFLAGS CFLAGS} {
lassign $i var default
define $var [get-env $var $default]
}
if {[env-is-set CC]} {
# Set by the user, so don't try anything else
set try [list [get-env CC ""]]
} else {
# Try some reasonable options
set try [list [get-define cross]cc [get-define cross]gcc]
}
define CC [find-an-executable {*}$try]
if {[get-define CC] eq ""} {
user-error "Could not find a C compiler. Tried: [join $try ", "]"
}
define CPP [get-env CPP "[get-define CC] -E"]
# XXX: Could avoid looking for a C++ compiler until requested
# If CXX isn't found, it is set to the empty string.
if {[env-is-set CXX]} {
define CXX [find-an-executable -required [get-env CXX ""]]
} else {
define CXX [find-an-executable [get-define cross]c++ [get-define cross]g++]
}
# CXXFLAGS default to CFLAGS if not specified
define CXXFLAGS [get-env CXXFLAGS [get-define CFLAGS]]
# May need a CC_FOR_BUILD, so look for one
define CC_FOR_BUILD [find-an-executable [get-env CC_FOR_BUILD ""] cc gcc false]
# These start empty and never come from the user or environment
define AS_CFLAGS ""
define AS_CPPFLAGS ""
define AS_CXXFLAGS ""
define CCACHE [find-an-executable [get-env CCACHE ccache]]
# If any of these are set in the environment, propagate them to the AUTOREMAKE commandline
foreach i {CC CXX CCACHE CPP CFLAGS CXXFLAGS CXXFLAGS LDFLAGS LIBS CROSS CPPFLAGS LINKFLAGS CC_FOR_BUILD LD} {
if {[env-is-set $i]} {
# Note: If the variable is set on the command line, get-env will return that value
# so the command line will continue to override the environment
define-append-argv AUTOREMAKE $i=[get-env $i ""]
}
}
# Initial cctest settings
cc-store-settings {-cflags {} -includes {} -declare {} -link 0 -lang c -libs {} -code {} -nooutput 0}
set autosetup(cc-include-deps) {}
msg-result "C compiler...[get-define CCACHE] [get-define CC] [get-define CFLAGS] [get-define CPPFLAGS]"
if {[get-define CXX] ne "false"} {
msg-result "C++ compiler...[get-define CCACHE] [get-define CXX] [get-define CXXFLAGS] [get-define CPPFLAGS]"
}
msg-result "Build C compiler...[get-define CC_FOR_BUILD]"
# On Darwin, we prefer to use -g0 to avoid creating .dSYM directories
# but some compilers may not support it, so test here.
switch -glob -- [get-define host] {
*-*-darwin* {
if {[cctest -cflags {-g0}]} {
define cc-default-debug -g0
}
}
}
if {![cc-check-includes stdlib.h]} {
user-error "Compiler does not work. See config.log"
}
}
cc-init
autosetup/find_tclconfig.tcl
+24
View File
@@ -0,0 +1,24 @@
#
# Run this TCL script to find and print the pathname for the tclConfig.sh
# file. Used by ../configure
#
if {[catch {
set libdir [tcl::pkgconfig get libdir,install]
}]} {
puts stderr "tclsh too old: does not support tcl::pkgconfig"
exit 1
}
if {![file exists $libdir]} {
puts stderr "tclsh reported library directory \"$libdir\" does not exist"
exit 1
}
if {![file exists $libdir/tclConfig.sh]} {
set n1 $libdir/tcl$::tcl_version
if {[file exists $n1/tclConfig.sh]} {
set libdir $n1
} else {
puts stderr "cannot find tclConfig.sh in either $libdir or $n1"
exit 1
}
}
puts $libdir
autosetup/jimsh0.c
+24519
View File
File diff suppressed because it is too large. Load diff
autosetup/pkg-config.tcl
+168
View File
@@ -0,0 +1,168 @@
# Copyright (c) 2016 WorkWare Systems http://www.workware.net.au/
# All rights reserved
# @synopsis:
#
# The 'pkg-config' module allows package information to be found via 'pkg-config'.
#
# If not cross-compiling, the package path should be determined automatically
# by 'pkg-config'.
# If cross-compiling, the default package path is the compiler sysroot.
# If the C compiler doesn't support '-print-sysroot', the path can be supplied
# by the '--sysroot' option or by defining 'SYSROOT'.
#
# 'PKG_CONFIG' may be set to use an alternative to 'pkg-config'.
use cc
options {
sysroot:dir => "Override compiler sysroot for pkg-config search path"
}
# @pkg-config-init ?required?
#
# Initialises the 'pkg-config' system. Unless '$required' is set to 0,
# it is a fatal error if a usable 'pkg-config' is not found .
#
# This command will normally be called automatically as required,
# but it may be invoked explicitly if lack of 'pkg-config' is acceptable.
#
# Returns 1 if ok, or 0 if 'pkg-config' not found/usable (only if '$required' is 0).
#
proc pkg-config-init {{required 1}} {
if {[is-defined HAVE_PKG_CONFIG]} {
return [get-define HAVE_PKG_CONFIG]
}
set found 0
define PKG_CONFIG [get-env PKG_CONFIG pkg-config]
msg-checking "Checking for pkg-config..."
if {[catch {exec [get-define PKG_CONFIG] --version} version]} {
msg-result "[get-define PKG_CONFIG] (not found)"
if {$required} {
user-error "No usable pkg-config"
}
} else {
msg-result $version
define PKG_CONFIG_VERSION $version
set found 1
if {[opt-str sysroot o]} {
define SYSROOT [file-normalize $o]
msg-result "Using specified sysroot [get-define SYSROOT]"
} elseif {[get-define build] ne [get-define host]} {
if {[catch {exec-with-stderr {*}[get-define CC] -print-sysroot} result errinfo] == 0} {
# Use the compiler sysroot, if there is one
define SYSROOT $result
msg-result "Found compiler sysroot $result"
} else {
configlog "[get-define CC] -print-sysroot: $result"
set msg "pkg-config: Cross compiling, but no compiler sysroot and no --sysroot supplied"
if {$required} {
user-error $msg
} else {
msg-result $msg
}
set found 0
}
}
if {[is-defined SYSROOT]} {
set sysroot [get-define SYSROOT]
# XXX: It's possible that these should be set only when invoking pkg-config
global env
set env(PKG_CONFIG_DIR) ""
# Supposedly setting PKG_CONFIG_LIBDIR means that PKG_CONFIG_PATH is ignored,
# but it doesn't seem to work that way in practice
set env(PKG_CONFIG_PATH) ""
# Do we need to try /usr/local as well or instead?
set env(PKG_CONFIG_LIBDIR) $sysroot/usr/lib/pkgconfig:$sysroot/usr/share/pkgconfig
set env(PKG_CONFIG_SYSROOT_DIR) $sysroot
}
}
define HAVE_PKG_CONFIG $found
return $found
}
# @pkg-config module ?requirements?
#
# Use 'pkg-config' to find the given module meeting the given requirements.
# e.g.
#
## pkg-config pango >= 1.37.0
#
# If found, returns 1 and sets 'HAVE_PKG_PANGO' to 1 along with:
#
## PKG_PANGO_VERSION to the found version
## PKG_PANGO_LIBS to the required libs (--libs-only-l)
## PKG_PANGO_LDFLAGS to the required linker flags (--libs-only-L)
## PKG_PANGO_CFLAGS to the required compiler flags (--cflags)
#
# If not found, returns 0.
#
proc pkg-config {module args} {
set ok [pkg-config-init]
msg-checking "Checking for $module $args..."
if {!$ok} {
msg-result "no pkg-config"
return 0
}
set pkgconfig [get-define PKG_CONFIG]
set ret [catch {exec $pkgconfig --modversion "$module $args"} version]
configlog "$pkgconfig --modversion $module $args: $version"
if {$ret} {
msg-result "not found"
return 0
}
# Sometimes --modversion succeeds but because of dependencies it isn't usable
# This seems to show up with --cflags
set ret [catch {exec $pkgconfig --cflags $module} cflags]
if {$ret} {
msg-result "unusable ($version - see config.log)"
configlog "$pkgconfig --cflags $module"
configlog $cflags
return 0
}
msg-result $version
set prefix [feature-define-name $module PKG_]
define HAVE_${prefix}
define ${prefix}_VERSION $version
define ${prefix}_CFLAGS $cflags
define ${prefix}_LIBS [exec $pkgconfig --libs-only-l $module]
define ${prefix}_LDFLAGS [exec $pkgconfig --libs-only-L $module]
return 1
}
# @pkg-config-get module setting
#
# Convenience access to the results of 'pkg-config'.
#
# For example, '[pkg-config-get pango CFLAGS]' returns
# the value of 'PKG_PANGO_CFLAGS', or '""' if not defined.
proc pkg-config-get {module name} {
set prefix [feature-define-name $module PKG_]
get-define ${prefix}_${name} ""
}
# @pkg-config-get-var module variable
#
# Return the value of the given variable from the given pkg-config module.
# The module must already have been successfully detected with pkg-config.
# e.g.
#
## if {[pkg-config harfbuzz >= 2.5]} {
## define harfbuzz_libdir [pkg-config-get-var harfbuzz libdir]
## }
#
# Returns the empty string if the variable isn't defined.
proc pkg-config-get-var {module variable} {
set pkgconfig [get-define PKG_CONFIG]
set prefix [feature-define-name $module HAVE_PKG_]
exec $pkgconfig $module --variable $variable
}
autosetup/proj.tcl
+2236
View File
File diff suppressed because it is too large. Load diff
autosetup/sqlite-config.tcl
+2174
View File
File diff suppressed because it is too large. Load diff
autosetup/system.tcl
+420
View File
@@ -0,0 +1,420 @@
# Copyright (c) 2010 WorkWare Systems http://www.workware.net.au/
# All rights reserved
# @synopsis:
#
# This module supports common system interrogation and options
# such as '--host', '--build', '--prefix', and setting 'srcdir', 'builddir', and 'EXEEXT'.
#
# It also support the "feature" naming convention, where searching
# for a feature such as 'sys/type.h' defines 'HAVE_SYS_TYPES_H'.
#
# It defines the following variables, based on '--prefix' unless overridden by the user:
#
## datadir
## sysconfdir
## sharedstatedir
## localstatedir
## infodir
## mandir
## includedir
#
# If '--prefix' is not supplied, it defaults to '/usr/local' unless 'options-defaults { prefix ... }' is used *before*
# including the 'system' module.
if {[is-defined defaultprefix]} {
user-notice "Note: defaultprefix is deprecated. Use options-defaults to set default options"
options-defaults [list prefix [get-define defaultprefix]]
}
options {
host:host-alias => {a complete or partial cpu-vendor-opsys for the system where
the application will run (defaults to the same value as --build)}
build:build-alias => {a complete or partial cpu-vendor-opsys for the system
where the application will be built (defaults to the
result of running config.guess)}
prefix:dir=/usr/local => {the target directory for the build (default: '@default@')}
# These (hidden) options are supported for autoconf/automake compatibility
exec-prefix:
bindir:
sbindir:
includedir:
mandir:
infodir:
libexecdir:
datadir:
libdir:
sysconfdir:
sharedstatedir:
localstatedir:
runstatedir:
maintainer-mode=0
dependency-tracking=0
silent-rules=0
program-prefix:
program-suffix:
program-transform-name:
x-includes:
x-libraries:
}
# @check-feature name { script }
#
# defines feature '$name' to the return value of '$script',
# which should be 1 if found or 0 if not found.
#
# e.g. the following will define 'HAVE_CONST' to 0 or 1.
#
## check-feature const {
## cctest -code {const int _x = 0;}
## }
proc check-feature {name code} {
msg-checking "Checking for $name..."
set r [uplevel 1 $code]
define-feature $name $r
if {$r} {
msg-result "ok"
} else {
msg-result "not found"
}
return $r
}
# @have-feature name ?default=0?
#
# Returns the value of feature '$name' if defined, or '$default' if not.
#
# See 'feature-define-name' for how the "feature" name
# is translated into the "define" name.
#
proc have-feature {name {default 0}} {
get-define [feature-define-name $name] $default
}
# @define-feature name ?value=1?
#
# Sets the feature 'define' to '$value'.
#
# See 'feature-define-name' for how the "feature" name
# is translated into the "define" name.
#
proc define-feature {name {value 1}} {
define [feature-define-name $name] $value
}
# @feature-checked name
#
# Returns 1 if feature '$name' has been checked, whether true or not.
#
proc feature-checked {name} {
is-defined [feature-define-name $name]
}
# @feature-define-name name ?prefix=HAVE_?
#
# Converts a "feature" name to the corresponding "define",
# e.g. 'sys/stat.h' becomes 'HAVE_SYS_STAT_H'.
#
# Converts '*' to 'P' and all non-alphanumeric to underscore.
#
proc feature-define-name {name {prefix HAVE_}} {
string toupper $prefix[regsub -all {[^a-zA-Z0-9]} [regsub -all {[*]} $name p] _]
}
# @write-if-changed filename contents ?script?
#
# If '$filename' doesn't exist, or it's contents are different to '$contents',
# the file is written and '$script' is evaluated.
#
# Otherwise a "file is unchanged" message is displayed.
proc write-if-changed {file buf {script {}}} {
set old [readfile $file ""]
if {$old eq $buf && [file exists $file]} {
msg-result "$file is unchanged"
} else {
writefile $file $buf\n
uplevel 1 $script
}
}
# @include-file infile mapping
#
# The core of make-template, called recursively for each @include
# directive found within that template so that this proc's result
# is the fully-expanded template.
#
# The mapping parameter is how we expand @varname@ within the template.
# We do that inline within this step only for @include directives which
# can have variables in the filename arg. A separate substitution pass
# happens when this recursive function returns, expanding the rest of
# the variables.
#
proc include-file {infile mapping} {
# A stack of true/false conditions, one for each nested conditional
# starting with "true"
set condstack {1}
set result {}
set linenum 0
foreach line [split [readfile $infile] \n] {
incr linenum
if {[regexp {^@(if|else|endif)(\s*)(.*)} $line -> condtype condspace condargs]} {
if {$condtype eq "if"} {
if {[string length $condspace] == 0} {
autosetup-error "$infile:$linenum: Invalid expression: $line"
}
if {[llength $condargs] == 1} {
# ABC => [get-define ABC] ni {0 ""}
# !ABC => [get-define ABC] in {0 ""}
lassign $condargs condvar
if {[regexp {^!(.*)} $condvar -> condvar]} {
set op in
} else {
set op ni
}
set condexpr "\[[list get-define $condvar]\] $op {0 {}}"
} else {
# Translate alphanumeric ABC into [get-define ABC] and leave the
# rest of the expression untouched
regsub -all {([A-Z][[:alnum:]_]*)} $condargs {[get-define \1]} condexpr
}
if {[catch [list expr $condexpr] condval]} {
dputs $condval
autosetup-error "$infile:$linenum: Invalid expression: $line"
}
dputs "@$condtype: $condexpr => $condval"
}
if {$condtype ne "if"} {
if {[llength $condstack] <= 1} {
autosetup-error "$infile:$linenum: Error: @$condtype missing @if"
} elseif {[string length $condargs] && [string index $condargs 0] ne "#"} {
autosetup-error "$infile:$linenum: Error: Extra arguments after @$condtype"
}
}
switch -exact $condtype {
if {
# push condval
lappend condstack $condval
}
else {
# Toggle the last entry
set condval [lpop condstack]
set condval [expr {!$condval}]
lappend condstack $condval
}
endif {
if {[llength $condstack] == 0} {
user-notice "$infile:$linenum: Error: @endif missing @if"
}
lpop condstack
}
}
continue
}
# Only continue if the stack contains all "true"
if {"0" in $condstack} {
continue
}
if {[regexp {^@include\s+(.*)} $line -> filearg]} {
set incfile [string map $mapping $filearg]
if {[file exists $incfile]} {
lappend ::autosetup(deps) [file-normalize $incfile]
lappend result {*}[include-file $incfile $mapping]
} else {
user-error "$infile:$linenum: Include file $incfile is missing"
}
continue
}
if {[regexp {^@define\s+(\w+)\s+(.*)} $line -> var val]} {
define $var $val
continue
}
lappend result $line
}
return $result
}
# @make-template template ?outfile?
#
# Reads the input file '<srcdir>/$template' and writes the output file '$outfile'
# (unless unchanged).
# If '$outfile' is blank/omitted, '$template' should end with '.in' which
# is removed to create the output file name.
#
# Each pattern of the form '@define@' is replaced with the corresponding
# "define", if it exists, or left unchanged if not.
#
# The special value '@srcdir@' is substituted with the relative
# path to the source directory from the directory where the output
# file is created, while the special value '@top_srcdir@' is substituted
# with the relative path to the top level source directory.
#
# Conditional sections may be specified as follows:
## @if NAME eq "value"
## lines
## @else
## lines
## @endif
#
# Where 'NAME' is a defined variable name and '@else' is optional.
# Note that variables names *must* start with an uppercase letter.
# If the expression does not match, all lines through '@endif' are ignored.
#
# The alternative forms may also be used:
## @if NAME (true if the variable is defined, but not empty and not "0")
## @if !NAME (opposite of the form above)
## @if <general-tcl-expression>
#
# In the general Tcl expression, any words beginning with an uppercase letter
# are translated into [get-define NAME]
#
# Expressions may be nested
#
proc make-template {template {out {}}} {
set infile [file join $::autosetup(srcdir) $template]
if {![file exists $infile]} {
user-error "Template $template is missing"
}
# Define this as late as possible
define AUTODEPS $::autosetup(deps)
if {$out eq ""} {
if {[file ext $template] ne ".in"} {
autosetup-error "make_template $template has no target file and can't guess"
}
set out [file rootname $template]
}
set outdir [file dirname $out]
# Make sure the directory exists
file mkdir $outdir
# Set up srcdir and top_srcdir to be relative to the target dir
define srcdir [relative-path [file join $::autosetup(srcdir) $outdir] $outdir]
define top_srcdir [relative-path $::autosetup(srcdir) $outdir]
# Build map from global defines to their values so they can be
# substituted into @include file names.
proc build-define-mapping {} {
set mapping {}
foreach {n v} [array get ::define] {
lappend mapping @$n@ $v
}
return $mapping
}
set mapping [build-define-mapping]
set result [include-file $infile $mapping]
# Rebuild the define mapping in case we ran across @define
# directives in the template or a file it @included, then
# apply that mapping to the expanded template.
set mapping [build-define-mapping]
write-if-changed $out [string map $mapping [join $result \n]] {
msg-result "Created [relative-path $out] from [relative-path $template]"
}
}
proc system-init {} {
global autosetup
# build/host tuples and cross-compilation prefix
opt-str build build ""
define build_alias $build
if {$build eq ""} {
define build [config_guess]
} else {
define build [config_sub $build]
}
opt-str host host ""
define host_alias $host
if {$host eq ""} {
define host [get-define build]
set cross ""
} else {
define host [config_sub $host]
set cross $host-
}
define cross [get-env CROSS $cross]
# build/host _cpu, _vendor and _os
foreach type {build host} {
set v [get-define $type]
if {![regexp {^([^-]+)-([^-]+)-(.*)$} $v -> cpu vendor os]} {
user-error "Invalid canonical $type: $v"
}
define ${type}_cpu $cpu
define ${type}_vendor $vendor
define ${type}_os $os
}
opt-str prefix prefix /usr/local
# These are for compatibility with autoconf
define target [get-define host]
define prefix $prefix
define builddir $autosetup(builddir)
define srcdir $autosetup(srcdir)
define top_srcdir $autosetup(srcdir)
define abs_top_srcdir [file-normalize $autosetup(srcdir)]
define abs_top_builddir [file-normalize $autosetup(builddir)]
# autoconf supports all of these
define exec_prefix [opt-str exec-prefix exec_prefix $prefix]
foreach {name defpath} {
bindir /bin
sbindir /sbin
libexecdir /libexec
libdir /lib
} {
define $name [opt-str $name o $exec_prefix$defpath]
}
foreach {name defpath} {
datadir /share
sharedstatedir /com
infodir /share/info
mandir /share/man
includedir /include
} {
define $name [opt-str $name o $prefix$defpath]
}
if {$prefix ne {/usr}} {
opt-str sysconfdir sysconfdir $prefix/etc
} else {
opt-str sysconfdir sysconfdir /etc
}
define sysconfdir $sysconfdir
define localstatedir [opt-str localstatedir o /var]
define runstatedir [opt-str runstatedir o /run]
define SHELL [get-env SHELL [find-an-executable sh bash ksh]]
# These could be used to generate Makefiles following some automake conventions
define AM_SILENT_RULES [opt-bool silent-rules]
define AM_MAINTAINER_MODE [opt-bool maintainer-mode]
define AM_DEPENDENCY_TRACKING [opt-bool dependency-tracking]
# Windows vs. non-Windows
switch -glob -- [get-define host] {
*-*-ming* - *-*-cygwin - *-*-msys {
define-feature windows
define EXEEXT .exe
}
default {
define EXEEXT ""
}
}
# Display
msg-result "Host System...[get-define host]"
msg-result "Build System...[get-define build]"
}
system-init
autosetup/teaish/README.txt
+4
View File
@@ -0,0 +1,4 @@
The *.tcl files in this directory are part of the SQLite's "autoconf"
bundle which are specific to the TEA(-ish) build. During the tarball
generation process, they are copied into <TOP>/autoconf/autosetup/teaish
(which itself is created as part of that process).
autosetup/teaish/core.tcl
+2539
View File
File diff suppressed because it is too large. Load diff
autosetup/teaish/feature.tcl
+214
View File
@@ -0,0 +1,214 @@
########################################################################
# 2025 April 7
#
# The author disclaims copyright to this source code. In place of
# a legal notice, here is a blessing:
#
# * May you do good and not evil.
# * May you find forgiveness for yourself and forgive others.
# * May you share freely, never taking more than you give.
#
########################################################################
# ----- @module feature-tests.tcl -----
# @section TEA-ish collection of feature tests.
#
# Functions in this file with a prefix of teaish__ are
# private/internal APIs. Those with a prefix of teaish- are
# public APIs.
# @teaish-check-libz
#
# Checks for zlib.h and the function deflate in libz. If found,
# prepends -lz to the extension's ldflags and returns 1, else returns
# 0. It also defines LDFLAGS_LIBZ to the libs flag.
#
proc teaish-check-libz {} {
teaish-check-cached "Checking for libz" {
set rc 0
if {[msg-quiet cc-check-includes zlib.h] && [msg-quiet proj-check-function-in-lib deflate z]} {
teaish-ldflags-prepend [define LDFLAGS_LIBZ [get-define lib_deflate]]
undefine lib_deflate
incr rc
}
expr $rc
}
}
# @teaish-check-librt ?funclist?
#
# Checks whether -lrt is needed for any of the given functions. If
# so, appends -lrt via [teaish-ldflags-prepend] and returns 1, else
# returns 0. It also defines LDFLAGS_LIBRT to the libs flag or an
# empty string.
#
# Some systems (ex: SunOS) require -lrt in order to use nanosleep.
#
proc teaish-check-librt {{funclist {fdatasync nanosleep}}} {
teaish-check-cached -nostatus "Checking whether ($funclist) need librt" {
define LDFLAGS_LIBRT ""
foreach func $funclist {
if {[msg-quiet proj-check-function-in-lib $func rt]} {
set ldrt [get-define lib_${func}]
undefine lib_${func}
if {"" ne $ldrt} {
teaish-ldflags-prepend -r [define LDFLAGS_LIBRT $ldrt]
msg-result $ldrt
return 1
} else {
msg-result "no lib needed"
return 1
}
}
}
msg-result "not found"
return 0
}
}
# @teaish-check-stdint
#
# A thin proxy for [cc-with] which checks for <stdint.h> and the
# various fixed-size int types it declares. It defines HAVE_STDINT_T
# to 0 or 1 and (if it's 1) defines HAVE_XYZ_T for each XYZ int type
# to 0 or 1, depending on whether its available.
proc teaish-check-stdint {} {
teaish-check-cached "Checking for stdint.h" {
msg-quiet cc-with {-includes stdint.h} \
{cc-check-types int8_t int16_t int32_t int64_t intptr_t \
uint8_t uint16_t uint32_t uint64_t uintptr_t}
}
}
# @teaish-is-mingw
#
# Returns 1 if building for mingw, else 0.
proc teaish-is-mingw {} {
return [expr {
[string match *mingw* [get-define host]] &&
![file exists /dev/null]
}]
}
# @teaish-check-libdl
#
# Checks for whether dlopen() can be found and whether it requires
# -ldl for linking. If found, returns 1, defines LDFLAGS_DLOPEN to the
# linker flags (if any), and passes those flags to
# teaish-ldflags-prepend. It unconditionally defines HAVE_DLOPEN to 0
# or 1 (the its return result value).
proc teaish-check-dlopen {} {
teaish-check-cached -nostatus "Checking for dlopen()" {
set rc 0
set lfl ""
if {[cc-with {-includes dlfcn.h} {
cctest -link 1 -declare "extern char* dlerror(void);" -code "dlerror();"}]} {
msg-result "-ldl not needed"
incr rc
} elseif {[cc-check-includes dlfcn.h]} {
incr rc
if {[cc-check-function-in-lib dlopen dl]} {
set lfl [get-define lib_dlopen]
undefine lib_dlopen
msg-result " dlopen() needs $lfl"
} else {
msg-result " - dlopen() not found in libdl. Assuming dlopen() is built-in."
}
} else {
msg-result "not found"
}
teaish-ldflags-prepend [define LDFLAGS_DLOPEN $lfl]
define HAVE_DLOPEN $rc
}
}
#
# @teaish-check-libmath
#
# Handles the --enable-math flag. Returns 1 if found, else 0.
# If found, it prepends -lm (if needed) to the linker flags.
proc teaish-check-libmath {} {
teaish-check-cached "Checking for libc math library" {
set lfl ""
set rc 0
if {[msg-quiet proj-check-function-in-lib ceil m]} {
incr rc
set lfl [get-define lib_ceil]
undefine lib_ceil
teaish-ldflags-prepend $lfl
msg-checking "$lfl "
}
define LDFLAGS_LIBMATH $lfl
expr $rc
}
}
# @teaish-import-features ?-flags? feature-names...
#
# For each $name in feature-names... it invokes:
#
# use teaish/feature/$name
#
# to load TEAISH_AUTOSETUP_DIR/feature/$name.tcl
#
# By default, if a proc named teaish-check-${name}-options is defined
# after sourcing a file, it is called and its result is passed to
# proj-append-options. This can be suppressed with the -no-options
# flag.
#
# Flags:
#
# -no-options: disables the automatic running of
# teaish-check-NAME-options,
#
# -run: if the function teaish-check-NAME exists after importing
# then it is called. This flag must not be used when calling this
# function from teaish-options. This trumps both -pre and -post.
#
# -pre: if the function teaish-check-NAME exists after importing
# then it is passed to [teaish-checks-queue -pre].
#
# -post: works like -pre but instead uses[teaish-checks-queue -post].
proc teaish-import-features {args} {
set pk ""
set doOpt 1
proj-parse-simple-flags args flags {
-no-options 0 {set doOpt 0}
-run 0 {expr 1}
-pre 0 {set pk -pre}
-post 0 {set pk -post}
}
#
# TODO: never import the same module more than once. The "use"
# command is smart enough to not do that but we would need to
# remember whether or not any teaish-check-${arg}* procs have been
# called before, and skip them.
#
if {$flags(-run) && "" ne $pk} {
proj-error "Cannot use both -run and $pk" \
" (called from [proj-scope 1])"
}
foreach arg $args {
uplevel "use teaish/feature/$arg"
if {$doOpt} {
set n "teaish-check-${arg}-options"
if {[llength [info proc $n]] > 0} {
if {"" ne [set x [$n]]} {
options-add $x
}
}
}
if {$flags(-run)} {
set n "teaish-check-${arg}"
if {[llength [info proc $n]] > 0} {
uplevel 1 $n
}
} elseif {"" ne $pk} {
set n "teaish-check-${arg}"
if {[llength [info proc $n]] > 0} {
teaish-checks-queue {*}$pk $n
}
}
}
}
autosetup/teaish/tester.tcl
+228
View File
@@ -0,0 +1,228 @@
########################################################################
# 2025 April 5
#
# The author disclaims copyright to this source code. In place of
# a legal notice, here is a blessing:
#
# * May you do good and not evil.
# * May you find forgiveness for yourself and forgive others.
# * May you share freely, never taking more than you give.
#
########################################################################
#
# Helper routines for running tests on teaish extensions
#
########################################################################
# ----- @module teaish/tester.tcl -----
#
# @section TEA-ish Testing APIs.
#
# Though these are part of the autosup dir hierarchy, they are not
# intended to be run from autosetup code. Rather, they're for use
# with/via teaish.tester.tcl and target canonical Tcl only, not JimTcl
# (which the autosetup pieces do target).
#
# @test-current-scope ?lvl?
#
# Returns the name of the _calling_ proc from ($lvl + 1) levels up the
# call stack (where the caller's level will be 1 up from _this_
# call). If $lvl would resolve to global scope "global scope" is
# returned and if it would be negative then a string indicating such
# is returned (as opposed to throwing an error).
#
proc test-current-scope {{lvl 0}} {
#uplevel [expr {$lvl + 1}] {lindex [info level 0] 0}
set ilvl [info level]
set offset [expr {$ilvl - $lvl - 1}]
if { $offset < 0} {
return "invalid scope ($offset)"
} elseif { $offset == 0} {
return "global scope"
} else {
return [lindex [info level $offset] 0]
}
}
# @test-msg
#
# Emits all arugments to stdout.
#
proc test-msg {args} {
puts "$args"
}
# @test-warn
#
# Emits all arugments to stderr.
#
proc test-warn {args} {
puts stderr "WARNING: $args"
}
#
# @test-error msg
#
# Triggers a test-failed error with a string describing the calling
# scope and the provided message.
#
proc test-fail {args} {
#puts stderr "ERROR: \[[test-current-scope 1]]: $msg"
#exit 1
error "FAIL: \[[test-current-scope 1]]: $args"
}
array set ::test__Counters {}
array set ::test__Config {
verbose-assert 0 verbose-affirm 0
}
# Internal impl for affirm and assert.
#
# $args = ?-v? script {msg-on-fail ""}
proc test__affert {failMode args} {
if {$failMode} {
set what assert
} else {
set what affirm
}
set verbose $::test__Config(verbose-$what)
if {"-v" eq [lindex $args 0]} {
lassign $args - script msg
if {1 == [llength $args]} {
# If -v is the only arg, toggle default verbose mode
set ::test__Config(verbose-$what) [expr {!$::test__Config(verbose-$what)}]
return
}
incr verbose
} else {
lassign $args script msg
}
incr ::test__Counters($what)
if {![uplevel 1 [concat expr [list $script]]]} {
if {"" eq $msg} {
set msg $script
}
set txt [join [list $what # $::test__Counters($what) "failed:" $msg]]
if {$failMode} {
puts stderr $txt
exit 1
} else {
error $txt
}
} elseif {$verbose} {
puts stderr [join [list $what # $::test__Counters($what) "passed:" $script]]
}
}
#
# @affirm ?-v? script ?msg?
#
# Works like a conventional assert method does, but reports failures
# using [error] instead of [exit]. If -v is used, it reports passing
# assertions to stderr. $script is evaluated in the caller's scope as
# an argument to [expr].
#
proc affirm {args} {
tailcall test__affert 0 {*}$args
}
#
# @assert ?-v? script ?msg?
#
# Works like [affirm] but exits on error.
#
proc assert {args} {
tailcall test__affert 1 {*}$args
}
#
# @test-assert testId script ?msg?
#
# Works like [assert] but emits $testId to stdout first.
#
proc test-assert {testId script {msg ""}} {
puts "test $testId"
tailcall test__affert 1 $script $msg
}
#
# @test-expect testId script result
#
# Runs $script in the calling scope and compares its result to
# $result, minus any leading or trailing whitespace. If they differ,
# it triggers an [assert].
#
proc test-expect {testId script result} {
puts "test $testId"
set x [string trim [uplevel 1 $script]]
set result [string trim $result]
tailcall test__affert 0 [list $x eq $result] \
"\nEXPECTED: <<$result>>\nGOT: <<$x>>"
}
#
# @test-catch cmd ?...args?
#
# Runs [cmd ...args], repressing any exception except to possibly log
# the failure. Returns 1 if it caught anything, 0 if it didn't.
#
proc test-catch {cmd args} {
if {[catch {
$cmd {*}$args
} rc xopts]} {
puts "[test-current-scope] ignoring failure of: $cmd [lindex $args 0]: $rc"
return 1
}
return 0
}
if {![array exists ::teaish__BuildFlags]} {
array set ::teaish__BuildFlags {}
}
#
# @teaish-build-flag3 flag tgtVar ?dflt?
#
# If the current build has the configure-time flag named $flag set
# then tgtVar is assigned its value and 1 is returned, else tgtVal is
# assigned $dflt and 0 is returned.
#
# Caveat #1: only valid when called in the context of teaish's default
# "make test" recipe, e.g. from teaish.test.tcl. It is not valid from
# a teaish.tcl configure script because (A) the state it relies on
# doesn't fully exist at that point and (B) that level of the API has
# more direct access to the build state. This function requires that
# an external script have populated its internal state, which is
# normally handled via teaish.tester.tcl.in.
#
# Caveat #2: defines in the style of HAVE_FEATURENAME with a value of
# 0 are, by long-standing configure script conventions, treated as
# _undefined_ here.
#
proc teaish-build-flag3 {flag tgtVar {dflt ""}} {
upvar $tgtVar tgt
if {[info exists ::teaish__BuildFlags($flag)]} {
set tgt $::teaish__BuildFlags($flag)
return 1;
} elseif {0==[array size ::teaish__BuildFlags]} {
test-warn \
"\[[test-current-scope]] was called from " \
"[test-current-scope 1] without the build flags imported."
}
set tgt $dflt
return 0
}
#
# @teaish-build-flag flag ?dflt?
#
# Convenience form of teaish-build-flag3 which returns the
# configure-time-defined value of $flag or "" if it's not defined (or
# if it's an empty string).
#
proc teaish-build-flag {flag {dflt ""}} {
set tgt ""
teaish-build-flag3 $flag tgt $dflt
return $tgt
}
configure
Vendored Executable
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
dir="`dirname "$0"`/autosetup"
#@@INITCHECK@@#
WRAPPER="$0"; export WRAPPER; exec "`"$dir/autosetup-find-tclsh"`" "$dir/autosetup" "$@"
contrib/sqlitecon.tcl
+679
View File
@@ -0,0 +1,679 @@
# A Tk console widget for SQLite. Invoke sqlitecon::create with a window name,
# a prompt string, a title to set a new top-level window, and the SQLite
# database handle. For example:
#
# sqlitecon::create .sqlcon {sql:- } {SQL Console} db
#
# A toplevel window is created that allows you to type in SQL commands to
# be processed on the spot.
#
# A limited set of dot-commands are supported:
#
# .table
# .schema ?TABLE?
# .mode list|column|multicolumn|line
# .exit
#
# In addition, a new SQL function named "edit()" is created. This function
# takes a single text argument and returns a text result. Whenever the
# the function is called, it pops up a new toplevel window containing a
# text editor screen initialized to the argument. When the "OK" button
# is pressed, whatever revised text is in the text editor is returned as
# the result of the edit() function. This allows text fields of SQL tables
# to be edited quickly and easily as follows:
#
# UPDATE table1 SET dscr = edit(dscr) WHERE rowid=15;
#
# Create a namespace to work in
#
namespace eval ::sqlitecon {
# do nothing
}
# Create a console widget named $w. The prompt string is $prompt.
# The title at the top of the window is $title. The database connection
# object is $db
#
proc sqlitecon::create {w prompt title db} {
upvar #0 $w.t v
if {[winfo exists $w]} {destroy $w}
if {[info exists v]} {unset v}
toplevel $w
wm title $w $title
wm iconname $w $title
frame $w.mb -bd 2 -relief raised
pack $w.mb -side top -fill x
menubutton $w.mb.file -text File -menu $w.mb.file.m
menubutton $w.mb.edit -text Edit -menu $w.mb.edit.m
pack $w.mb.file $w.mb.edit -side left -padx 8 -pady 1
set m [menu $w.mb.file.m -tearoff 0]
$m add command -label {Close} -command "destroy $w"
sqlitecon::create_child $w $prompt $w.mb.edit.m
set v(db) $db
$db function edit ::sqlitecon::_edit
}
# This routine creates a console as a child window within a larger
# window. It also creates an edit menu named "$editmenu" if $editmenu!="".
# The calling function is responsible for posting the edit menu.
#
proc sqlitecon::create_child {w prompt editmenu} {
upvar #0 $w.t v
if {$editmenu!=""} {
set m [menu $editmenu -tearoff 0]
$m add command -label Cut -command "sqlitecon::Cut $w.t"
$m add command -label Copy -command "sqlitecon::Copy $w.t"
$m add command -label Paste -command "sqlitecon::Paste $w.t"
$m add command -label {Clear Screen} -command "sqlitecon::Clear $w.t"
$m add separator
$m add command -label {Save As...} -command "sqlitecon::SaveFile $w.t"
catch {$editmenu config -postcommand "sqlitecon::EnableEditMenu $w"}
}
scrollbar $w.sb -orient vertical -command "$w.t yview"
pack $w.sb -side right -fill y
text $w.t -font fixed -yscrollcommand "$w.sb set"
pack $w.t -side right -fill both -expand 1
bindtags $w.t Sqlitecon
set v(editmenu) $editmenu
set v(history) 0
set v(historycnt) 0
set v(current) -1
set v(prompt) $prompt
set v(prior) {}
set v(plength) [string length $v(prompt)]
set v(x) 0
set v(y) 0
set v(mode) column
set v(header) on
$w.t mark set insert end
$w.t tag config ok -foreground blue
$w.t tag config err -foreground red
$w.t insert end $v(prompt)
$w.t mark set out 1.0
after idle "focus $w.t"
}
bind Sqlitecon <1> {sqlitecon::Button1 %W %x %y}
bind Sqlitecon <B1-Motion> {sqlitecon::B1Motion %W %x %y}
bind Sqlitecon <B1-Leave> {sqlitecon::B1Leave %W %x %y}
bind Sqlitecon <B1-Enter> {sqlitecon::cancelMotor %W}
bind Sqlitecon <ButtonRelease-1> {sqlitecon::cancelMotor %W}
bind Sqlitecon <KeyPress> {sqlitecon::Insert %W %A}
bind Sqlitecon <Left> {sqlitecon::Left %W}
bind Sqlitecon <Control-b> {sqlitecon::Left %W}
bind Sqlitecon <Right> {sqlitecon::Right %W}
bind Sqlitecon <Control-f> {sqlitecon::Right %W}
bind Sqlitecon <BackSpace> {sqlitecon::Backspace %W}
bind Sqlitecon <Control-h> {sqlitecon::Backspace %W}
bind Sqlitecon <Delete> {sqlitecon::Delete %W}
bind Sqlitecon <Control-d> {sqlitecon::Delete %W}
bind Sqlitecon <Home> {sqlitecon::Home %W}
bind Sqlitecon <Control-a> {sqlitecon::Home %W}
bind Sqlitecon <End> {sqlitecon::End %W}
bind Sqlitecon <Control-e> {sqlitecon::End %W}
bind Sqlitecon <Return> {sqlitecon::Enter %W}
bind Sqlitecon <KP_Enter> {sqlitecon::Enter %W}
bind Sqlitecon <Up> {sqlitecon::Prior %W}
bind Sqlitecon <Control-p> {sqlitecon::Prior %W}
bind Sqlitecon <Down> {sqlitecon::Next %W}
bind Sqlitecon <Control-n> {sqlitecon::Next %W}
bind Sqlitecon <Control-k> {sqlitecon::EraseEOL %W}
bind Sqlitecon <<Cut>> {sqlitecon::Cut %W}
bind Sqlitecon <<Copy>> {sqlitecon::Copy %W}
bind Sqlitecon <<Paste>> {sqlitecon::Paste %W}
bind Sqlitecon <<Clear>> {sqlitecon::Clear %W}
# Insert a single character at the insertion cursor
#
proc sqlitecon::Insert {w a} {
$w insert insert $a
$w yview insert
}
# Move the cursor one character to the left
#
proc sqlitecon::Left {w} {
upvar #0 $w v
scan [$w index insert] %d.%d row col
if {$col>$v(plength)} {
$w mark set insert "insert -1c"
}
}
# Erase the character to the left of the cursor
#
proc sqlitecon::Backspace {w} {
upvar #0 $w v
scan [$w index insert] %d.%d row col
if {$col>$v(plength)} {
$w delete {insert -1c}
}
}
# Erase to the end of the line
#
proc sqlitecon::EraseEOL {w} {
upvar #0 $w v
scan [$w index insert] %d.%d row col
if {$col>=$v(plength)} {
$w delete insert {insert lineend}
}
}
# Move the cursor one character to the right
#
proc sqlitecon::Right {w} {
$w mark set insert "insert +1c"
}
# Erase the character to the right of the cursor
#
proc sqlitecon::Delete w {
$w delete insert
}
# Move the cursor to the beginning of the current line
#
proc sqlitecon::Home w {
upvar #0 $w v
scan [$w index insert] %d.%d row col
$w mark set insert $row.$v(plength)
}
# Move the cursor to the end of the current line
#
proc sqlitecon::End w {
$w mark set insert {insert lineend}
}
# Add a line to the history
#
proc sqlitecon::addHistory {w line} {
upvar #0 $w v
if {$v(historycnt)>0} {
set last [lindex $v(history) [expr $v(historycnt)-1]]
if {[string compare $last $line]} {
lappend v(history) $line
incr v(historycnt)
}
} else {
set v(history) [list $line]
set v(historycnt) 1
}
set v(current) $v(historycnt)
}
# Called when "Enter" is pressed. Do something with the line
# of text that was entered.
#
proc sqlitecon::Enter w {
upvar #0 $w v
scan [$w index insert] %d.%d row col
set start $row.$v(plength)
set line [$w get $start "$start lineend"]
$w insert end \n
$w mark set out end
if {$v(prior)==""} {
set cmd $line
} else {
set cmd $v(prior)\n$line
}
if {[string index $cmd 0]=="." || [$v(db) complete $cmd]} {
regsub -all {\n} [string trim $cmd] { } cmd2
addHistory $w $cmd2
set rc [catch {DoCommand $w $cmd} res]
if {![winfo exists $w]} return
if {$rc} {
$w insert end $res\n err
} elseif {[string length $res]>0} {
$w insert end $res\n ok
}
set v(prior) {}
$w insert end $v(prompt)
} else {
set v(prior) $cmd
regsub -all {[^ ]} $v(prompt) . x
$w insert end $x
}
$w mark set insert end
$w mark set out {insert linestart}
$w yview insert
}
# Execute a single SQL command. Pay special attention to control
# directives that begin with "."
#
# The return value is the text output from the command, properly
# formatted.
#
proc sqlitecon::DoCommand {w cmd} {
upvar #0 $w v
set mode $v(mode)
set header $v(header)
if {[regexp {^(\.[a-z]+)} $cmd all word]} {
if {$word==".mode"} {
regexp {^.[a-z]+ +([a-z]+)} $cmd all v(mode)
return {}
} elseif {$word==".exit"} {
destroy [winfo toplevel $w]
return {}
} elseif {$word==".header"} {
regexp {^.[a-z]+ +([a-z]+)} $cmd all v(header)
return {}
} elseif {$word==".tables"} {
set mode multicolumn
set cmd {SELECT name FROM sqlite_master WHERE type='table'
UNION ALL
SELECT name FROM sqlite_temp_master WHERE type='table'}
$v(db) eval {PRAGMA database_list} {
if {$name!="temp" && $name!="main"} {
append cmd "UNION ALL SELECT name FROM $name.sqlite_master\
WHERE type='table'"
}
}
append cmd { ORDER BY 1}
} elseif {$word==".fullschema"} {
set pattern %
regexp {^.[a-z]+ +([^ ]+)} $cmd all pattern
set mode list
set header 0
set cmd "SELECT sql FROM sqlite_master WHERE tbl_name LIKE '$pattern'
AND sql NOT NULL UNION ALL SELECT sql FROM sqlite_temp_master
WHERE tbl_name LIKE '$pattern' AND sql NOT NULL"
$v(db) eval {PRAGMA database_list} {
if {$name!="temp" && $name!="main"} {
append cmd " UNION ALL SELECT sql FROM $name.sqlite_master\
WHERE tbl_name LIKE '$pattern' AND sql NOT NULL"
}
}
} elseif {$word==".schema"} {
set pattern %
regexp {^.[a-z]+ +([^ ]+)} $cmd all pattern
set mode list
set header 0
set cmd "SELECT sql FROM sqlite_master WHERE name LIKE '$pattern'
AND sql NOT NULL UNION ALL SELECT sql FROM sqlite_temp_master
WHERE name LIKE '$pattern' AND sql NOT NULL"
$v(db) eval {PRAGMA database_list} {
if {$name!="temp" && $name!="main"} {
append cmd " UNION ALL SELECT sql FROM $name.sqlite_master\
WHERE name LIKE '$pattern' AND sql NOT NULL"
}
}
} else {
return \
".exit\n.mode line|list|column\n.schema ?TABLENAME?\n.tables"
}
}
set res {}
if {$mode=="list"} {
$v(db) eval $cmd x {
set sep {}
foreach col $x(*) {
append res $sep$x($col)
set sep |
}
append res \n
}
if {[info exists x(*)] && $header} {
set sep {}
set hdr {}
foreach col $x(*) {
append hdr $sep$col
set sep |
}
set res $hdr\n$res
}
} elseif {[string range $mode 0 2]=="col"} {
set y {}
$v(db) eval $cmd x {
foreach col $x(*) {
if {![info exists cw($col)] || $cw($col)<[string length $x($col)]} {
set cw($col) [string length $x($col)]
}
lappend y $x($col)
}
}
if {[info exists x(*)] && $header} {
set hdr {}
set ln {}
set dash ---------------------------------------------------------------
append dash ------------------------------------------------------------
foreach col $x(*) {
if {![info exists cw($col)] || $cw($col)<[string length $col]} {
set cw($col) [string length $col]
}
lappend hdr $col
lappend ln [string range $dash 1 $cw($col)]
}
set y [concat $hdr $ln $y]
}
if {[info exists x(*)]} {
set format {}
set arglist {}
set arglist2 {}
set i 0
foreach col $x(*) {
lappend arglist x$i
append arglist2 " \$x$i"
incr i
append format " %-$cw($col)s"
}
set format [string trimleft $format]\n
if {[llength $arglist]>0} {
foreach $arglist $y "append res \[format [list $format] $arglist2\]"
}
}
} elseif {$mode=="multicolumn"} {
set y [$v(db) eval $cmd]
set max 0
foreach e $y {
if {$max<[string length $e]} {set max [string length $e]}
}
set ncol [expr {int(80/($max+2))}]
if {$ncol<1} {set ncol 1}
set nelem [llength $y]
set nrow [expr {($nelem+$ncol-1)/$ncol}]
set format "%-${max}s"
for {set i 0} {$i<$nrow} {incr i} {
set j $i
while 1 {
append res [format $format [lindex $y $j]]
incr j $nrow
if {$j>=$nelem} break
append res { }
}
append res \n
}
} else {
$v(db) eval $cmd x {
foreach col $x(*) {append res "$col = $x($col)\n"}
append res \n
}
}
return [string trimright $res]
}
# Change the line to the previous line
#
proc sqlitecon::Prior w {
upvar #0 $w v
if {$v(current)<=0} return
incr v(current) -1
set line [lindex $v(history) $v(current)]
sqlitecon::SetLine $w $line
}
# Change the line to the next line
#
proc sqlitecon::Next w {
upvar #0 $w v
if {$v(current)>=$v(historycnt)} return
incr v(current) 1
set line [lindex $v(history) $v(current)]
sqlitecon::SetLine $w $line
}
# Change the contents of the entry line
#
proc sqlitecon::SetLine {w line} {
upvar #0 $w v
scan [$w index insert] %d.%d row col
set start $row.$v(plength)
$w delete $start end
$w insert end $line
$w mark set insert end
$w yview insert
}
# Called when the mouse button is pressed at position $x,$y on
# the console widget.
#
proc sqlitecon::Button1 {w x y} {
global tkPriv
upvar #0 $w v
set v(mouseMoved) 0
set v(pressX) $x
set p [sqlitecon::nearestBoundry $w $x $y]
scan [$w index insert] %d.%d ix iy
scan $p %d.%d px py
if {$px==$ix} {
$w mark set insert $p
}
$w mark set anchor $p
focus $w
}
# Find the boundry between characters that is nearest
# to $x,$y
#
proc sqlitecon::nearestBoundry {w x y} {
set p [$w index @$x,$y]
set bb [$w bbox $p]
if {![string compare $bb ""]} {return $p}
if {($x-[lindex $bb 0])<([lindex $bb 2]/2)} {return $p}
$w index "$p + 1 char"
}
# This routine extends the selection to the point specified by $x,$y
#
proc sqlitecon::SelectTo {w x y} {
upvar #0 $w v
set cur [sqlitecon::nearestBoundry $w $x $y]
if {[catch {$w index anchor}]} {
$w mark set anchor $cur
}
set anchor [$w index anchor]
if {[$w compare $cur != $anchor] || (abs($v(pressX) - $x) >= 3)} {
if {$v(mouseMoved)==0} {
$w tag remove sel 0.0 end
}
set v(mouseMoved) 1
}
if {[$w compare $cur < anchor]} {
set first $cur
set last anchor
} else {
set first anchor
set last $cur
}
if {$v(mouseMoved)} {
$w tag remove sel 0.0 $first
$w tag add sel $first $last
$w tag remove sel $last end
update idletasks
}
}
# Called whenever the mouse moves while button-1 is held down.
#
proc sqlitecon::B1Motion {w x y} {
upvar #0 $w v
set v(y) $y
set v(x) $x
sqlitecon::SelectTo $w $x $y
}
# Called whenever the mouse leaves the boundries of the widget
# while button 1 is held down.
#
proc sqlitecon::B1Leave {w x y} {
upvar #0 $w v
set v(y) $y
set v(x) $x
sqlitecon::motor $w
}
# This routine is called to automatically scroll the window when
# the mouse drags offscreen.
#
proc sqlitecon::motor w {
upvar #0 $w v
if {![winfo exists $w]} return
if {$v(y)>=[winfo height $w]} {
$w yview scroll 1 units
} elseif {$v(y)<0} {
$w yview scroll -1 units
} else {
return
}
sqlitecon::SelectTo $w $v(x) $v(y)
set v(timer) [after 50 sqlitecon::motor $w]
}
# This routine cancels the scrolling motor if it is active
#
proc sqlitecon::cancelMotor w {
upvar #0 $w v
catch {after cancel $v(timer)}
catch {unset v(timer)}
}
# Do a Copy operation on the stuff currently selected.
#
proc sqlitecon::Copy w {
if {![catch {set text [$w get sel.first sel.last]}]} {
clipboard clear -displayof $w
clipboard append -displayof $w $text
}
}
# Return 1 if the selection exists and is contained
# entirely on the input line. Return 2 if the selection
# exists but is not entirely on the input line. Return 0
# if the selection does not exist.
#
proc sqlitecon::canCut w {
set r [catch {
scan [$w index sel.first] %d.%d s1x s1y
scan [$w index sel.last] %d.%d s2x s2y
scan [$w index insert] %d.%d ix iy
}]
if {$r==1} {return 0}
if {$s1x==$ix && $s2x==$ix} {return 1}
return 2
}
# Do a Cut operation if possible. Cuts are only allowed
# if the current selection is entirely contained on the
# current input line.
#
proc sqlitecon::Cut w {
if {[sqlitecon::canCut $w]==1} {
sqlitecon::Copy $w
$w delete sel.first sel.last
}
}
# Do a paste operation.
#
proc sqlitecon::Paste w {
if {[sqlitecon::canCut $w]==1} {
$w delete sel.first sel.last
}
if {[catch {selection get -displayof $w -selection CLIPBOARD} topaste]
&& [catch {selection get -displayof $w -selection PRIMARY} topaste]} {
return
}
if {[info exists ::$w]} {
set prior 0
foreach line [split $topaste \n] {
if {$prior} {
sqlitecon::Enter $w
update
}
set prior 1
$w insert insert $line
}
} else {
$w insert insert $topaste
}
}
# Enable or disable entries in the Edit menu
#
proc sqlitecon::EnableEditMenu w {
upvar #0 $w.t v
set m $v(editmenu)
if {$m=="" || ![winfo exists $m]} return
switch [sqlitecon::canCut $w.t] {
0 {
$m entryconf Copy -state disabled
$m entryconf Cut -state disabled
}
1 {
$m entryconf Copy -state normal
$m entryconf Cut -state normal
}
2 {
$m entryconf Copy -state normal
$m entryconf Cut -state disabled
}
}
}
# Prompt the user for the name of a writable file. Then write the
# entire contents of the console screen to that file.
#
proc sqlitecon::SaveFile w {
set types {
{{Text Files} {.txt}}
{{All Files} *}
}
set f [tk_getSaveFile -filetypes $types -title "Write Screen To..."]
if {$f!=""} {
if {[catch {open $f w} fd]} {
tk_messageBox -type ok -icon error -message $fd
} else {
puts $fd [string trimright [$w get 1.0 end] \n]
close $fd
}
}
}
# Erase everything from the console above the insertion line.
#
proc sqlitecon::Clear w {
$w delete 1.0 {insert linestart}
}
# An in-line editor for SQL
#
proc sqlitecon::_edit {origtxt {title {}}} {
for {set i 0} {[winfo exists .ed$i]} {incr i} continue
set w .ed$i
toplevel $w
wm protocol $w WM_DELETE_WINDOW "$w.b.can invoke"
wm title $w {Inline SQL Editor}
frame $w.b
pack $w.b -side bottom -fill x
button $w.b.can -text Cancel -width 6 -command [list set ::$w 0]
button $w.b.ok -text OK -width 6 -command [list set ::$w 1]
button $w.b.cut -text Cut -width 6 -command [list ::sqlitecon::Cut $w.t]
button $w.b.copy -text Copy -width 6 -command [list ::sqlitecon::Copy $w.t]
button $w.b.paste -text Paste -width 6 -command [list ::sqlitecon::Paste $w.t]
set ::$w {}
pack $w.b.cut $w.b.copy $w.b.paste $w.b.can $w.b.ok\
-side left -padx 5 -pady 5 -expand 1
if {$title!=""} {
label $w.title -text $title
pack $w.title -side top -padx 5 -pady 5
}
text $w.t -bg white -fg black -yscrollcommand [list $w.sb set]
pack $w.t -side left -fill both -expand 1
scrollbar $w.sb -orient vertical -command [list $w.t yview]
pack $w.sb -side left -fill y
$w.t insert end $origtxt
vwait ::$w
if {[set ::$w]} {
set txt [string trimright [$w.t get 1.0 end]]
} else {
set txt $origtxt
}
destroy $w
return $txt
}
doc/F2FS.txt
+87
View File
@@ -0,0 +1,87 @@
SQLite's OS layer contains the following definitions used in F2FS related
calls:
#define F2FS_IOCTL_MAGIC 0xf5
#define F2FS_IOC_START_ATOMIC_WRITE _IO(F2FS_IOCTL_MAGIC, 1)
#define F2FS_IOC_COMMIT_ATOMIC_WRITE _IO(F2FS_IOCTL_MAGIC, 2)
#define F2FS_IOC_START_VOLATILE_WRITE _IO(F2FS_IOCTL_MAGIC, 3)
#define F2FS_IOC_ABORT_VOLATILE_WRITE _IO(F2FS_IOCTL_MAGIC, 5)
#define F2FS_IOC_GET_FEATURES _IOR(F2FS_IOCTL_MAGIC, 12, u32)
#define F2FS_FEATURE_ATOMIC_WRITE 0x0004
After opening a database file on Linux (including Android), SQLite determines
whether or not a file supports F2FS atomic commits as follows:
u32 flags = 0;
rc = ioctl(fd, F2FS_IOC_GET_FEATURES, &flags);
if( rc==0 && (flags & F2FS_FEATURE_ATOMIC_WRITE) ){
/* File supports F2FS atomic commits */
}else{
/* File does NOT support F2FS atomic commits */
}
where "fd" is the file-descriptor open on the database file.
Usually, when writing to a database file that supports atomic commits, SQLite
accumulates the entire transaction in heap memory, deferring all writes to the
db file until the transaction is committed.
When it is time to commit a transaction on a file that supports atomic
commits, SQLite does:
/* Take an F_WRLCK lock on the database file. This prevents any other
** SQLite clients from reading or writing the file until the lock
** is released. */
rc = fcntl(fd, F_SETLK, ...);
if( rc!=0 ) goto failed;
rc = ioctl(fd, F2FS_IOC_START_ATOMIC_WRITE);
if( rc!=0 ) goto fallback_to_legacy_journal_commit;
foreach (dirty page){
rc = write(fd, ...dirty page...);
if( rc!=0 ){
ioctl(fd, F2FS_IOC_ABORT_VOLATILE_WRITE);
goto fallback_to_legacy_journal_commit;
}
}
rc = ioctl(fd, F2FS_IOC_COMMIT_ATOMIC_WRITE);
if( rc!=0 ){
ioctl(fd, F2FS_IOC_ABORT_VOLATILE_WRITE);
goto fallback_to_legacy_journal_commit;
}
/* If we get there, the transaction has been successfully
** committed to persistent storage. The following call
** relinquishes the F_WRLCK lock. */
fcntl(fd, F_SETLK, ...);
Assumptions:
1. After either of the F2FS_IOC_ABORT_VOLATILE_WRITE calls return,
the database file is in the state that it was in before
F2FS_IOC_START_ATOMIC_WRITE was invoked. Even if the ioctl()
fails - we're ignoring the return code.
This is true regardless of the type of error that occurred in
ioctl() or write().
2. If the system fails before the F2FS_IOC_COMMIT_ATOMIC_WRITE is
completed, then following a reboot the database file is in the
state that it was in before F2FS_IOC_START_ATOMIC_WRITE was invoked.
Or, if the write was commited right before the system failed, in a
state indicating that all write() calls were successfully committed
to persistent storage before the failure occurred.
3. If the process crashes before the F2FS_IOC_COMMIT_ATOMIC_WRITE is
completed then the file is automatically restored to the state that
it was in before F2FS_IOC_START_ATOMIC_WRITE was called. This occurs
before the posix advisory lock is automatically dropped - there is
no chance that another client will be able to read the file in a
half-committed state before the rollback operation occurs.
doc/compile-for-unix.md
+70
View File
@@ -0,0 +1,70 @@
# Notes On Compiling SQLite On All Kinds Of Unix
Here are step-by-step instructions on how to build SQLite from
canonical source on any modern machine that isn't Windows. These
notes are tested (on 2024-10-11) on Ubuntu and on MacOS, but they
are general and should work on most any modern unix platform.
See the companion document ([](./compile-for-windows.md>)) for
guidance on building for Windows.
1. Install a C-compiler. GCC or Clang both work fine. If you are
reading this document, you've probably already done that.
2. *(Optional):* Install TCL development libraries. In this note,
we'll do a private install in the $HOME/local directory,
but you can make adjustments to install TCL wherever you like.
This document assumes you are working with TCL version 9.0.
See also the [](./tcl-extension-testing.md) document that contains
more details on compiling Tcl for use with SQLite.
<ol type="a">
<li>Get the TCL source archive, perhaps from
<https://www.tcl.tk/software/tcltk/download.html>
or <https://sqlite.org/tmp/tcl9.0.0.tar.gz>.
<li>Untar the source archive. CD into the "unix/" subfolder
of the source tree.
<li>Run: `mkdir $HOME/local`
<li>Run: `./configure --prefix=$HOME/local`
<li>Run: `make install`
</ol>
<p>
As of 2024-10-25, TCL is not longer required for many
common build targets, such as "sqlite3.c" or the "sqlite3"
command-line tool. So you can skip this step if that is all
you want to build. TCL is still required to run "make test"
and similar, or to build the TCL extension, of course.
4. Download the SQLite source tree and unpack it. CD into the
toplevel directory of the source tree.
5. Run: `./configure --enable-all --with-tclsh=$HOME/local/bin/tclsh9.0`
You do not need to use --with-tclsh if the tclsh you want to use is the
first one on your PATH or if you are building without TCL.
6. Run the "`Makefile`" makefile with an appropriate target.
Examples:
<ul>
<li> `make sqlite3.c`
<li> `make sqlite3`
<li> `make sqldiff`
<li> `make sqlite3_rsync`
</ul>
<p>None of the targets above require TCL. TCL is needed
for the following targets:
<ul>
<li> `make tclextension-install`
<li> `make devtest`
<li> `make releasetest`
<li> `make sqlite3_analyzer`
</ul>
It is not required that you run the "tclextension-install" target prior to
running tests. However, the tests will run more smoothly if you do.
The version of SQLite used for the TCL extension does *not* need to
correspond to the version of SQLite under test. So you can install the
SQLite TCL extension once, and then use it to test many different versions
of SQLite.
7. For a debugging build of the CLI, where the ".treetrace" and ".wheretrace"
commands work, add the the --with-debug argument to configure.
doc/compile-for-windows.md
+191
View File
@@ -0,0 +1,191 @@
# Notes On Compiling SQLite On Windows 11
Below are step-by-step instructions on how to build SQLite from
canonical source on a new Windows 11 PC, as of 2024-10-09.
See [](./compile-for-unix.md) for a similar guide for unix-like
systems, including MacOS.
1. Install Microsoft Visual Studio. The free "community edition"
will work fine. Do a standard install for C++ development.
SQLite only needs the
"cl" compiler and the "nmake" build tool.
<ul><li><b>Note:</b>
VS2015 or later is required for the procedures below to
all work. You *might* be able to get the build to work with
earlier versions of MSVC, but in that case the TCL installation
of step 3 will be required, since the "jimsh0.c" program of
Autosetup that is used as a substitute for "tclsh.exe" won't
compile with versions of Visual Studio prior to VS2015. In any
event, building SQLite from canonical source code on Windows
is not supported for earlier versions of Visual Studio.</ul>
2. Under the "Start" menu, find "All Apps" then go to "Visual Studio 20XX"
and find "x64 Native Tools Command Prompt for VS 20XX". Pin that
application to your task bar, as you will use it a lot. Bring up
an instance of this command prompt and do all of the subsequent steps
in that "x64 Native Tools" command prompt. (Or use "x86" if you want
a 32-bit build.) The subsequent steps will not work in a vanilla
DOS prompt. Nor will they work in PowerShell.
3. *(Optional):* Install TCL development libraries.
This note assumes that you will
install the TCL development libraries in the "`c:\Tcl`" directory.
Make adjustments
if you want TCL installed somewhere else. SQLite needs both the
"tclsh90.exe" command-line tool as part of the build process, and
the "tcl90.lib" and "tclstub.lib" libraries in order to run tests.
This document assumes you are working with TCL version 9.0.
See [](./tcl-extension-testing.md#windows) for guidance on how
to compile TCL version 8.6 for use with SQLite.
<ol type="a">
<li>Get the TCL source archive, perhaps from
<https://www.tcl.tk/software/tcltk/download.html>
or <https://sqlite.org/tmp/tcl9.0.0.tar.gz>.
<li>Untar or unzip the source archive. CD into the "win/" subfolder
of the source tree.
<li>Run: `nmake /f makefile.vc INSTALLDIR=c:\Tcl release`
<li>Run: `nmake /f makefile.vc INSTALLDIR=c:\Tcl install` <br>
Notes:
<ol type="i">
<li> The previous two `nmake` commands must be run separately.
<li> Also, the INSTALLDIR=... argument is required on both.
</ol>
<li><i>Optional:</i> CD to `c:\Tcl\bin` and make a copy of
`tclsh90.exe` over into just `tclsh.exe`.
<li><i>Optional:</i>
Add `c:\Tcl\bin` to your %PATH%. To do this, go to Settings
and search for "path". Select "edit environment variables for
your account" and modify your default PATH accordingly.
You will need to close and reopen your command prompts after
making this change.
</ol>
As of 2024-10-25, TCL is not longer required for many
common build targets, such as "sqlite3.c" or the "sqlite3.exe"
command-line tool. So you can skip this step if that is all
you want to build. TCL is still required to run "make test"
and similar, or to build the TCL extension, of course.
4. Download the SQLite source tree and unpack it. CD into the
toplevel directory of the source tree.
5. Run the "`Makefile.msc`" makefile with an appropriate target.
Examples:
<ul>
<li> `nmake /f makefile.msc`
<li> `nmake /f makefile.msc sqlite3.c`
<li> `nmake /f makefile.msc sqlite3.exe`
<li> `nmake /f makefile.msc sqldiff.exe`
<li> `nmake /f makefile.msc sqlite3_rsync.exe`
</ul>
<p>No TCL is required for the nmake targets above. But for the ones
that follow, you will need a TCL installation, as described in step 3
above. If you install TCL in some directory other than C:\\Tcl, then
you will also need to add the "TCLDIR=<i>&lt;dir&gt;</i>" option on the
nmake command line to tell nmake where your TCL is installed.
<ul>
<li> `nmake /f makefile.msc tclextension-install`
<li> `nmake /f makefile.msc devtest`
<li> `nmake /f makefile.msc releasetest`
<li> `nmake /f makefile.msc sqlite3_analyzer.exe`
</ul>
It is not required that you run the "tclextension-install" target prior to
running tests. However, the tests will run more smoothly if you do.
The version of SQLite used for the TCL extension does *not* need to
correspond to the version of SQLite under test. So you can install the
SQLite TCL extension once, and then use it to test many different versions
of SQLite.
7. For a debugging build of the CLI, where the ".treetrace" and ".wheretrace"
commands work, add the DEBUG=3 argument to nmake. Like this:
<ul>
<li> `nmake /f makefile.msc DEBUG=3 clean sqlite3.exe`
</ul>
## 32-bit Builds
Doing a 32-bit build is just like doing a 64-bit build with the
following minor changes:
1. Use the "x86 Native Tools Command Prompt" instead of
"x64 Native Tools Command Prompt". "**x86**" instead of "**x64**".
2. Use a different installation directory for TCL.
The recommended directory is `c:\tcl32`. Thus you end up
with two TCL builds:
<ul>
<li> `c:\tcl` &larr; 64-bit (the default)
<li> `c:\tcl32` &larr; 32-bit
</ul>
3. Ensure that `c:\tcl32\bin` comes before `c:\tcl\bin` on
your PATH environment variable. You can achieve this using
a command like:
<ul>
<li> `set PATH=c:\tcl32\bin;%PATH%`
</ul>
## Building a DLL
The command the developers use for building the deliverable DLL on the
[download page](https://sqlite.org/download.html) is as follows:
> ~~~~
nmake /f Makefile.msc sqlite3.dll USE_NATIVE_LIBPATHS=1 "OPTS=-DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_FTS5=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1=1 -DSQLITE_ENABLE_GEOPOLY=1 -DSQLITE_ENABLE_SESSION=1 -DSQLITE_ENABLE_PREUPDATE_HOOK=1 -DSQLITE_ENABLE_SERIALIZE=1 -DSQLITE_ENABLE_MATH_FUNCTIONS=1"
~~~~
That command generates both the sqlite3.dll and sqlite3.def files. The same
command works for both 32-bit and 64-bit builds.
## Statically Linking The TCL Library
Some utility programs associated with SQLite need to be linked
with TCL in order to function. The [sqlite3_analyzer.exe program](https://sqlite.org/sqlanalyze.html)
is an example. You can build as described above, and then
enter:
> ~~~~
nmake /f Makefile.msc sqlite3_analyzer.exe
~~~~
And you will end up with a working executable. However, that executable
will depend on having the "tcl98.dll" library somewhere on your %PATH%.
Use the following steps to build an executable that has the TCL library
statically linked so that it does not depend on separate DLL:
1. Use the appropriate "Command Prompt" window - either x86 or
x64, depending on whether you want a 32-bit or 64-bit executable.
2. Untar the TCL source tarball into a fresh directory. CD into
the "win/" subfolder.
3. Run: `nmake /f makefile.vc OPTS=static shell`
4. CD into the "Release*" subfolder that is created (note the
wildcard - the full name of the directory might vary). There
you will find the "tcl90s.lib" file. Copy this file into the
same directory that you put the "tcl90.lib" on your initial
installation. (In this document, that directory is
"`C:\Tcl32\lib`" for 32-bit builds and
"`C:\Tcl\lib`" for 64-bit builds.)
5. CD into your SQLite source code directory and build the desired
utility program, but add the following extra argument to the
nmake command line:
<blockquote><pre>
STATICALLY_LINK_TCL=1
</pre></blockquote>
<p>So, for example, to build a statically linked version of
sqlite3_analyzer.exe, you might type:
<blockquote><pre>
nmake /f Makefile.msc STATICALLY_LINK_TCL=1 sqlite3_analyzer.exe
</pre></blockquote>
6. After your executable is built, you can verify that it does not
depend on the TCL DLL by running:
<blockquote><pre>
dumpbin /dependents sqlite3_analyzer.exe
</pre></blockquote>
doc/json-enhancements.md
+144
View File
@@ -0,0 +1,144 @@
# JSON Functions Enhancements (2022)
This document summaries enhancements to the SQLite JSON support added in
early 2022.
## 1.0 Change summary:
1. New **->** and **->>** operators that work like MySQL and PostgreSQL (PG).
2. JSON functions are built-in rather than being an extension. They
are included by default, but can be omitted using the
-DSQLITE_OMIT_JSON compile-time option.
## 2.0 New operators **->** and **->>**
The SQLite language adds two new binary operators **->** and **->>**.
Both operators are similar to json_extract(). The left operand is
JSON and the right operand is a JSON path expression (possibly abbreviated
for compatibility with PG - see below). So they are similar to a
two-argument call to json_extract().
The difference between -> and ->> (and json_extract()) is as follows:
* The -> operator always returns JSON.
* The ->> operator converts the answer into a primitive SQL datatype
such as TEXT, INTEGER, REAL, or NULL. If a JSON object or array
is selected, that object or array is rendered as text. If a JSON
value is selected, that value is converted into its corresponding
SQL type
* The json_extract() interface returns JSON when a JSON object or
array is selected, or a primitive SQL datatype when a JSON value
is selected. This is different from MySQL, in which json_extract()
always returns JSON, but the difference is retained because it has
worked that way for 6 years and changing it now would likely break
a lot of legacy code.
In MySQL and PG, the ->> operator always returns TEXT (or NULL) and never
INTEGER or REAL. This is due to limitations in the type handling capabilities
of those systems. In MySQL and PG, the result type a function or operator
may only depend on the type of its arguments, never the value of its arguments.
But the underlying JSON type depends on the value of the JSON path
expression, not the type of the JSON path expression (which is always TEXT).
Hence, the result type of ->> in MySQL and PG is unable to vary according
to the type of the JSON value being extracted.
The type system in SQLite is more general. Functions in SQLite are able
to return different datatypes depending on the value of their arguments.
So the ->> operator in SQLite is able to return TEXT, INTEGER, REAL, or NULL
depending on the JSON type of the value being extracted. This means that
the behavior of the ->> is slightly different in SQLite versus MySQL and PG
in that it will sometimes return INTEGER and REAL values, depending on its
inputs. It is possible to implement the ->> operator in SQLite so that it
always operates exactly like MySQL and PG and always returns TEXT or NULL,
but I have been unable to think of any situations where returning the
actual JSON value this would cause problems, so I'm including the enhanced
functionality in SQLite.
The table below attempts to summarize the differences between the
-> and ->> operators and the json_extract() function, for SQLite, MySQL,
and PG. JSON values are shown using their SQL text representation but
in a bold font.
<table border=1 cellpadding=5 cellspacing=0>
<tr><th>JSON<th>PATH<th>-&gt; operator<br>(all)<th>-&gt;&gt; operator<br>(MySQL/PG)
<th>-&gt;&gt; operator<br>(SQLite)<th>json_extract()<br>(SQLite)
<tr><td> **'{"a":123}'** <td>'$.a'<td> **'123'** <td> '123' <td> 123 <td> 123
<tr><td> **'{"a":4.5}'** <td>'$.a'<td> **'4.5'** <td> '4.5' <td> 4.5 <td> 4.5
<tr><td> **'{"a":"xyz"}'** <td>'$.a'<td> **'"xyz"'** <td> 'xyz' <td> 'xyz' <td> 'xyz'
<tr><td> **'{"a":null}'** <td>'$.a'<td> **'null'** <td> NULL <td> NULL <td> NULL
<tr><td> **'{"a":[6,7,8]}'** <td>'$.a'<td> **'[6,7,8]'** <td> '[6,7,8]' <td> '[6,7,8]' <td> **'[6,7,8]'**
<tr><td> **'{"a":{"x":9}}'** <td>'$.a'<td> **'{"x":9}'** <td> '{"x":9}' <td> '{"x":9}' <td> **'{"x":9}'**
<tr><td> **'{"b":999}'** <td>'$.a'<td> NULL <td> NULL <td> NULL <td> NULL
</table>
Important points about the table above:
* The -> operator always returns either JSON or NULL.
* The ->> operator never returns JSON. It always returns TEXT or NULL, or in the
case of SQLite, INTEGER or REAL.
* The MySQL json_extract() function works exactly the same
as the MySQL -> operator.
* The SQLite json_extract() operator works like -> for JSON objects and
arrays, and like ->> for JSON values.
* The -> operator works the same for all systems.
* The only difference in ->> between SQLite and other systems is that
when the JSON value is numeric, SQLite returns a numeric SQL value,
whereas the other systems return a text representation of the numeric
value.
### 2.1 Abbreviated JSON path expressions for PG compatibility
The table above always shows the full JSON path expression: '$.a'. But
PG does not accept this syntax. PG only allows a single JSON object label
name or a single integer array index. In order to provide compatibility
with PG, The -> and ->> operators in SQLite are extended to also support
a JSON object label or an integer array index for the right-hand side
operand, in addition to a full JSON path expression.
Thus, a -> or ->> operator that works on MySQL will work in
SQLite. And a -> or ->> operator that works in PG will work in SQLite.
But because SQLite supports the union of the disjoint capabilities of
MySQL and PG, there will always be -> and ->> operators that work in
SQLite that do not work in one of MySQL and PG. This is an unavoidable
consequence of the different syntax for -> and ->> in MySQL and PG.
In the following table, assume that "value1" is a JSON object and
"value2" is a JSON array.
<table border=1 cellpadding=5 cellspacing=0>
<tr><th>SQL expression <th>Works in MySQL?<th>Works in PG?<th>Works in SQLite
<tr><td>value1-&gt;'$.a' <td> yes <td> no <td> yes
<tr><td>value1-&gt;'a' <td> no <td> yes <td> yes
<tr><td>value2-&gt;'$[2]' <td> yes <td> no <td> yes
<tr><td>value2-&gt;2 <td> no <td> yes <td> yes
</table>
The abbreviated JSON path expressions only work for the -> and ->> operators
in SQLite. The json_extract() function, and all other built-in SQLite
JSON functions, continue to require complete JSON path expressions for their
PATH arguments.
## 3.0 JSON moved into the core
The JSON interface is now moved into the SQLite core.
When originally written in 2015, the JSON functions were an extension
that could be optionally included at compile-time, or loaded at run-time.
The implementation was in a source file named ext/misc/json1.c in the
source tree. JSON functions were only compiled in if the
-DSQLITE_ENABLE_JSON1 compile-time option was used.
After these enhancements, the JSON functions are now built-ins.
The source file that implements the JSON functions is moved to src/json.c.
No special compile-time options are needed to load JSON into the build.
Instead, there is a new -DSQLITE_OMIT_JSON compile-time option to leave
them out.
doc/jsonb.md
+290
View File
@@ -0,0 +1,290 @@
# The JSONB Format
This document describes SQLite's JSONB binary encoding of
JSON.
## 1.0 What Is JSONB?
Beginning with version 3.45.0 (circa 2024-01-01), SQLite supports an
alternative binary encoding of JSON which we call "JSONB". JSONB is
a binary format that stored as a BLOB.
The advantage of JSONB over ordinary text RFC 8259 JSON is that JSONB
is both slightly smaller (by between 5% and 10% in most cases) and
can be processed in less than half the number of CPU cycles. The built-in
[JSON SQL functions] of SQLite can accept either ordinary text JSON
or the binary JSONB encoding for any of their JSON inputs.
The "JSONB" name is inspired by [PostgreSQL](https://postgresql.org), but the
on-disk format for SQLite's JSONB is not the same as PostgreSQL's.
The two formats have the same name, but they have wildly different internal
representations and are not in any way binary compatible.
The central idea behind this JSONB specification is that each element
begins with a header that includes the size and type of that element.
The header takes the place of punctuation such as double-quotes,
curly-brackes, square-brackets, commas, and colons. Since the size
and type of each element is contained in its header, the element can
be read faster since it is no longer necessary to carefully scan forward
looking for the closing delimiter. The payload of JSONB is the same
as for corresponding text JSON. The same payload bytes occur in the
same order. The only real difference between JSONB and ordinary text
JSON is that JSONB includes a binary header on
each element and omits delimiter and separator punctuation.
### 1.1 Internal Use Only
The details of the JSONB are not intended to be visible to application
developers. Application developers should look at JSONB as an opaque BLOB
used internally by SQLite. Nevertheless, we want the format to be backwards
compatible across all future versions of SQLite. To that end, the format
is documented by this file in the source tree. But this file should be
used only by SQLite core developers, not by developers of applications
that only use SQLite.
## 2.0 The Purpose Of This Document
JSONB is not intended as an external format to be used by
applications. JSONB is designed for internal use by SQLite only.
Programmers do not need to understand the JSONB format in order to
use it effectively.
Applications should access JSONB only through the [JSON SQL functions],
not by looking at individual bytes of the BLOB.
However, JSONB is intended to be portable and backwards compatible
for all future versions of SQLite. In other words, you should not have
to export and reimport your SQLite database files when you upgrade to
a newer SQLite version. For that reason, the JSONB format needs to
be well-defined.
This document is therefore similar in purpose to the
[SQLite database file format] document that describes the on-disk
format of an SQLite database file. Applications are not expected
to directly read and write the bits and bytes of SQLite database files.
The SQLite database file format is carefully documented so that it
can be stable and enduring. In the same way, the JSONB representation
of JSON is documented here so that it too can be stable and enduring,
not so that applications can read or writes individual bytes.
## 3.0 Encoding
JSONB is a direct translation of the underlying text JSON. The difference
is that JSONB uses a binary encoding that is faster to parse compared to
the detailed syntax of text JSON.
Each JSON element is encoded as a header and a payload. The header
determines type of element (string, numeric, boolean, null, object, or
array) and the size of the payload. The header can be between 1 and
9 bytes in size. The payload can be any size from zero bytes up to the
maximum allowed BLOB size.
### 3.1 Payload Size
The upper four bits of the first byte of the header determine size of the
header and possibly also the size of the payload.
If the upper four bits have a value between 0 and 11, then the header is
exactly one byte in size and the payload size is determined by those
upper four bits. If the upper four bits have a value between 12 and 15,
that means that the total header size is 2, 3, 5, or 9 bytes and the
payload size is unsigned big-endian integer that is contained in the
subsequent bytes. The size integer is the one byte that following the
initial header byte if the upper four bits
are 12, two bytes if the upper bits are 13, four bytes if the upper bits
are 14, and eight bytes if the upper bits are 15. The current design
of SQLite does not support BLOB values larger than 2GiB, so the eight-byte
variant of the payload size integer will never be used by the current code.
The eight-byte payload size integer is included in the specification
to allow for future expansion.
The header for an element does *not* need to be in its simplest
form. For example, consider the JSON numeric value "`1`".
That element can be encode in five different ways:
* `0x13 0x31`
* `0xc3 0x01 0x31`
* `0xd3 0x00 0x01 0x31`
* `0xe3 0x00 0x00 0x00 0x01 0x31`
* `0xf3 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x31`
The shortest encoding is preferred, of course, and usually happens with
primitive elements such as numbers. However the total size of an array
or object might not be known exactly when the header of the element is
first generated. It is convenient to reserve space for the largest
possible header and then go back and fill in the correct payload size
at the end. This technique can result in array or object headers that
are larger than absolutely necessary.
### 3.2 Element Type
The least-significant four bits of the first byte of the header (the first
byte masked against 0x0f) determine element type. The following codes are
used:
<ol>
<li type="0"><p><b>NULL</b> &rarr;
The element is a JSON "null". The payload size for a true JSON NULL must
must be zero. Future versions of SQLite might extend the JSONB format
with elements that have a zero element type but a non-zero size. In that
way, legacy versions of SQLite will interpret the element as a NULL
for backwards compatibility while newer versions will interpret the
element in some other way.
<li value="1"><p><b>TRUE</b> &rarr;
The element is a JSON "true". The payload size must be zero for a actual
"true" value. Elements with type 1 and a non-zero payload size are
reserved for future expansion. Legacy implementations that see an element
type of 1 with a non-zero payload size should continue to interpret that
element as "true" for compatibility.
<li value="2"><p><b>FALSE</b> &rarr;
The element is a JSON "false". The payload size must be zero for a actual
"false" value. Elements with type 2 and a non-zero payload size are
reserved for future expansion. Legacy implementations that see an element
type of 2 with a non-zero payload size should continue to interpret that
element as "false" for compatibility.
<li value="3"><p><b>INT</b> &rarr;
The element is a JSON integer value in the canonical
RFC 8259 format, without extensions. The payload is the ASCII
text representation of that numeric value.
<li value="4"><p><b>INT5</b> &rarr;
The element is a JSON integer value that is not in the
canonical format. The payload is the ASCII
text representation of that numeric value. Because the payload is in a
non-standard format, it will need to be translated when the JSONB is
converted into RFC 8259 text JSON.
<li value="5"><p><b>FLOAT</b> &rarr;
The element is a JSON floating-point value in the canonical
RFC 8259 format, without extensions. The payload is the ASCII
text representation of that numeric value.
<li value="6"><p><b>FLOAT5</b> &rarr;
The element is a JSON floating-point value that is not in the
canonical format. The payload is the ASCII
text representation of that numeric value. Because the payload is in a
non-standard format, it will need to be translated when the JSONB is
converted into RFC 8259 text JSON.
<li value="7"><p><b>TEXT</b> &rarr;
The element is a JSON string value that does not contain
any escapes nor any characters that need to be escaped for either SQL or
JSON. The payload is the UTF8 text representation of the string value.
The payload does <i>not</i> include string delimiters.
<li value="8"><p><b>TEXTJ</b> &rarr;
The element is a JSON string value that contains
RFC 8259 character escapes (such as "<tt>\n</tt>" or "<tt>\u0020</tt>").
Those escapes will need to be translated into actual UTF8 if this element
is [json_extract|extracted] into SQL.
The payload is the UTF8 text representation of the escaped string value.
The payload does <i>not</i> include string delimiters.
<li value="9"><p><b>TEXT5</b> &rarr;
The element is a JSON string value that contains
character escapes, including some character escapes that part of JSON5
and which are not found in the canonical RFC 8259 spec.
Those escapes will need to be translated into standard JSON prior to
rendering the JSON as text, or into their actual UTF8 characters if this
element is [json_extract|extracted] into SQL.
The payload is the UTF8 text representation of the escaped string value.
The payload does <i>not</i> include string delimiters.
<li value="10"><p><b>TEXTRAW</b> &rarr;
The element is a JSON string value that contains
UTF8 characters that need to be escaped if this string is rendered into
standard JSON text.
The payload does <i>not</i> include string delimiters.
<li value="11"><p><b>ARRAY</b> &rarr;
The element is a JSON array. The payload contains
JSONB elements that comprise values contained within the array.
<li value="12"><p><b>OBJECT</b> &rarr;
The element is a JSON object. The payload contains
pairs of JSONB elements that comprise entries for the JSON object.
The first element in each pair must be a string (types 7 through 10).
The second element of each pair may be any types, including nested
arrays or objects.
<li value="13"><p><b>RESERVED-13</b> &rarr;
Reserved for future expansion. Legacy implements that encounter this
element type should raise an error.
<li value="14"><p><b>RESERVED-14</b> &rarr;
Reserved for future expansion. Legacy implements that encounter this
element type should raise an error.
<li value="15"><p><b>RESERVED-15</b> &rarr;
Reserved for future expansion. Legacy implements that encounter this
element type should raise an error.
</ol>
Element types outside the range of 0 to 12 are reserved for future
expansion. The current implement raises an error if see an element type
other than those listed above. However, future versions of SQLite might
use of the three remaining element types to implement indexing or similar
optimizations, to speed up lookup against large JSON arrays and/or objects.
### 3.3 Design Rationale For Element Types
A key goal of JSONB is that it should be quick to translate
to and from text JSON and/or be constructed from SQL values.
When converting from text into JSONB, we do not want the
converter subroutine to burn CPU cycles converting elements
values into some standard format which might never be used.
Format conversion is "lazy" - it is deferred until actually
needed. This has implications for the JSONB format design:
1. Numeric values are stored as text, not a numbers. The values are
a direct copy of the text JSON values from which they are derived.
2. There are multiple element types depending on the details of value
formats. For example, INT is used for pure RFC-8259 integer
literals and INT5 exists for JSON5 extensions such as hexadecimal
notation. FLOAT is used for pure RFC-8259 floating point literals
and FLOAT5 is used for JSON5 extensions. There are four different
representations of strings, depending on where the string came from
and how special characters within the string are escaped.
A second goal of JSONB is that it should be capable of serving as the
"parse tree" for JSON when a JSON value is being processed by the
various [JSON SQL functions] built into SQLite. Before JSONB was
developed, operations such [json_replace()] and [json_patch()]
and similar worked in three stages:
1. Translate the text JSON into a internal format that is
easier to scan and edit.
2. Perform the requested operation on the JSON.
3. Translate the internal format back into text.
JSONB seeks to serve as the internal format directly - bypassing
the first and third stages of that process. Since most of the CPU
cycles are spent on the first and third stages, that suggests that
JSONB processing will be much faster than text JSON processing.
So when processing JSONB, only the second stage of the three-stage
process is required. But when processing text JSON, it is still necessary
to do stages one and three. If JSONB is to be used as the internal
binary representation, this is yet another reason to store numeric
values as text. Storing numbers as text minimizes the amount of
conversion work needed for stages one and three. This is also why
there are four different representations of text in JSONB. Different
text representations are used for text coming from different sources
(RFC-8259 JSON, JSON5, or SQL string values) and conversions only
happen if and when they are actually needed.
### 3.4 Valid JSONB BLOBs
A valid JSONB BLOB consists of a single JSON element. The element must
exactly fill the BLOB. This one element is often a JSON object or array
and those usually contain additional elements as its payload, but the
element can be a primitive value such a string, number, boolean, or null.
When the built-in JSON functions are attempting to determine if a BLOB
argument is a JSONB or just a random BLOB, they look at the header of
the outer element to see that it is well-formed and that the element
completely fills the BLOB. If these conditions are met, then the BLOB
is accepted as a JSONB value.
doc/lemon.html
+1282
View File
File diff suppressed because it is too large. Load diff
doc/pager-invariants.txt
+76
View File
@@ -0,0 +1,76 @@
*** Throughout this document, a page is deemed to have been synced
automatically as soon as it is written when PRAGMA synchronous=OFF.
Otherwise, the page is not synced until the xSync method of the VFS
is called successfully on the file containing the page.
*** Definition: A page of the database file is said to be "overwriteable" if
one or more of the following are true about the page:
(a) The original content of the page as it was at the beginning of
the transaction has been written into the rollback journal and
synced.
(b) The page was a freelist leaf page at the start of the transaction.
(c) The page number is greater than the largest page that existed in
the database file at the start of the transaction.
(1) A page of the database file is never overwritten unless one of the
following are true:
(a) The page and all other pages on the same sector are overwriteable.
(b) The atomic page write optimization is enabled, and the entire
transaction other than the update of the transaction sequence
number consists of a single page change.
(2) The content of a page written into the rollback journal exactly matches
both the content in the database when the rollback journal was written
and the content in the database at the beginning of the current
transaction.
(3) Writes to the database file are an integer multiple of the page size
in length and are aligned to a page boundary.
(4) Reads from the database file are either aligned on a page boundary and
an integer multiple of the page size in length or are taken from the
first 100 bytes of the database file.
(5) All writes to the database file are synced prior to the rollback journal
being deleted, truncated, or zeroed.
(6) If a master journal file is used, then all writes to the database file
are synced prior to the master journal being deleted.
*** Definition: Two databases (or the same database at two points it time)
are said to be "logically equivalent" if they give the same answer to
all queries. Note in particular the content of freelist leaf
pages can be changed arbitrarily without effecting the logical equivalence
of the database.
(7) At any time, if any subset, including the empty set and the total set,
of the unsynced changes to a rollback journal are removed and the
journal is rolled back, the resulting database file will be logical
equivalent to the database file at the beginning of the transaction.
(8) When a transaction is rolled back, the xTruncate method of the VFS
is called to restore the database file to the same size it was at
the beginning of the transaction. (In some VFSes, the xTruncate
method is a no-op, but that does not change the fact the SQLite will
invoke it.)
(9) Whenever the database file is modified, at least one bit in the range
of bytes from 24 through 39 inclusive will be changed prior to releasing
the EXCLUSIVE lock.
(10) The pattern of bits in bytes 24 through 39 shall not repeat in less
than one billion transactions.
(11) A database file is well-formed at the beginning and at the conclusion
of every transaction.
(12) An EXCLUSIVE lock must be held on the database file before making
any changes to the database file.
(13) A SHARED lock must be held on the database file before reading any
content out of the database file.
doc/tcl-extension-testing.md
+264
View File
@@ -0,0 +1,264 @@
# Test Procedures For The SQLite TCL Extension
## 1.0 Background
The SQLite TCL extension logic (in the
"[tclsqlite.c](/file/src/tclsqlite.c)" source
file) is statically linked into "textfixture" executable
which is the program used to do most of the testing
associated with "make test", "make devtest", and/or
"make releasetest". So the functionality of the SQLite
TCL extension is thoroughly vetted during normal testing. The
procedures below are designed to test the loadable extension
aspect of the SQLite TCL extension, and in particular to verify
that the "make tclextension-install" build target works and that
an ordinary tclsh can subsequently run "package require sqlite3".
This procedure can also be used as a template for how to set up
a local TCL+SQLite development environment. In other words, it
can be be used as a guide on how to compile per-user copies of
Tcl that are used to develop, test, and debug SQLite. In that
case, perhaps make minor changes to the procedure such as:
* Make TCLBUILD directory is permanent.
* Enable debugging symbols on the Tcl library build.
* Reduce the optimization level to -O0 for easier debugging.
* Also compile "wish" to go with each "tclsh".
<a id="unix"></a>
## 2.0 Testing On Unix-like Systems (Including Mac)
See also the [](./compile-for-unix.md) document which provides another
perspective on how to compile SQLite on unix-like systems.
### 2.1 Setup
<ol type="1">
<li value="1">
[Fossil][] installed.
<li> Check out source code and set environment variables:
<ol type="a">
<li> **TCLSOURCE** &rarr;
The top-level directory of a [Fossil][] check-out of the
[TCL source tree][tcl-fossil].
<li> **SQLITESOURCE** &rarr;
A Fossil check-out of the SQLite source tree.
<li> **TCLHOME** &rarr;
A directory that does not exist at the start of the test and which
will be deleted at the end of the test, and that will contain the
test builds of the TCL libraries and the SQLite TCL Extensions.
It is the top-most installation directory, i.e. the one provided
to Tcl's `./configure --prefix=/path/to/tcl`.
<li> **TCLVERSION** &rarr;
The `X.Y`-form version of Tcl being used: 8.6, 9.0, 9.1...
</ol>
</ol>
### 2.2 Testing TCL 8.x and 9.x on unix
From a checked-out copy of [the core Tcl tree][tcl-fossil]
<ol type="1">
<li value="3">`TCLVERSION=8.6` <br>
&uarr; A version of your choice. This process has been tested with
values of 8.6, 9.0, and 9.1 (as of 2025-04-16). The out-of-life
version 8.5 fails some of `make devtest` for undetermined reasons.
<li>`TCLHOME=$HOME/tcl/$TCLVERSION`
<li>`TCLSOURCE=/path/to/tcl/checkout`
<li>`SQLITESOURCE=/path/to/sqlite/checkout`
<li>`rm -fr $TCLHOME` <br>
&uarr; Ensure that no stale Tcl installation is laying around.
<li> `cd $TCLSOURCE`
<li> `fossil up core-8-6-branch` <br>
&uarr; The branch corresponding to `$TCLVERSION`, e.g.
`core-9-0-branch` or `trunk`.
<li> `fossil clean -x`
<li> `cd unix`
<li> `./configure --prefix=$TCLHOME --disable-shared` <br>
&uarr; The `--disable-shared` is to avoid the need to set `LD_LIBRARY_PATH`
when using this Tcl build.
<li> `make install`
<li> `cd $SQLITESOURCE`
<li> `fossil clean -x`
<li> `./configure --with-tcl=$TCLHOME --all`
<li> `make tclextension-install` <br>
&uarr; Verify extension installed at
`$TCLHOME/lib/tcl${TCLVERSION}/sqlite<SQLITE_VERSION>`.
<li> `make tclextension-list` <br>
&uarr; Verify TCL extension correctly installed.
<li> `make tclextension-verify` <br>
&uarr; Verify that the correct version is installed.
<li> `$TCLHOME/bin/tclsh[89].[0-9] test/testrunner.tcl release --explain` <br>
&uarr; Verify thousands of lines of output with no errors. Or
consider running "devtest" without --explain instead of "release".
</ol>
### 2.3 Cleanup
<ol type="1">
<li value="29"> `rm -rf $TCLHOME`
</ol>
<a id="windows"></a>
## 3.0 Testing On Windows
See also the [](./compile-for-windows.md) document which provides another
perspective on how to compile SQLite on Windows.
### 3.1 Setup for Windows
(These docs are not as up-to-date as the Unix docs, above.)
<ol type="1">
<li value="1">
[Fossil][] installed.
<li>
Unix-like command-line tools installed. Example:
[unxutils](https://unxutils.sourceforge.net/)
<li> [Visual Studio](https://visualstudio.microsoft.com/vs/community/)
installed. VS2015 or later required.
<li> Check out source code and set environment variables.
<ol type="a">
<li> **TCLSOURCE** &rarr;
The top-level directory of a Fossil check-out of the TCL source tree.
<li> **SQLITESOURCE** &rarr;
A Fossil check-out of the SQLite source tree.
<li> **TCLBUILD** &rarr;
A directory that does not exist at the start of the test and which
will be deleted at the end of the test, and that will contain the
test builds of the TCL libraries and the SQLite TCL Extensions.
<li> **ORIGINALPATH** &rarr;
The original value of %PATH%. In other words, set as follows:
`set ORIGINALPATH %PATH%`
</ol>
</ol>
### 3.2 Testing TCL 8.6 on Windows
<ol type="1">
<li value="5"> `mkdir %TCLBUILD%\tcl86`
<li> `cd %TCLSOURCE%\win`
<li> `fossil up core-8-6-16` <br>
&uarr; Or some other version of Tcl8.6.
<li> `fossil clean -x`
<li> `set INSTALLDIR=%TCLBUILD%\tcl86`
<li> `nmake /f makefile.vc release` <br>
&udarr; You *must* invoke the "release" and "install" targets
using separate "nmake" commands or tclsh86t.exe won't be
installed.
<li> `nmake /f makefile.vc install`
<li> `cd %SQLITESOURCE%`
<li> `fossil clean -x`
<li> `set TCLDIR=%TCLBUILD%\tcl86`
<li> `set PATH=%TCLBUILD%\tcl86\bin;%ORIGINALPATH%`
<li> `set TCLSH_CMD=%TCLBUILD%\tcl86\bin\tclsh86t.exe`
<li> `nmake /f Makefile.msc tclextension-install` <br>
&uarr; Verify extension installed at %TCLBUILD%\\tcl86\\lib\\tcl8.6\\sqlite3.*
<li> `nmake /f Makefile.msc tclextension-verify`
<li>`tclsh86t test/testrunner.tcl release --explain` <br>
&uarr; Verify thousands of lines of output with no errors. Or
consider running "devtest" without --explain instead of "release".
</ol>
### 3.3 Testing TCL 9.0 on Windows
<ol>
<li value="20"> `mkdir %TCLBUILD%\tcl90`
<li> `cd %TCLSOURCE%\win`
<li> `fossil up core-9-0-0` <br>
&uarr; Or some other version of Tcl9
<li> `fossil clean -x`
<li> `set INSTALLDIR=%TCLBUILD%\tcl90`
<li> `nmake /f makefile.vc release` <br>
&udarr; You *must* invoke the "release" and "install" targets
using separate "nmake" commands or tclsh90.exe won't be
installed.
<li> `nmake /f makefile.vc install`
<li> `cd %SQLITESOURCE%`
<li> `fossil clean -x`
<li> `set TCLDIR=%TCLBUILD%\tcl90`
<li> `set PATH=%TCLBUILD%\tcl90\bin;%ORIGINALPATH%`
<li> `set TCLSH_CMD=%TCLBUILD%\tcl90\bin\tclsh90.exe`
<li> `nmake /f Makefile.msc tclextension-install` <br>
&uarr; Verify extension installed at %TCLBUILD%\\tcl90\\lib\\sqlite3.*
<li> `nmake /f Makefile.msc tclextension-verify`
<li> `tclsh90 test/testrunner.tcl release --explain` <br>
&uarr; Verify thousands of lines of output with no errors. Or
consider running "devtest" without --explain instead of "release".
</ol>
### 3.4 Cleanup
<ol type="1">
<li value="35"> `rm -rf %TCLBUILD%`
</ol>
## 4.0 Testing the TEA(ish) Build (unix only)
This part requires following the setup instructions for Unix systems,
at the top of this document.
The former TEA, now TEA(ish), build of this extension uses the same
code as the builds described above but is provided in a form more
convenient for downstream Tcl users.
It lives in `autoconf/tea` and, as part of the `autoconf` bundle,
_cannot be tested directly from the canonical tree_. Instead it has to
be packaged.
### 4.1 Teaish Setup
Follow the same Tcl- and environment-related related setup described
in the first section of this document, up to and including the
installation of Tcl (unless, of course, it was already installed using
those same instructions).
### 4.2 Teaish Testing
<ol>
<li>`cd $SQLITESOURCE`
<li>Run either `make snapshot-tarball` or `make amalgamation-tarball`
&uarr;
Those steps will leave behind a temp dir called `mkpkg_tmp_dir`,
under which the extension is most readily reached. It can optionally
be extracted from the generated tarball, but that tarball was
generated from this dir, and reusing this dir is a time saver
during development.
<li> `cd mkpkg_tmp/tea`
<li> `./configure --with-tcl=$TCLHOME`
<li> `make test install` <br>
&uarr; Should run to completion without any errors.
<li> `make uninstall` <br>
&uarr; Will uninstall the extension. This _can_ be run
in the same invocation as the `install` target, but only
if the `-j#` make flag is _not_ used. If it is, the
install/uninstall steps will race and make a mess of things.
Parallel builds do not help in this build, anyway, as there's
only a single C file to compile.
</ol>
When actively developing and testing the teaish build, which requires
going through the tarball generation, there's a caveat about the
`mkpkg_tmp_dir` dir: it will be deleted every time a tarball is
built, the shell console which is parked in that
directory for testing needs to add `cd $PWD &&` to the start of the
build commands, like:
>
```
[user@host:.../mkpkg_tmp_dir/tea]$ \
cd $PWD && ./configure CFLAGS=-O0 --with-tcl=$TCLHOME \
&& make test install uninstall
```
### 4.3 Teaish Cleanup
<ol type="1">
<li> `rm -rf $TCLHOME`
<li> `cd $SQLITESOURCE; rm -fr mkpkg_tmp_dir; fossil clean -x`
</ol>
[Fossil]: https://fossil-scm.org/home
[tcl-fossil]: https://core.tcl-lang.org/tcl
doc/testrunner.md
+404
View File
@@ -0,0 +1,404 @@
# The testrunner.tcl Script
<ul type=none>
<li> 1. <a href=#overview>Overview</a>
<li> 2. <a href=#binary_tests>Binary Tests</a>
<ul type=none>
<li> 2.1. <a href=#organization_tests>Organization of Tcl Tests</a>
<li> 2.2. <a href=#run_tests>Commands to Run Tests</a>
<li> 2.3. <a href=#binary_test_failures>Investigating Binary Test Failures</a>
</ul>
<li> 3. <a href=#source_code_tests>Source Tests</a>
<ul type=none>
<li> 3.1. <a href=#commands_to_run_tests>Commands to Run SQLite Tests</a>
<li> 3.2. <a href=#zipvfs_tests>Running ZipVFS Tests</a>
<li> 3.3. <a href=#source_code_test_failures>Investigating Source Code Test Failures</a>
<li> 3.4. <a href=#fuzzdb>External Fuzzcheck Databases</a>
</ul>
<li> 4. <a href=#testrunner_options>Extra testrunner.tcl Options</a>
<li> 5. <a href=#cpu_cores>Controlling CPU Core Utilization</a>
</ul>
<a name=overview></a>
# 1. Overview
The testrunner.tcl program is a Tcl script used to run multiple SQLite
tests in parallel, thus reducing testing time on multi-core machines.
It supports the following types of tests:
* Tcl test scripts.
* Fuzzcheck tests, including using an external fuzzcheck database.
* Tests run with `make` commands. Examples:
- `make devtest`
- `make releasetest`
- `make sdevtest`
- `make testrunner`
The testrunner.tcl program stores output of all tests and builds run in
log file **testrunner.log**, created in the current working directory.
Search this file to find details of errors. Suggested search commands:
* `grep "^!" testrunner.log`
* `grep failed testrunner.log`
The testrunner.tcl program also populates SQLite database **testrunner.db**.
This database contains details of all tests run, running and to be run.
A useful query might be:
```
SELECT * FROM script WHERE state='failed'
```
You can get a summary of errors in a prior run by invoking commands like
these:
```
tclsh $(TESTDIR)/testrunner.tcl errors
tclsh $(TESTDIR)/testrunner.tcl errors -v
```
Running the command:
```
tclsh $(TESTDIR)/testrunner.tcl status
```
in the directory containing the testrunner.db database runs various queries
to produce a succinct report on the state of a running testrunner.tcl script.
A good way to keep and eye on test progress is to run either of the two
following commands:
```
watch tclsh $(TESTDIR)/testrunner.tcl status
tclsh $(TESTDIR)/testrunner.tcl status -d 2
```
Both of the commands above accomplish about the same thing, but the second
one has the advantage of not requiring "watch" to be installed on your
system.
Sometimes testrunner.tcl uses the `testfixture` binary that it is run with
to run tests (see "Binary Tests" below). Sometimes it builds testfixture and
other binaries in specific configurations to test (see "Source Tests").
<a name=binary_tests></a>
# 2. Binary Tests
The commands described in this section all run various combinations of the Tcl
test scripts using the `testfixture` binary used to run the testrunner.tcl
script (i.e. they do not invoke the compiler to build new binaries, or the
`make` command to run tests that are not Tcl scripts). The procedure to run
these tests is therefore:
1. Build the "testfixture" (or "testfixture.exe" for windows) binary using
whatever method seems convenient.
2. Test the binary built in step 1 by running testrunner.tcl with it,
perhaps with various options.
The following sub-sections describe the various options that can be
passed to testrunner.tcl to test binary testfixture builds.
<a name=organization_tests></a>
## 2.1. Organization of Tcl Tests
Tcl tests are stored in files that match the pattern *\*.test*. They are
found in both the $TOP/test/ directory, and in the various sub-directories
of the $TOP/ext/ directory of the source tree. Not all *\*.test* files
contain Tcl tests - a handful are Tcl scripts designed to invoke other
*\*.test* files.
The **veryquick** set of tests is a subset of all Tcl test scripts in the
source tree. In includes most tests, but excludes some that are very slow.
Almost all fault-injection tests (those that test the response of the library
to OOM or IO errors) are excluded. It is defined in source file
*test/permutations.test*.
The **full** set of tests includes all Tcl test scripts in the source tree.
To run a "full" test is to run all Tcl test scripts that can be found in the
source tree.
File *permutations.test* defines various test "permutations". A permutation
consists of:
* A subset of Tcl test scripts, and
* Runtime configuration to apply before running each test script
(e.g. enabling auto-vacuum, or disable lookaside).
Running **all** tests is to run all tests in the full test set, plus a dozen
or so permutations. The specific permutations that are run as part of "all"
are defined in file *testrunner_data.tcl*.
<a name=run_tests></a>
## 2.2. Commands to Run Tests
To run the "veryquick" test set, use either of the following:
```
./testfixture $TESTDIR/testrunner.tcl
./testfixture $TESTDIR/testrunner.tcl veryquick
```
To run the "full" test suite:
```
./testfixture $TESTDIR/testrunner.tcl full
```
To run the subset of the "full" test suite for which the test file name matches
a specified pattern (e.g. all tests that start with "fts5"), either of:
```
./testfixture $TESTDIR/testrunner.tcl fts5%
./testfixture $TESTDIR/testrunner.tcl 'fts5*'
```
Strictly speaking, for a test to be run the pattern must match the script
filename, not including the directory, using the rules of Tcl's
\[string match\] command. Except that before the matching is done, any "%"
characters specified as part of the pattern are transformed to "\*".
To run "all" tests (full + permutations):
```
./testfixture $TESTDIR/testrunner.tcl all
```
<a name=binary_test_failures></a>
## 2.3. Investigating Binary Test Failures
If a test fails, testrunner.tcl reports name of the Tcl test script and, if
applicable, the name of the permutation, to stdout. This information can also
be retrieved from either *testrunner.log* or *testrunner.db*.
If there is no permutation, the individual test script may be run with:
```
./testfixture $PATH_TO_SCRIPT
```
Or, if the failure occured as part of a permutation:
```
./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
```
TODO: An example instead of "$PERMUTATION" and $PATH\_TO\_SCRIPT?
<a name=source_code_tests></a>
# 3. Source Code Tests
The commands described in this section invoke the C compiler to build
binaries from the source tree, then use those binaries to run Tcl and
other tests. The advantages of this are that:
* it is possible to test multiple build configurations with a single
command, and
* it ensures that tests are always run using binaries created with the
same set of compiler options.
The testrunner.tcl commands described in this section may be run using
either a *testfixture* (or testfixture.exe) build, or with any other Tcl
shell that supports SQLite 3.31.1 or newer via "package require sqlite3".
TODO: ./configure + Makefile.msc build systems.
<a name=commands_to_run_tests></a>
## 3.1. Commands to Run SQLite Tests
The **mdevtest** command is equivalent to running the veryquick tests and
the `make fuzztest` target once for each of two --enable-all builds - one
with debugging enabled and one without:
```
tclsh $TESTDIR/testrunner.tcl mdevtest
```
In other words, it is equivalent to running:
```
$TOP/configure --enable-all --enable-debug
make fuzztest
make testfixture
./testfixture $TOP/test/testrunner.tcl veryquick
# Then, after removing files created by the tests above:
$TOP/configure --enable-all OPTS="-O0"
make fuzztest
make testfixture
./testfixture $TOP/test/testrunner.tcl veryquick
```
The **sdevtest** command is identical to the mdevtest command, except that the
second of the two builds is a sanitizer build. Specifically, this means that
OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":
```
tclsh $TESTDIR/testrunner.tcl sdevtest
```
The **release** command runs lots of tests under lots of builds. It runs
different combinations of builds and tests depending on whether it is run
on Linux, Windows or OSX. Refer to *testrunner\_data.tcl* for the details
of the specific tests run.
```
tclsh $TESTDIR/testrunner.tcl release
```
As with <a href=#source code tests>source code tests</a>, one or more patterns
may be appended to any of the above commands (mdevtest, sdevtest or release).
Pattern matching is used for both Tcl tests and fuzz tests.
```
tclsh $TESTDIR/testrunner.tcl release rtree%
```
<a name=zipvfs_tests></a>
## 3.2. Running ZipVFS Tests
testrunner.tcl can build a zipvfs-enabled testfixture and use it to run
tests from the Zipvfs project with the following command:
```
tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
```
This can be combined with any of "mdevtest", "sdevtest" or "release" to
test both SQLite and Zipvfs with a single command:
```
tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
```
<a name=source_code_test_failures></a>
## 3.3. Investigating Source Code Test Failures
Investigating a test failure that occurs during source code testing is a
two step process:
1. Recreating the build configuration in which the test failed, and
2. Re-running the actual test.
To recreate a build configuration, use the testrunner.tcl **script** command
to create a build script. A build script is a bash script on Linux or OSX, or
a dos \*.bat file on windows. For example:
```
# Create a script that recreates build configuration "Device-One" on
# Linux or OSX:
tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh
# Create a script that recreates build configuration "Have-Not" on Windows:
tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat
```
The generated bash or \*.bat file script accepts a single argument - a makefile
target to build. This may be used either to run a `make` command test directly,
or else to build a testfixture (or testfixture.exe) binary with which to
run a Tcl test script, as <a href=#binary_test_failures>described above</a>.
<a name=fuzzdb></a>
## 3.4 External Fuzzcheck Databases
Testrunner.tcl will also run fuzzcheck against an external (out of tree)
database, for example fuzzcheck databases generated by dbsqlfuzz. To do
this, simply add the "`--fuzzdb` *FILENAME*" command-line option or set
the FUZZDB environment variable to the name of the external
database. For large external databases, testrunner.tcl will automatically use
the "`--slice`" command-line option of fuzzcheck to divide the work up into
multiple jobs, to increase parallelism.
Thus, for example, to run a full releasetest including an external
dbsqlfuzz database, run a command like one of these:
```
tclsh test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
FUZZDB=../fuzz/20250415.db make releasetest
nmake /f Makefile.msc FUZZDB=../fuzz/20250415.db releasetest
```
The patternlist option to testrunner.tcl will match against fuzzcheck
databases. So if you want to run *only* tests involving the external
database, you can use a command something like this:
```
tclsh test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db
```
<a name=testrunner_options></a>
# 4. Extra testrunner.tcl Options
The testrunner.tcl script options in this section may be used with both source
code and binary tests.
The **--buildonly** option instructs testrunner.tcl just to build the binaries
required by a test, not to run any actual tests. For example:
```
# Build binaries required by release test.
tclsh $TESTDIR/testrunner.tcl --buildonly release"
```
The **--dryrun** option prevents testrunner.tcl from building any binaries
or running any tests. Instead, it just writes the shell commands that it
would normally execute into the testrunner.log file. Example:
```
# Log the shell commmands that make up the mdevtest test.
tclsh $TESTDIR/testrunner.tcl --dryrun mdevtest"
```
The **--explain** option is similar to --dryrun in that it prevents
testrunner.tcl from building any binaries or running any tests. The
difference is that --explain prints on standard output a human-readable
summary of all the builds and tests that would have been run.
```
# Show what builds and tests would have been run
tclsh $TESTDIR/testrunner.tcl --explain mdevtest
```
The **--status** option uses VT100 escape sequences to display the test
status full-screen. This is similar to running
"`watch test/testrunner status`" in a separate window, just more convenient.
Unfortunately, this option does not work correctly on Windows, due to the
sketchy implementation of VT100 escapes on the Windows console.
<a name=cpu_cores></a>
# 5. Controlling CPU Core Utilization
When running either binary or source code tests, testrunner.tcl reports the
number of jobs it intends to use to stdout. e.g.
```
$ ./testfixture $TESTDIR/testrunner.tcl
splitting work across 16 jobs
... more output ...
```
By default, testfixture.tcl attempts to set the number of jobs to the number
of real cores on the machine. This can be overridden using the "--jobs" (or -j)
switch:
```
$ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
splitting work across 8 jobs
... more output ...
```
The number of jobs may also be changed while an instance of testrunner.tcl is
running by exucuting the following command from the directory containing the
testrunner.log and testrunner.db files:
```
$ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
```
doc/trusted-schema.md
+142
View File
@@ -0,0 +1,142 @@
# The new-security-options branch
## The problem that the [new-security-options](/timeline?r=new-security-options) branch tries to solve
An attacker might modify the schema of an SQLite database by adding
structures that cause code to run when some other application opens and
reads the database. For example, the attacker might replace a table
definition with a view. Or the attacker might add triggers to tables
or views, or add new CHECK constraints or generated columns or indexes
with expressions in the index list or in the WHERE clause. If the
added features invoke SQL functions or virtual tables with side effects,
that might cause harm to the system if run by a high-privilege victim.
Or, the added features might exfiltrate information if the database is
read by a high-privilege victim.
The changes in this branch strive to make it easier for high-privilege
applications to safely read SQLite database files that might have been
maliciously corrupted by an attacker.
## Overview of changes in [new-security-options](/timeline?r=new-security-options)
The basic idea is to tag every SQL function and virtual table with one
of three risk levels:
1. Innocuous
2. Normal
3. Direct-Only
Innocuous functions/vtabs are safe and can be used at any time.
Direct-only elements, in contrast, might have cause side-effects and
should only be used from top-level SQL, not from within triggers or views nor
in elements of the schema such as CHECK constraint, DEFAULT values,
generated columns, index expressions, or in the WHERE clause of a
partial index that are potentially under the control of an attacker.
Normal elements behave like Innocuous if TRUSTED\_SCHEMA=on
and behave like direct-only if TRUSTED\_SCHEMA=off.
Application-defined functions and virtual tables go in as Normal unless
the application takes deliberate steps to change the risk level.
For backwards compatibility, the default is TRUSTED\_SCHEMA=on. Documentation
will be updated to recommend applications turn TRUSTED\_SCHEMA to off.
An innocuous function or virtual table is one that can only read content
from the database file in which it resides, and can only alter the database
in which it resides. Most SQL functions are innocuous. For example, there
is no harm in an attacker running the abs() function.
Direct-only elements that have side-effects that go outside the database file
in which it lives, or return information from outside of the database file.
Examples of direct-only elements include:
1. The fts3\_tokenizer() function
2. The writefile() function
3. The readfile() function
4. The zipvfs virtual table
5. The csv virtual table
We do not want an attacker to be able to add these kinds of things to
the database schema and possibly trick a high-privilege application
from performing any of these actions. Therefore, functions and vtabs
with side-effects are marked as Direct-Only.
Legacy applications might add other risky functions or vtabs. Those will
go in as "Normal" by default. For optimal security, we want those risky
app-defined functions and vtabs to be direct-only, but making that the
default might break some legacy applications. Hence, all app-defined
functions and vtabs go in as Normal, but the application can switch them
over to "Direct-Only" behavior using a single pragma.
The restrictions on the use of functions and virtual tables do not apply
to TEMP. A TEMP VIEW or a TEMP TRIGGER can use any valid SQL function
or virtual table. The idea is that TEMP views and triggers must be
directly created by the application and are thus under the control of the
application. TEMP views and triggers cannot be created by an attacker who
corrupts the schema of a persistent database file. Hence TEMP views and
triggers are safe.
## Specific changes
1. New sqlite3\_db\_config() option SQLITE\_DBCONFIG\_TRUSTED\_SCHEMA for
turning TRUSTED\_SCHEMA on and off. It defaults to ON.
2. Compile-time option -DSQLITE\_TRUSTED\_SCHEMA=0 causes the default
TRUSTED\_SCHEMA setting to be off.
3. New pragma "PRAGMA trusted\_schema=(ON\|OFF);". This provides access
to the TRUSTED_SCHEMA setting for application coded using scripting
languages or other secondary languages where they are unable to make
calls to sqlite3\_db\_config().
4. New options for the "enc" parameter to sqlite3\_create\_function() and
its kin:
<ol type="a">
<li> _SQLITE\_INNOCUOUS_ &rarr; tags the new functions as Innocuous
<li> _SQLITE\_DIRECTONLY_ &rarr; tags the new functions as Direct-Only
</ol>
5. New options to sqlite3\_vtab\_config():
<ol type="a">
<li> _SQLITE\_VTAB\_INNOCUOUS_ &rarr; tags the vtab as Innocuous
<li> _SQLITE\_VTAB\_DIRECTONLY_ &rarr; tags the vtab as Direct-Only
</ol>
6. Change many of the functions and virtual tables in the SQLite source
tree to use one of the tags above.
7. Enhanced PRAGMA function\_list and virtual-table "pragma\_function\_list"
with additional columns. The columns now are:
<ul>
<li> _name_ &rarr; Name of the function
<li> _builtin_ &rarr; 1 for built-in functions. 0 otherwise.
<li> _type_ &rarr; 's'=Scalar, 'a'=Aggregate, 'w'=Window
<li> _enc_ &rarr; 'utf8', 'utf16le', or 'utf16be'
<li> _narg_ &rarr; number of argument
<li> _flags_ &rarr; Bitmask of SQLITE\_INNOCUOUS, SQLITE\_DIRECTONLY,
SQLITE\_DETERMINISTIC, SQLITE\_SUBTYPE, and
SQLITE\_FUNC\_INTERNAL flags.
</ul>
<p>The last four columns are new.
8. The function\_list PRAGMA now also shows all entries for each function.
So, for example, if a function can take either 2 or 3 arguments,
there are separate rows for the 2-argument and 3-argument versions of
the function.
## Additional Notes
The function_list enhancements allow the application to query the set
of SQL functions that meet various criteria. For example, to see all
SQL functions that are never allowed to be used in the schema or in
trigger or views:
~~~
SELECT DISTINCT name FROM pragma_function_list
WHERE (flags & 0x80000)!=0
ORDER BY name;
~~~
Doing the same is not possible for virtual tables, as a virtual table
might be Innocuous, Normal, or Direct-Only depending on the arguments
passed into the xConnect method.
doc/vdbesort-memory.md
+49
View File
@@ -0,0 +1,49 @@
20-11-2020
# Memory Allocation In vdbesort.c
Memory allocation is slightly different depending on:
* whether or not SQLITE_CONFIG_SMALL_MALLOC is set, and
* whether or not worker threads are enabled.
## SQLITE_CONFIG_SMALL_MALLOC=0
Assuming SQLITE_CONFIG_SMALL_MALLOC is not set, keys passed to the sorter are
added to an in-memory buffer. This buffer is grown using sqlite3Realloc() as
required it reaches the size configured for the main pager cache using "PRAGMA
cache_size". i.e. if the user has executed "PRAGMA main.cache_size = -2048",
then this buffer is allowed to grow up to 2MB in size.
Once the buffer has grown to its threshold, keys are sorted and written to
a temp file. If worker threads are not enabled, this is the only significant
allocation the sorter module makes. After keys are sorted and flushed out to
the temp file, the buffer is reused to accumulate the next batch of keys.
If worker threads are available, then the buffer is passed to a worker thread
to sort and flush once it is full, and a new buffer allocated to allow the
main thread to continue to accumulate keys. Buffers are reused once they
have been flushed, so in this case at most (nWorker+1) buffers are allocated
and used, where nWorker is the number of configured worker threads.
There are no other significant users of heap memory in the sorter module.
Once sorted buffers of keys have been flushed to disk, they are read back
either by mapping the file (via sqlite3_file.xFetch()) or else read back
in one page at a time.
All buffers are allocated by the main thread. A sorter object is associated
with a single database connection, to which it holds a pointer.
## SQLITE_CONFIG_SMALL_MALLOC=1
This case is similar to the above, except that instead of accumulating
multiple keys in a single large buffer, sqlite3VdbeSorterWrite() stores
keys in a regular heap-memory linked list (one allocation per element).
List elements are freed as they are flushed to disk, either by the main
thread or by a worker thread.
Each time a key is added the sorter (and an allocation made),
sqlite3HeapNearlyFull() is called. If it returns true, the current
list of keys is flushed to a temporary file, even if it has not yet
reached the size threshold.
doc/vfs-shm.txt
+130
View File
@@ -0,0 +1,130 @@
The 5 states of an historical rollback lock as implemented by the
xLock, xUnlock, and xCheckReservedLock methods of the sqlite3_io_methods
object are:
UNLOCKED
SHARED
RESERVED
PENDING
EXCLUSIVE
The wal-index file has a similar locking hierarchy implemented using
the xShmLock method of the sqlite3_vfs object, but with 7
states. Each connection to a wal-index file must be in one of
the following 7 states:
UNLOCKED
READ
READ_FULL
WRITE
PENDING
CHECKPOINT
RECOVER
These roughly correspond to the 5 states of a rollback lock except
that SHARED is split out into 2 states: READ and READ_FULL and
there is an extra RECOVER state used for wal-index reconstruction.
The meanings of the various wal-index locking states is as follows:
UNLOCKED - The wal-index is not in use.
READ - Some prefix of the wal-index is being read. Additional
wal-index information can be appended at any time. The
newly appended content will be ignored by the holder of
the READ lock.
READ_FULL - The entire wal-index is being read. No new information
can be added to the wal-index. The holder of a READ_FULL
lock promises never to read pages from the database file
that are available anywhere in the wal-index.
WRITE - It is OK to append to the wal-index file and to adjust
the header to indicate the new "last valid frame".
PENDING - Waiting on all READ locks to clear so that a
CHECKPOINT lock can be acquired.
CHECKPOINT - It is OK to write any WAL data into the database file
and zero the last valid frame field of the wal-index
header. The wal-index file itself may not be changed
other than to zero the last valid frame field in the
header.
RECOVER - Held during wal-index recovery. Used to prevent a
race if multiple clients try to recover a wal-index at
the same time.
A particular lock manager implementation may coalesce one or more of
the wal-index locking states, though with a reduction in concurrency.
For example, an implementation might implement only exclusive locking,
in which case all states would be equivalent to CHECKPOINT, meaning that
only one reader or one writer or one checkpointer could be active at a
time. Or, an implementation might combine READ and READ_FULL into
a single state equivalent to READ, meaning that a writer could
coexist with a reader, but no reader or writers could coexist with a
checkpointer.
The lock manager must obey the following rules:
(1) A READ cannot coexist with CHECKPOINT.
(2) A READ_FULL cannot coexist with WRITE.
(3) None of WRITE, PENDING, CHECKPOINT, or RECOVER can coexist.
The SQLite core will obey the next set of rules. These rules are
assertions on the behavior of the SQLite core which might be verified
during testing using an instrumented lock manager.
(5) No part of the wal-index will be read without holding either some
kind of SHM lock or an EXCLUSIVE lock on the original database.
The original database is the file named in the 2nd parameter to
the xShmOpen method.
(6) A holder of a READ_FULL will never read any page of the database
file that is contained anywhere in the wal-index.
(7) No part of the wal-index other than the header will be written nor
will the size of the wal-index grow without holding a WRITE or
an EXCLUSIVE on the original database file.
(8) The wal-index header will not be written without holding one of
WRITE, CHECKPOINT, or RECOVER on the wal-index or an EXCLUSIVE on
the original database files.
(9) A CHECKPOINT or RECOVER must be held on the wal-index, or an
EXCLUSIVE on the original database file, in order to reset the
last valid frame counter in the header of the wal-index back to zero.
(10) A WRITE can only increase the last valid frame pointer in the header.
The SQLite core will only ever send requests for UNLOCK, READ, WRITE,
CHECKPOINT, or RECOVER to the lock manager. The SQLite core will never
request a READ_FULL or PENDING lock though the lock manager may deliver
those locking states in response to READ and CHECKPOINT requests,
respectively, if and only if the requested READ or CHECKPOINT cannot
be delivered.
The following are the allowed lock transitions:
Original-State Request New-State
-------------- ---------- ----------
(11a) UNLOCK READ READ
(11b) UNLOCK READ READ_FULL
(11c) UNLOCK CHECKPOINT PENDING
(11d) UNLOCK CHECKPOINT CHECKPOINT
(11e) READ UNLOCK UNLOCK
(11f) READ WRITE WRITE
(11g) READ RECOVER RECOVER
(11h) READ_FULL UNLOCK UNLOCK
(11i) READ_FULL WRITE WRITE
(11j) READ_FULL RECOVER RECOVER
(11k) WRITE READ READ
(11l) PENDING UNLOCK UNLOCK
(11m) PENDING CHECKPOINT CHECKPOINT
(11n) CHECKPOINT UNLOCK UNLOCK
(11o) RECOVER READ READ
These 15 transitions are all that needs to be supported. The lock
manager implementation can assert that fact. The other 27 possible
transitions among the 7 locking states will never occur.
doc/wal-lock.md
+88
View File
@@ -0,0 +1,88 @@
# Wal-Mode Blocking Locks
On some Unix-like systems, SQLite may be configured to use POSIX blocking locks
by:
* building the library with SQLITE\_ENABLE\_SETLK\_TIMEOUT defined, and
* configuring a timeout in ms using the sqlite3\_busy\_timeout() API.
Blocking locks may be advantageous as (a) waiting database clients do not
need to continuously poll the database lock, and (b) using blocking locks
facilitates transfer of OS priority between processes when a high priority
process is blocked by a lower priority one.
Only read/write clients use blocking locks. Clients that have read-only access
to the \*-shm file never use blocking locks.
Threads or processes that access a single database at a time never deadlock as
a result of blocking database locks. But it is of course possible for threads
that lock multiple databases simultaneously to do so. In most cases the OS will
detect the deadlock and return an error.
## Wal Recovery
Wal database "recovery" is a process required when the number of connected
database clients changes from zero to one. In this case, a client is
considered to connect to the database when it first reads data from it.
Before recovery commences, an exclusive WRITER lock is taken.
Without blocking locks, if two clients attempt recovery simultaneously, one
fails to obtain the WRITER lock and either invokes the busy-handler callback or
returns SQLITE\_BUSY to the user. With blocking locks configured, the second
client blocks on the WRITER lock.
## Database Readers
Usually, read-only are not blocked by any other database clients, so they
have no need of blocking locks.
If a read-only transaction is being opened on a snapshot, the CHECKPOINTER
lock is required briefly as part of opening the transaction (to check that a
checkpointer is not currently overwriting the snapshot being opened). A
blocking lock is used to obtain the CHECKPOINTER lock in this case. A snapshot
opener may therefore block on and transfer priority to a checkpointer in some
cases.
## Database Writers
A database writer must obtain the exclusive WRITER lock. It uses a blocking
lock to do so if any of the following are true:
* the transaction is an implicit one consisting of a single DML or DDL
statement, or
* the transaction is opened using BEGIN IMMEDIATE or BEGIN EXCLUSIVE, or
* the first SQL statement executed following the BEGIN command is a DML or
DDL statement (not a read-only statement like a SELECT).
In other words, in all cases except when an open read-transaction is upgraded
to a write-transaction. In that case a non-blocking lock is used.
## Database Checkpointers
Database checkpointers takes the following locks, in order:
* The exclusive CHECKPOINTER lock.
* The exclusive WRITER lock (FULL, RESTART and TRUNCATE only).
* Exclusive lock on read-mark slots 1-N. These are immediately released after being taken.
* Exclusive lock on read-mark 0.
* Exclusive lock on read-mark slots 1-N again. These are immediately released
after being taken (RESTART and TRUNCATE only).
All of the above use blocking locks.
## Summary
With blocking locks configured, the only cases in which clients should see an
SQLITE\_BUSY error are:
* if the OS does not grant a blocking lock before the configured timeout
expires, and
* when an open read-transaction is upgraded to a write-transaction.
In all other cases the blocking locks implementation should prevent clients
from having to handle SQLITE\_BUSY errors and facilitate appropriate transfer
of priorities between competing clients.
Clients that lock multiple databases simultaneously must be wary of deadlock.
ext/README.md
+8
View File
@@ -0,0 +1,8 @@
## Loadable Extensions
Various [loadable extensions](https://sqlite.org/loadext.html) for
SQLite are found in subfolders.
Most subfolders are dedicated to a single loadable extension (for
example FTS5, or RTREE). But the misc/ subfolder contains a collection
of smaller single-file extensions.
ext/expert/README.md
+83
View File
@@ -0,0 +1,83 @@
## SQLite Expert Extension
This folder contains code for a simple system to propose useful indexes
given a database and a set of SQL queries. It works as follows:
1. The user database schema is copied to a temporary database.
1. All SQL queries are prepared against the temporary database.
Information regarding the WHERE and ORDER BY clauses, and other query
features that affect index selection are recorded.
1. The information gathered in step 2 is used to create candidate
indexes - indexes that the planner might have made use of in the previous
step, had they been available.
1. A subset of the data in the user database is used to generate statistics
for all existing indexes and the candidate indexes generated in step 3
above.
1. The SQL queries are prepared a second time. If the planner uses any
of the indexes created in step 3, they are recommended to the user.
# C API
The SQLite expert C API is defined in sqlite3expert.h. Most uses will proceed
as follows:
1. An sqlite3expert object is created by calling **sqlite3\_expert\_new()**.
A database handle opened by the user is passed as an argument.
1. The sqlite3expert object is configured with one or more SQL statements
by making one or more calls to **sqlite3\_expert\_sql()**. Each call may
specify a single SQL statement, or multiple statements separated by
semi-colons.
1. Optionally, the **sqlite3\_expert\_config()** API may be used to
configure the size of the data subset used to generate index statistics.
Using a smaller subset of the data can speed up the analysis.
1. **sqlite3\_expert\_analyze()** is called to run the analysis.
1. One or more calls are made to **sqlite3\_expert\_report()** to extract
components of the results of the analysis.
1. **sqlite3\_expert\_destroy()** is called to free all resources.
Refer to comments in sqlite3expert.h for further details.
# sqlite3_expert application
The file "expert.c" contains the code for a command line application that
uses the API described above. It can be compiled with (for example):
<pre>
gcc -O2 sqlite3.c expert.c sqlite3expert.c -o sqlite3_expert
</pre>
Assuming the database is named "test.db", it can then be run to analyze a
single query:
<pre>
./sqlite3_expert -sql &lt;sql-query&gt; test.db
</pre>
Or an entire text file worth of queries with:
<pre>
./sqlite3_expert -file &lt;text-file&gt; test.db
</pre>
By default, sqlite3\_expert generates index statistics using all the data in
the user database. For a large database, this may be prohibitively time
consuming. The "-sample" option may be used to configure sqlite3\_expert to
generate statistics based on an integer percentage of the user database as
follows:
<pre>
# Generate statistics based on 25% of the user database rows:
./sqlite3_expert -sample 25 -sql &lt;sql-query&gt; test.db
# Do not generate any statistics at all:
./sqlite3_expert -sample 0 -sql &lt;sql-query&gt; test.db
</pre>
ext/expert/expert.c
+156
View File
@@ -0,0 +1,156 @@
/*
** 2017 April 07
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
*/
#include <sqlite3.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "sqlite3expert.h"
static void option_requires_argument(const char *zOpt){
fprintf(stderr, "Option requires an argument: %s\n", zOpt);
exit(-3);
}
static int option_integer_arg(const char *zVal){
return atoi(zVal);
}
static void usage(char **argv){
fprintf(stderr, "\n");
fprintf(stderr, "Usage %s ?OPTIONS? DATABASE\n", argv[0]);
fprintf(stderr, "\n");
fprintf(stderr, "Options are:\n");
fprintf(stderr, " -sql SQL (analyze SQL statements passed as argument)\n");
fprintf(stderr, " -file FILE (read SQL statements from file FILE)\n");
fprintf(stderr, " -verbose LEVEL (integer verbosity level. default 1)\n");
fprintf(stderr, " -sample PERCENT (percent of db to sample. default 100)\n");
exit(-1);
}
static int readSqlFromFile(sqlite3expert *p, const char *zFile, char **pzErr){
FILE *in = fopen(zFile, "rb");
long nIn;
size_t nRead;
char *pBuf;
int rc;
if( in==0 ){
*pzErr = sqlite3_mprintf("failed to open file %s\n", zFile);
return SQLITE_ERROR;
}
fseek(in, 0, SEEK_END);
nIn = ftell(in);
rewind(in);
pBuf = sqlite3_malloc64( nIn+1 );
nRead = fread(pBuf, nIn, 1, in);
fclose(in);
if( nRead!=1 ){
sqlite3_free(pBuf);
*pzErr = sqlite3_mprintf("failed to read file %s\n", zFile);
return SQLITE_ERROR;
}
pBuf[nIn] = 0;
rc = sqlite3_expert_sql(p, pBuf, pzErr);
sqlite3_free(pBuf);
return rc;
}
int main(int argc, char **argv){
const char *zDb;
int rc = 0;
char *zErr = 0;
int i;
int iVerbose = 1; /* -verbose option */
sqlite3 *db = 0;
sqlite3expert *p = 0;
if( argc<2 ) usage(argv);
zDb = argv[argc-1];
if( zDb[0]=='-' ) usage(argv);
rc = sqlite3_open(zDb, &db);
if( rc!=SQLITE_OK ){
fprintf(stderr, "Cannot open db file: %s - %s\n", zDb, sqlite3_errmsg(db));
exit(-2);
}
p = sqlite3_expert_new(db, &zErr);
if( p==0 ){
fprintf(stderr, "Cannot run analysis: %s\n", zErr);
rc = 1;
}else{
for(i=1; i<(argc-1); i++){
char *zArg = argv[i];
int nArg;
if( zArg[0]=='-' && zArg[1]=='-' && zArg[2]!=0 ) zArg++;
nArg = (int)strlen(zArg);
if( nArg>=2 && 0==sqlite3_strnicmp(zArg, "-file", nArg) ){
if( ++i==(argc-1) ) option_requires_argument("-file");
rc = readSqlFromFile(p, argv[i], &zErr);
}
else if( nArg>=3 && 0==sqlite3_strnicmp(zArg, "-sql", nArg) ){
if( ++i==(argc-1) ) option_requires_argument("-sql");
rc = sqlite3_expert_sql(p, argv[i], &zErr);
}
else if( nArg>=3 && 0==sqlite3_strnicmp(zArg, "-sample", nArg) ){
int iSample;
if( ++i==(argc-1) ) option_requires_argument("-sample");
iSample = option_integer_arg(argv[i]);
sqlite3_expert_config(p, EXPERT_CONFIG_SAMPLE, iSample);
}
else if( nArg>=2 && 0==sqlite3_strnicmp(zArg, "-verbose", nArg) ){
if( ++i==(argc-1) ) option_requires_argument("-verbose");
iVerbose = option_integer_arg(argv[i]);
}
else{
usage(argv);
}
}
}
if( rc==SQLITE_OK ){
rc = sqlite3_expert_analyze(p, &zErr);
}
if( rc==SQLITE_OK ){
int nQuery = sqlite3_expert_count(p);
if( iVerbose>0 ){
const char *zCand = sqlite3_expert_report(p,0,EXPERT_REPORT_CANDIDATES);
fprintf(stdout, "-- Candidates -------------------------------\n");
fprintf(stdout, "%s\n", zCand);
}
for(i=0; i<nQuery; i++){
const char *zSql = sqlite3_expert_report(p, i, EXPERT_REPORT_SQL);
const char *zIdx = sqlite3_expert_report(p, i, EXPERT_REPORT_INDEXES);
const char *zEQP = sqlite3_expert_report(p, i, EXPERT_REPORT_PLAN);
if( zIdx==0 ) zIdx = "(no new indexes)\n";
if( iVerbose>0 ){
fprintf(stdout, "-- Query %d ----------------------------------\n",i+1);
fprintf(stdout, "%s\n\n", zSql);
}
fprintf(stdout, "%s\n%s\n", zIdx, zEQP);
}
}else{
fprintf(stderr, "Error: %s\n", zErr ? zErr : "?");
}
sqlite3_expert_destroy(p);
sqlite3_free(zErr);
return rc;
}
ext/expert/expert1.test
+609
View File
@@ -0,0 +1,609 @@
# 2009 Nov 11
#
# The author disclaims copyright to this source code. In place of
# a legal notice, here is a blessing:
#
# May you do good and not evil.
# May you find forgiveness for yourself and forgive others.
# May you share freely, never taking more than you give.
#
#***********************************************************************
# TESTRUNNER: shell
#
# The focus of this file is testing the CLI shell tool. Specifically,
# the ".recommend" command.
#
#
# Test plan:
#
#
if {![info exists testdir]} {
set testdir [file join [file dirname [info script]] .. .. test]
}
source $testdir/tester.tcl
set testprefix expert1
if {[info commands sqlite3_expert_new]==""} {
finish_test
return
}
set CLI [test_binary_name sqlite3]
set CMD [test_binary_name sqlite3_expert]
proc squish {txt} {
regsub -all {[[:space:]]+} $txt { }
}
proc do_setup_rec_test {tn setup sql res} {
reset_db
if {[info exists ::set_main_db_name]} {
dbconfig_maindbname_icecube db
}
db eval $setup
uplevel [list do_rec_test $tn $sql $res]
}
foreach {tn setup} {
1 {
if {![file executable $CMD]} { continue }
proc do_rec_test {tn sql res} {
set res [squish [string trim $res]]
set tst [subst -nocommands {
squish [string trim [exec $::CMD -verbose 0 -sql {$sql;} test.db]]
}]
uplevel [list do_test $tn $tst $res]
}
}
2 {
if {[info commands sqlite3_expert_new]==""} { continue }
proc do_rec_test {tn sql res} {
set expert [sqlite3_expert_new db]
$expert sql $sql
$expert analyze
set result [list]
for {set i 0} {$i < [$expert count]} {incr i} {
set idx [string trim [$expert report $i indexes]]
if {$idx==""} {set idx "(no new indexes)"}
lappend result $idx
lappend result [string trim [$expert report $i plan]]
}
$expert destroy
set tst [subst -nocommands {set {} [squish [join {$result}]]}]
uplevel [list do_test $tn $tst [string trim [squish $res]]]
}
}
3 {
if {[info commands sqlite3_expert_new]==""} { continue }
set ::set_main_db_name 1
}
4 {
if {![file executable $CLI]} { continue }
proc do_rec_test {tn sql res} {
set res [squish [string trim $res]]
set tst [subst -nocommands {
squish [string trim [exec $::CLI test.db ".expert" {$sql;}]]
}]
uplevel [list do_test $tn $tst $res]
}
}
} {
eval $setup
do_setup_rec_test $tn.1 { CREATE TABLE t1(a, b, c) } {
SELECT * FROM t1
} {
(no new indexes)
SCAN t1
}
do_setup_rec_test $tn.2 {
CREATE TABLE t1(a, b, c);
} {
SELECT * FROM t1 WHERE b>?;
} {
CREATE INDEX t1_idx_00000062 ON t1(b);
SEARCH t1 USING INDEX t1_idx_00000062 (b>?)
}
do_setup_rec_test $tn.3 {
CREATE TABLE t1(a, b, c);
} {
SELECT * FROM t1 WHERE b COLLATE nocase BETWEEN ? AND ?
} {
CREATE INDEX t1_idx_3e094c27 ON t1(b COLLATE NOCASE);
SEARCH t1 USING INDEX t1_idx_3e094c27 (b>? AND b<?)
}
do_setup_rec_test $tn.4 {
CREATE TABLE t1(a, b, c);
} {
SELECT a FROM t1 ORDER BY b;
} {
CREATE INDEX t1_idx_00000062 ON t1(b);
SCAN t1 USING INDEX t1_idx_00000062
}
do_setup_rec_test $tn.5 {
CREATE TABLE t1(a, b, c);
} {
SELECT a FROM t1 WHERE a=? ORDER BY b;
} {
CREATE INDEX t1_idx_000123a7 ON t1(a, b);
SEARCH t1 USING COVERING INDEX t1_idx_000123a7 (a=?)
}
if 0 {
do_setup_rec_test $tn.6 {
CREATE TABLE t1(a, b, c);
} {
SELECT min(a) FROM t1
} {
CREATE INDEX t1_idx_00000061 ON t1(a);
SEARCH t1 USING COVERING INDEX t1_idx_00000061
}
}
do_setup_rec_test $tn.7 {
CREATE TABLE t1(a, b, c);
} {
SELECT * FROM t1 ORDER BY a, b, c;
} {
CREATE INDEX t1_idx_033e95fe ON t1(a, b, c);
SCAN t1 USING COVERING INDEX t1_idx_033e95fe
}
#do_setup_rec_test $tn.1.8 {
# CREATE TABLE t1(a, b, c);
#} {
# SELECT * FROM t1 ORDER BY a ASC, b COLLATE nocase DESC, c ASC;
#} {
# CREATE INDEX t1_idx_5be6e222 ON t1(a, b COLLATE NOCASE DESC, c);
# 0|0|0|SCAN t1 USING COVERING INDEX t1_idx_5be6e222
#}
do_setup_rec_test $tn.8.1 {
CREATE TABLE t1(a COLLATE NOCase, b, c);
} {
SELECT * FROM t1 WHERE a=?
} {
CREATE INDEX t1_idx_00000061 ON t1(a);
SEARCH t1 USING INDEX t1_idx_00000061 (a=?)
}
do_setup_rec_test $tn.8.2 {
CREATE TABLE t1(a, b COLLATE nocase, c);
} {
SELECT * FROM t1 ORDER BY a ASC, b DESC, c ASC;
} {
CREATE INDEX t1_idx_5cb97285 ON t1(a, b DESC, c);
SCAN t1 USING COVERING INDEX t1_idx_5cb97285
}
# Tables with names that require quotes.
#
do_setup_rec_test $tn.9.1 {
CREATE TABLE "t t"(a, b, c);
} {
SELECT * FROM "t t" WHERE a=?
} {
CREATE INDEX "t t_idx_00000061" ON "t t"(a);
SEARCH t t USING INDEX t t_idx_00000061 (a=?)
}
do_setup_rec_test $tn.9.2 {
CREATE TABLE "t t"(a, b, c);
} {
SELECT * FROM "t t" WHERE b BETWEEN ? AND ?
} {
CREATE INDEX "t t_idx_00000062" ON "t t"(b);
SEARCH t t USING INDEX t t_idx_00000062 (b>? AND b<?)
}
# Columns with names that require quotes.
#
do_setup_rec_test $tn.10.1 {
CREATE TABLE t3(a, "b b", c);
} {
SELECT * FROM t3 WHERE "b b" = ?
} {
CREATE INDEX t3_idx_00050c52 ON t3('b b');
SEARCH t3 USING INDEX t3_idx_00050c52 (b b=?)
}
do_setup_rec_test $tn.10.2 {
CREATE TABLE t3(a, "b b", c);
} {
SELECT * FROM t3 ORDER BY "b b"
} {
CREATE INDEX t3_idx_00050c52 ON t3('b b');
SCAN t3 USING INDEX t3_idx_00050c52
}
# Transitive constraints
#
do_setup_rec_test $tn.11.1 {
CREATE TABLE t5(a, b);
CREATE TABLE t6(c, d);
} {
SELECT * FROM t5, t6 WHERE a=? AND b=c AND c=?
} {
CREATE INDEX t5_idx_000123a7 ON t5(a, b);
CREATE INDEX t6_idx_00000063 ON t6(c);
SEARCH t6 USING INDEX t6_idx_00000063 (c=?)
SEARCH t5 USING COVERING INDEX t5_idx_000123a7 (a=? AND b=?)
}
# OR terms.
#
do_setup_rec_test $tn.12.1 {
CREATE TABLE t7(a, b);
} {
SELECT * FROM t7 WHERE a=? OR b=?
} {
CREATE INDEX t7_idx_00000062 ON t7(b);
CREATE INDEX t7_idx_00000061 ON t7(a);
MULTI-INDEX OR
INDEX 1
SEARCH t7 USING INDEX t7_idx_00000061 (a=?)
INDEX 2
SEARCH t7 USING INDEX t7_idx_00000062 (b=?)
}
# rowid terms.
#
do_setup_rec_test $tn.13.1 {
CREATE TABLE t8(a, b);
} {
SELECT * FROM t8 WHERE rowid=?
} {
(no new indexes)
SEARCH t8 USING INTEGER PRIMARY KEY (rowid=?)
}
do_setup_rec_test $tn.13.2 {
CREATE TABLE t8(a, b);
} {
SELECT * FROM t8 ORDER BY rowid
} {
(no new indexes)
SCAN t8
}
do_setup_rec_test $tn.13.3 {
CREATE TABLE t8(a, b);
} {
SELECT * FROM t8 WHERE a=? ORDER BY rowid
} {
CREATE INDEX t8_idx_00000061 ON t8(a);
SEARCH t8 USING INDEX t8_idx_00000061 (a=?)
}
# Triggers
#
do_setup_rec_test $tn.14 {
CREATE TABLE t9(a, b, c);
CREATE TABLE t10(a, b, c);
CREATE TRIGGER t9t AFTER INSERT ON t9 BEGIN
UPDATE t10 SET a=new.a WHERE b = new.b;
END;
} {
INSERT INTO t9 VALUES(?, ?, ?);
} {
CREATE INDEX t10_idx_00000062 ON t10(b);
SEARCH t10 USING INDEX t10_idx_00000062 (b=?)
}
do_setup_rec_test $tn.15 {
CREATE TABLE t1(a, b);
CREATE TABLE t2(c, d);
WITH s(i) AS ( VALUES(1) UNION ALL SELECT i+1 FROM s WHERE i<100)
INSERT INTO t1 SELECT (i-1)/50, (i-1)/20 FROM s;
WITH s(i) AS ( VALUES(1) UNION ALL SELECT i+1 FROM s WHERE i<100)
INSERT INTO t2 SELECT (i-1)/20, (i-1)/5 FROM s;
} {
SELECT * FROM t2, t1 WHERE b=? AND d=? AND t2.rowid=t1.rowid
} {
CREATE INDEX t2_idx_00000064 ON t2(d);
SEARCH t2 USING INDEX t2_idx_00000064 (d=?)
SEARCH t1 USING INTEGER PRIMARY KEY (rowid=?)
}
do_setup_rec_test $tn.16 {
CREATE TABLE t1(a, b);
} {
SELECT * FROM t1 WHERE b IS NOT NULL;
} {
(no new indexes)
SCAN t1
}
do_setup_rec_test $tn.17.1 {
CREATE TABLE example (A INTEGER, B INTEGER, C INTEGER, PRIMARY KEY (A,B));
} {
SELECT * FROM example WHERE a=?
} {
(no new indexes)
SEARCH example USING INDEX sqlite_autoindex_example_1 (A=?)
}
do_setup_rec_test $tn.17.2 {
CREATE TABLE example (A INTEGER, B INTEGER, C INTEGER, PRIMARY KEY (A,B));
} {
SELECT * FROM example WHERE b=?
} {
CREATE INDEX example_idx_00000042 ON example(B);
SEARCH example USING INDEX example_idx_00000042 (B=?)
}
do_setup_rec_test $tn.17.3 {
CREATE TABLE example (A INTEGER, B INTEGER, C INTEGER, PRIMARY KEY (A,B));
} {
SELECT * FROM example WHERE a=? AND b=?
} {
(no new indexes)
SEARCH example USING INDEX sqlite_autoindex_example_1 (A=? AND B=?)
}
do_setup_rec_test $tn.17.4 {
CREATE TABLE example (A INTEGER, B INTEGER, C INTEGER, PRIMARY KEY (A,B));
} {
SELECT * FROM example WHERE a=? AND b>?
} {
(no new indexes)
SEARCH example USING INDEX sqlite_autoindex_example_1 (A=? AND B>?)
}
do_setup_rec_test $tn.17.5 {
CREATE TABLE example (A INTEGER, B INTEGER, C INTEGER, PRIMARY KEY (A,B));
} {
SELECT * FROM example WHERE a>? AND b=?
} {
CREATE INDEX example_idx_0000cb3f ON example(B, A);
SEARCH example USING INDEX example_idx_0000cb3f (B=? AND A>?)
}
do_setup_rec_test $tn.18.0 {
CREATE TABLE SomeObject (
a INTEGER PRIMARY KEY,
x TEXT GENERATED ALWAYS AS(HEX(a)) VIRTUAL
);
} {
SELECT x FROM SomeObject;
} {
(no new indexes)
SCAN SomeObject
}
do_setup_rec_test $tn.18.1 {
CREATE TABLE SomeObject (
a INTEGER PRIMARY KEY,
x TEXT GENERATED ALWAYS AS(HEX(a)) VIRTUAL
);
} {
SELECT * FROM SomeObject WHERE x=?;
} {
CREATE INDEX SomeObject_idx_00000078 ON SomeObject(x);
SEARCH SomeObject USING COVERING INDEX SomeObject_idx_00000078 (x=?)
}
do_setup_rec_test $tn.19.0 {
CREATE TABLE t1("index");
} {
SELECT * FROM t1 ORDER BY "index";
} {
CREATE INDEX t1_idx_01a7214e ON t1('index');
SCAN t1 USING COVERING INDEX t1_idx_01a7214e
}
ifcapable fts5 {
do_setup_rec_test $tn.20.0 {
CREATE VIRTUAL TABLE ft USING fts5(a);
CREATE TABLE t1(x, y);
} {
SELECT * FROM ft, t1 WHERE a=x
} {
CREATE INDEX t1_idx_00000078 ON t1(x);
SCAN ft VIRTUAL TABLE INDEX 0:
SEARCH t1 USING INDEX t1_idx_00000078 (x=?)
}
}
}
proc do_candidates_test {tn sql res} {
set res [squish [string trim $res]]
set expert [sqlite3_expert_new db]
$expert sql $sql
$expert analyze
set candidates [squish [string trim [$expert report 0 candidates]]]
$expert destroy
uplevel [list do_test $tn [list set {} $candidates] $res]
}
reset_db
do_execsql_test 5.0 {
CREATE TABLE t1(a, b);
CREATE TABLE t2(c, d);
WITH s(i) AS ( VALUES(1) UNION ALL SELECT i+1 FROM s WHERE i<100)
INSERT INTO t1 SELECT (i-1)/50, (i-1)/20 FROM s;
WITH s(i) AS ( VALUES(1) UNION ALL SELECT i+1 FROM s WHERE i<100)
INSERT INTO t2 SELECT (i-1)/20, (i-1)/5 FROM s;
CREATE INDEX i1 ON t1( lower(a) );
}
do_candidates_test 5.1 {
SELECT * FROM t1,t2 WHERE (b=? OR a=?) AND (c=? OR d=?)
} {
CREATE INDEX t1_idx_00000062 ON t1(b); -- stat1: 100 20
CREATE INDEX t1_idx_00000061 ON t1(a); -- stat1: 100 50
CREATE INDEX t2_idx_00000063 ON t2(c); -- stat1: 100 20
CREATE INDEX t2_idx_00000064 ON t2(d); -- stat1: 100 5
}
do_candidates_test 5.2 {
SELECT * FROM t1,t2 WHERE a=? AND b=? AND c=? AND d=?
} {
CREATE INDEX t1_idx_000123a7 ON t1(a, b); -- stat1: 100 50 17
CREATE INDEX t2_idx_0001295b ON t2(c, d); -- stat1: 100 20 5
}
do_execsql_test 5.3 {
CREATE INDEX t1_idx_00000061 ON t1(a); -- stat1: 100 50
CREATE INDEX t1_idx_00000062 ON t1(b); -- stat1: 100 20
CREATE INDEX t1_idx_000123a7 ON t1(a, b); -- stat1: 100 50 16
CREATE INDEX t2_idx_00000063 ON t2(c); -- stat1: 100 20
CREATE INDEX t2_idx_00000064 ON t2(d); -- stat1: 100 5
CREATE INDEX t2_idx_0001295b ON t2(c, d); -- stat1: 100 20 5
ANALYZE;
SELECT * FROM sqlite_stat1 ORDER BY 1, 2;
} {
t1 i1 {100 50}
t1 t1_idx_00000061 {100 50}
t1 t1_idx_00000062 {100 20}
t1 t1_idx_000123a7 {100 50 17}
t2 t2_idx_00000063 {100 20}
t2 t2_idx_00000064 {100 5}
t2 t2_idx_0001295b {100 20 5}
}
do_catchsql_test 5.4 {
SELECT sqlite_expert_rem(123, 123);
} {1 {no such function: sqlite_expert_rem}}
do_catchsql_test 5.5 {
SELECT sqlite_expert_sample();
} {1 {no such function: sqlite_expert_sample}}
if 0 {
do_test expert1-6.0 {
catchcmd :memory: {
.expert
select base64('');
.expert
select name from pragma_collation_list order by name collate uint;
}
} {0 {(no new indexes)
SCAN CONSTANT ROW
(no new indexes)
SCAN pragma_collation_list VIRTUAL TABLE INDEX 0:
USE TEMP B-TREE FOR ORDER BY
}}
}
do_execsql_test 6.0 {
CREATE TABLE x1(a, b, c, d);
CREATE INDEX x1ab ON x1(a, lower(b));
CREATE INDEX x1dcba ON x1(d, b+c, a);
}
do_candidates_test 6.1 {
SELECT * FROM x1 WHERE b=? ORDER BY a;
} {
CREATE INDEX x1_idx_0001267f ON x1(b, a);
CREATE INDEX x1_idx_00000062 ON x1(b);
}
#-------------------------------------------------------------------------
ifcapable fts5 {
reset_db
do_execsql_test 7.0 {
CREATE VIRTUAL TABLE ft USING fts5(a);
CREATE TABLE t1(x, y);
}
do_candidates_test 7.1 {
SELECT * FROM ft, t1 WHERE a=x
} {
CREATE INDEX t1_idx_00000078 ON t1(x);
}
register_tcl_module db
proc vtab_command {method args} {
global G
switch -- $method {
xConnect {
return "CREATE TABLE t1(a, b, c);"
}
xBestIndex {
return [list]
}
xFilter {
return [list sql "SELECT rowid, * FROM t0"]
}
}
return {}
}
do_execsql_test 7.2 {
CREATE TABLE t0(a, b, c);
INSERT INTO t0 VALUES(1, 2, 3), (11, 22, 33);
CREATE VIRTUAL TABLE t2 USING tcl(vtab_command);
}
do_execsql_test 7.3 {
SELECT * FROM t2
} {
1 2 3
11 22 33
}
do_candidates_test 7.4 {
SELECT * FROM ft, t1 WHERE a=x
} {
CREATE INDEX t1_idx_00000078 ON t1(x);
}
do_test 7.5 {
set expert [sqlite3_expert_new db]
list [catch { $expert sql "SELECT * FROM ft, t2 WHERE b=1" } msg] $msg
} {1 {no such table: t2}}
$expert destroy
reset_db
do_execsql_test 7.6 {
BEGIN TRANSACTION;
CREATE TABLE IF NOT EXISTS 'bfts_idx_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'fts_idx_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO fts_idx_data VALUES(1,X'');
INSERT INTO fts_idx_data VALUES(10,X'00000000ff000001000000');
CREATE TABLE IF NOT EXISTS 'fts_idx_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'fts_idx_docsize'(id INTEGER PRIMARY KEY, sz BLOB, origin INTEGER);
CREATE TABLE IF NOT EXISTS 'fts_idx_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO fts_idx_config VALUES('version',4);
PRAGMA writable_schema=ON;
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','fts_idx','fts_idx',0,'CREATE VIRTUAL TABLE fts_idx USING fts5(Title, Description, Channel, Tags, content='''', contentless_delete=1)');
CREATE TABLE f(x BLOB, y);
COMMIT;
PRAGMA writable_schema = RESET;
}
do_candidates_test 7.4 {
SELECT * FROM fts_idx, f WHERE x = fts_idx.Channel
} {
CREATE INDEX f_idx_00000078 ON f(x);
}
}
finish_test
ext/expert/sqlite3expert.c
+2234
View File
File diff suppressed because it is too large. Load diff
ext/expert/sqlite3expert.h
+168
View File
@@ -0,0 +1,168 @@
/*
** 2017 April 07
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
*/
#if !defined(SQLITEEXPERT_H)
#define SQLITEEXPERT_H 1
#include "sqlite3.h"
typedef struct sqlite3expert sqlite3expert;
/*
** Create a new sqlite3expert object.
**
** If successful, a pointer to the new object is returned and (*pzErr) set
** to NULL. Or, if an error occurs, NULL is returned and (*pzErr) set to
** an English-language error message. In this case it is the responsibility
** of the caller to eventually free the error message buffer using
** sqlite3_free().
*/
sqlite3expert *sqlite3_expert_new(sqlite3 *db, char **pzErr);
/*
** Configure an sqlite3expert object.
**
** EXPERT_CONFIG_SAMPLE:
** By default, sqlite3_expert_analyze() generates sqlite_stat1 data for
** each candidate index. This involves scanning and sorting the entire
** contents of each user database table once for each candidate index
** associated with the table. For large databases, this can be
** prohibitively slow. This option allows the sqlite3expert object to
** be configured so that sqlite_stat1 data is instead generated based on a
** subset of each table, or so that no sqlite_stat1 data is used at all.
**
** A single integer argument is passed to this option. If the value is less
** than or equal to zero, then no sqlite_stat1 data is generated or used by
** the analysis - indexes are recommended based on the database schema only.
** Or, if the value is 100 or greater, complete sqlite_stat1 data is
** generated for each candidate index (this is the default). Finally, if the
** value falls between 0 and 100, then it represents the percentage of user
** table rows that should be considered when generating sqlite_stat1 data.
**
** Examples:
**
** // Do not generate any sqlite_stat1 data
** sqlite3_expert_config(pExpert, EXPERT_CONFIG_SAMPLE, 0);
**
** // Generate sqlite_stat1 data based on 10% of the rows in each table.
** sqlite3_expert_config(pExpert, EXPERT_CONFIG_SAMPLE, 10);
*/
int sqlite3_expert_config(sqlite3expert *p, int op, ...);
#define EXPERT_CONFIG_SAMPLE 1 /* int */
/*
** Specify zero or more SQL statements to be included in the analysis.
**
** Buffer zSql must contain zero or more complete SQL statements. This
** function parses all statements contained in the buffer and adds them
** to the internal list of statements to analyze. If successful, SQLITE_OK
** is returned and (*pzErr) set to NULL. Or, if an error occurs - for example
** due to a error in the SQL - an SQLite error code is returned and (*pzErr)
** may be set to point to an English language error message. In this case
** the caller is responsible for eventually freeing the error message buffer
** using sqlite3_free().
**
** If an error does occur while processing one of the statements in the
** buffer passed as the second argument, none of the statements in the
** buffer are added to the analysis.
**
** This function must be called before sqlite3_expert_analyze(). If a call
** to this function is made on an sqlite3expert object that has already
** been passed to sqlite3_expert_analyze() SQLITE_MISUSE is returned
** immediately and no statements are added to the analysis.
*/
int sqlite3_expert_sql(
sqlite3expert *p, /* From a successful sqlite3_expert_new() */
const char *zSql, /* SQL statement(s) to add */
char **pzErr /* OUT: Error message (if any) */
);
/*
** This function is called after the sqlite3expert object has been configured
** with all SQL statements using sqlite3_expert_sql() to actually perform
** the analysis. Once this function has been called, it is not possible to
** add further SQL statements to the analysis.
**
** If successful, SQLITE_OK is returned and (*pzErr) is set to NULL. Or, if
** an error occurs, an SQLite error code is returned and (*pzErr) set to
** point to a buffer containing an English language error message. In this
** case it is the responsibility of the caller to eventually free the buffer
** using sqlite3_free().
**
** If an error does occur within this function, the sqlite3expert object
** is no longer useful for any purpose. At that point it is no longer
** possible to add further SQL statements to the object or to re-attempt
** the analysis. The sqlite3expert object must still be freed using a call
** sqlite3_expert_destroy().
*/
int sqlite3_expert_analyze(sqlite3expert *p, char **pzErr);
/*
** Return the total number of statements loaded using sqlite3_expert_sql().
** The total number of SQL statements may be different from the total number
** to calls to sqlite3_expert_sql().
*/
int sqlite3_expert_count(sqlite3expert*);
/*
** Return a component of the report.
**
** This function is called after sqlite3_expert_analyze() to extract the
** results of the analysis. Each call to this function returns either a
** NULL pointer or a pointer to a buffer containing a nul-terminated string.
** The value passed as the third argument must be one of the EXPERT_REPORT_*
** #define constants defined below.
**
** For some EXPERT_REPORT_* parameters, the buffer returned contains
** information relating to a specific SQL statement. In these cases that
** SQL statement is identified by the value passed as the second argument.
** SQL statements are numbered from 0 in the order in which they are parsed.
** If an out-of-range value (less than zero or equal to or greater than the
** value returned by sqlite3_expert_count()) is passed as the second argument
** along with such an EXPERT_REPORT_* parameter, NULL is always returned.
**
** EXPERT_REPORT_SQL:
** Return the text of SQL statement iStmt.
**
** EXPERT_REPORT_INDEXES:
** Return a buffer containing the CREATE INDEX statements for all recommended
** indexes for statement iStmt. If there are no new recommeded indexes, NULL
** is returned.
**
** EXPERT_REPORT_PLAN:
** Return a buffer containing the EXPLAIN QUERY PLAN output for SQL query
** iStmt after the proposed indexes have been added to the database schema.
**
** EXPERT_REPORT_CANDIDATES:
** Return a pointer to a buffer containing the CREATE INDEX statements
** for all indexes that were tested (for all SQL statements). The iStmt
** parameter is ignored for EXPERT_REPORT_CANDIDATES calls.
*/
const char *sqlite3_expert_report(sqlite3expert*, int iStmt, int eReport);
/*
** Values for the third argument passed to sqlite3_expert_report().
*/
#define EXPERT_REPORT_SQL 1
#define EXPERT_REPORT_INDEXES 2
#define EXPERT_REPORT_PLAN 3
#define EXPERT_REPORT_CANDIDATES 4
/*
** Free an (sqlite3expert*) handle and all associated resources. There
** should be one call to this function for each successful call to
** sqlite3-expert_new().
*/
void sqlite3_expert_destroy(sqlite3expert*);
#endif /* !defined(SQLITEEXPERT_H) */
ext/expert/test_expert.c
+212
View File
@@ -0,0 +1,212 @@
/*
** 2017 April 07
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
*/
#if defined(SQLITE_TEST)
#include "sqlite3expert.h"
#include <assert.h>
#include <string.h>
#include "tclsqlite.h"
#ifndef SQLITE_OMIT_VIRTUALTABLE
/*
** Extract an sqlite3* db handle from the object passed as the second
** argument. If successful, set *pDb to point to the db handle and return
** TCL_OK. Otherwise, return TCL_ERROR.
*/
static int dbHandleFromObj(Tcl_Interp *interp, Tcl_Obj *pObj, sqlite3 **pDb){
Tcl_CmdInfo info;
if( 0==Tcl_GetCommandInfo(interp, Tcl_GetString(pObj), &info) ){
Tcl_AppendResult(interp, "no such handle: ", Tcl_GetString(pObj), NULL);
return TCL_ERROR;
}
*pDb = *(sqlite3 **)info.objClientData;
return TCL_OK;
}
/*
** Tclcmd: $expert sql SQL
** $expert analyze
** $expert count
** $expert report STMT EREPORT
** $expert destroy
*/
static int SQLITE_TCLAPI testExpertCmd(
void *clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[]
){
sqlite3expert *pExpert = (sqlite3expert*)clientData;
struct Subcmd {
const char *zSub;
int nArg;
const char *zMsg;
} aSub[] = {
{ "sql", 1, "TABLE", }, /* 0 */
{ "analyze", 0, "", }, /* 1 */
{ "count", 0, "", }, /* 2 */
{ "report", 2, "STMT EREPORT", }, /* 3 */
{ "destroy", 0, "", }, /* 4 */
{ 0 }
};
int iSub;
int rc = TCL_OK;
char *zErr = 0;
if( objc<2 ){
Tcl_WrongNumArgs(interp, 1, objv, "SUBCOMMAND ...");
return TCL_ERROR;
}
rc = Tcl_GetIndexFromObjStruct(interp,
objv[1], aSub, sizeof(aSub[0]), "sub-command", 0, &iSub
);
if( rc!=TCL_OK ) return rc;
if( objc!=2+aSub[iSub].nArg ){
Tcl_WrongNumArgs(interp, 2, objv, aSub[iSub].zMsg);
return TCL_ERROR;
}
switch( iSub ){
case 0: { /* sql */
char *zArg = Tcl_GetString(objv[2]);
rc = sqlite3_expert_sql(pExpert, zArg, &zErr);
break;
}
case 1: { /* analyze */
rc = sqlite3_expert_analyze(pExpert, &zErr);
break;
}
case 2: { /* count */
int n = sqlite3_expert_count(pExpert);
Tcl_SetObjResult(interp, Tcl_NewIntObj(n));
break;
}
case 3: { /* report */
const char *aEnum[] = {
"sql", "indexes", "plan", "candidates", 0
};
int iEnum;
int iStmt;
const char *zReport;
if( Tcl_GetIntFromObj(interp, objv[2], &iStmt)
|| Tcl_GetIndexFromObj(interp, objv[3], aEnum, "report", 0, &iEnum)
){
return TCL_ERROR;
}
assert( EXPERT_REPORT_SQL==1 );
assert( EXPERT_REPORT_INDEXES==2 );
assert( EXPERT_REPORT_PLAN==3 );
assert( EXPERT_REPORT_CANDIDATES==4 );
zReport = sqlite3_expert_report(pExpert, iStmt, 1+iEnum);
Tcl_SetObjResult(interp, Tcl_NewStringObj(zReport, -1));
break;
}
default: /* destroy */
assert( iSub==4 );
Tcl_DeleteCommand(interp, Tcl_GetString(objv[0]));
break;
}
if( rc!=TCL_OK ){
if( zErr ){
Tcl_SetObjResult(interp, Tcl_NewStringObj(zErr, -1));
}else{
extern const char *sqlite3ErrName(int);
Tcl_SetObjResult(interp, Tcl_NewStringObj(sqlite3ErrName(rc), -1));
}
}
sqlite3_free(zErr);
return rc;
}
static void SQLITE_TCLAPI testExpertDel(void *clientData){
sqlite3expert *pExpert = (sqlite3expert*)clientData;
sqlite3_expert_destroy(pExpert);
}
/*
** sqlite3_expert_new DB
*/
static int SQLITE_TCLAPI test_sqlite3_expert_new(
void * clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[]
){
static int iCmd = 0;
sqlite3 *db;
char *zCmd = 0;
char *zErr = 0;
sqlite3expert *pExpert;
int rc = TCL_OK;
if( objc!=2 ){
Tcl_WrongNumArgs(interp, 1, objv, "DB");
return TCL_ERROR;
}
if( dbHandleFromObj(interp, objv[1], &db) ){
return TCL_ERROR;
}
zCmd = sqlite3_mprintf("sqlite3expert%d", ++iCmd);
if( zCmd==0 ){
Tcl_AppendResult(interp, "out of memory", (char*)0);
return TCL_ERROR;
}
pExpert = sqlite3_expert_new(db, &zErr);
if( pExpert==0 ){
Tcl_AppendResult(interp, zErr, (char*)0);
rc = TCL_ERROR;
}else{
void *p = (void*)pExpert;
Tcl_CreateObjCommand(interp, zCmd, testExpertCmd, p, testExpertDel);
Tcl_SetObjResult(interp, Tcl_NewStringObj(zCmd, -1));
}
sqlite3_free(zCmd);
sqlite3_free(zErr);
return rc;
}
#endif /* ifndef SQLITE_OMIT_VIRTUALTABLE */
int TestExpert_Init(Tcl_Interp *interp){
#ifndef SQLITE_OMIT_VIRTUALTABLE
struct Cmd {
const char *zCmd;
Tcl_ObjCmdProc *xProc;
} aCmd[] = {
{ "sqlite3_expert_new", test_sqlite3_expert_new },
};
int i;
for(i=0; i<sizeof(aCmd)/sizeof(struct Cmd); i++){
struct Cmd *p = &aCmd[i];
Tcl_CreateObjCommand(interp, p->zCmd, p->xProc, 0, 0);
}
#endif
return TCL_OK;
}
#endif
ext/fts3/README.content
+176
View File
@@ -0,0 +1,176 @@
FTS4 CONTENT OPTION
Normally, in order to create a full-text index on a dataset, the FTS4
module stores a copy of all indexed documents in a specially created
database table.
As of SQLite version 3.7.9, FTS4 supports a new option - "content" -
designed to extend FTS4 to support the creation of full-text indexes where:
* The indexed documents are not stored within the SQLite database
at all (a "contentless" FTS4 table), or
* The indexed documents are stored in a database table created and
managed by the user (an "external content" FTS4 table).
Because the indexed documents themselves are usually much larger than
the full-text index, the content option can sometimes be used to achieve
significant space savings.
CONTENTLESS FTS4 TABLES
In order to create an FTS4 table that does not store a copy of the indexed
documents at all, the content option should be set to an empty string.
For example, the following SQL creates such an FTS4 table with three
columns - "a", "b", and "c":
CREATE VIRTUAL TABLE t1 USING fts4(content="", a, b, c);
Data can be inserted into such an FTS4 table using an INSERT statements.
However, unlike ordinary FTS4 tables, the user must supply an explicit
integer docid value. For example:
-- This statement is Ok:
INSERT INTO t1(docid, a, b, c) VALUES(1, 'a b c', 'd e f', 'g h i');
-- This statement causes an error, as no docid value has been provided:
INSERT INTO t1(a, b, c) VALUES('j k l', 'm n o', 'p q r');
It is not possible to UPDATE or DELETE a row stored in a contentless FTS4
table. Attempting to do so is an error.
Contentless FTS4 tables also support SELECT statements. However, it is
an error to attempt to retrieve the value of any table column other than
the docid column. The auxiliary function matchinfo() may be used, but
snippet() and offsets() may not. For example:
-- The following statements are Ok:
SELECT docid FROM t1 WHERE t1 MATCH 'xxx';
SELECT docid FROM t1 WHERE a MATCH 'xxx';
SELECT matchinfo(t1) FROM t1 WHERE t1 MATCH 'xxx';
-- The following statements all cause errors, as the value of columns
-- other than docid are required to evaluate them.
SELECT * FROM t1;
SELECT a, b FROM t1 WHERE t1 MATCH 'xxx';
SELECT docid FROM t1 WHERE a LIKE 'xxx%';
SELECT snippet(t1) FROM t1 WHERE t1 MATCH 'xxx';
Errors related to attempting to retrieve column values other than docid
are runtime errors that occur within sqlite3_step(). In some cases, for
example if the MATCH expression in a SELECT query matches zero rows, there
may be no error at all even if a statement does refer to column values
other than docid.
EXTERNAL CONTENT FTS4 TABLES
An "external content" FTS4 table is similar to a contentless table, except
that if evaluation of a query requires the value of a column other than
docid, FTS4 attempts to retrieve that value from a table (or view, or
virtual table) nominated by the user (hereafter referred to as the "content
table"). The FTS4 module never writes to the content table, and writing
to the content table does not affect the full-text index. It is the
responsibility of the user to ensure that the content table and the
full-text index are consistent.
An external content FTS4 table is created by setting the content option
to the name of a table (or view, or virtual table) that may be queried by
FTS4 to retrieve column values when required. If the nominated table does
not exist, then an external content table behaves in the same way as
a contentless table. For example:
CREATE TABLE t2(id INTEGER PRIMARY KEY, a, b, c);
CREATE VIRTUAL TABLE t3 USING fts4(content="t2", a, c);
Assuming the nominated table does exist, then its columns must be the same
as or a superset of those defined for the FTS table.
When a users query on the FTS table requires a column value other than
docid, FTS attempts to read this value from the corresponding column of
the row in the content table with a rowid value equal to the current FTS
docid. Or, if such a row cannot be found in the content table, a NULL
value is used instead. For example:
CREATE TABLE t2(id INTEGER PRIMARY KEY, a, b, c, d);
CREATE VIRTUAL TABLE t3 USING fts4(content="t2", b, c);
INSERT INTO t2 VALUES(2, 'a b', 'c d', 'e f');
INSERT INTO t2 VALUES(3, 'g h', 'i j', 'k l');
INSERT INTO t3(docid, b, c) SELECT id, b, c FROM t2;
-- The following query returns a single row with two columns containing
-- the text values "i j" and "k l".
--
-- The query uses the full-text index to discover that the MATCH
-- term matches the row with docid=3. It then retrieves the values
-- of columns b and c from the row with rowid=3 in the content table
-- to return.
--
SELECT * FROM t3 WHERE t3 MATCH 'k';
-- Following the UPDATE, the query still returns a single row, this
-- time containing the text values "xxx" and "yyy". This is because the
-- full-text index still indicates that the row with docid=3 matches
-- the FTS4 query 'k', even though the documents stored in the content
-- table have been modified.
--
UPDATE t2 SET b = 'xxx', c = 'yyy' WHERE rowid = 3;
SELECT * FROM t3 WHERE t3 MATCH 'k';
-- Following the DELETE below, the query returns one row containing two
-- NULL values. NULL values are returned because FTS is unable to find
-- a row with rowid=3 within the content table.
--
DELETE FROM t2;
SELECT * FROM t3 WHERE t3 MATCH 'k';
When a row is deleted from an external content FTS4 table, FTS4 needs to
retrieve the column values of the row being deleted from the content table.
This is so that FTS4 can update the full-text index entries for each token
that occurs within the deleted row to indicate that that row has been
deleted. If the content table row cannot be found, or if it contains values
inconsistent with the contents of the FTS index, the results can be difficult
to predict. The FTS index may be left containing entries corresponding to the
deleted row, which can lead to seemingly nonsensical results being returned
by subsequent SELECT queries. The same applies when a row is updated, as
internally an UPDATE is the same as a DELETE followed by an INSERT.
Instead of writing separately to the full-text index and the content table,
some users may wish to use database triggers to keep the full-text index
up to date with respect to the set of documents stored in the content table.
For example, using the tables from earlier examples:
CREATE TRIGGER t2_bu BEFORE UPDATE ON t2 BEGIN
DELETE FROM t3 WHERE docid=old.rowid;
END;
CREATE TRIGGER t2_bd BEFORE DELETE ON t2 BEGIN
DELETE FROM t3 WHERE docid=old.rowid;
END;
CREATE TRIGGER t2_bu AFTER UPDATE ON t2 BEGIN
INSERT INTO t3(docid, b, c) VALUES(new.rowid, new.b, new.c);
END;
CREATE TRIGGER t2_bd AFTER INSERT ON t2 BEGIN
INSERT INTO t3(docid, b, c) VALUES(new.rowid, new.b, new.c);
END;
The DELETE trigger must be fired before the actual delete takes place
on the content table. This is so that FTS4 can still retrieve the original
values in order to update the full-text index. And the INSERT trigger must
be fired after the new row is inserted, so as to handle the case where the
rowid is assigned automatically within the system. The UPDATE trigger must
be split into two parts, one fired before and one after the update of the
content table, for the same reasons.
FTS4 features a special command similar to the 'optimize' command that
deletes the entire full-text index and rebuilds it based on the current
set of documents in the content table. Assuming again that "t3" is the
name of the external content FTS4 table, the command is:
INSERT INTO t3(t3) VALUES('rebuild');
This command may also be used with ordinary FTS4 tables, although it may
only be useful if the full-text index has somehow become corrupt. It is an
error to attempt to rebuild the full-text index maintained by a contentless
FTS4 table.
ext/fts3/README.syntax
+209
View File
@@ -0,0 +1,209 @@
1. OVERVIEW
This README file describes the syntax of the arguments that may be passed to
the FTS3 MATCH operator used for full-text queries. For example, if table
"t1" is an Fts3 virtual table, the following SQL query:
SELECT * FROM t1 WHERE <col> MATCH <full-text query>
may be used to retrieve all rows that match a specified for full-text query.
The text "<col>" should be replaced by either the name of the fts3 table
(in this case "t1"), or by the name of one of the columns of the fts3
table. <full-text-query> should be replaced by an SQL expression that
computes to a string containing an Fts3 query.
If the left-hand-side of the MATCH operator is set to the name of the
fts3 table, then by default the query may be matched against any column
of the table. If it is set to a column name, then by default the query
may only match the specified column. In both cases this may be overriden
as part of the query text (see sections 2 and 3 below).
As of SQLite version 3.6.8, Fts3 supports two slightly different query
formats; the standard syntax, which is used by default, and the enhanced
query syntax which can be selected by compiling with the pre-processor
symbol SQLITE_ENABLE_FTS3_PARENTHESIS defined.
-DSQLITE_ENABLE_FTS3_PARENTHESIS
2. STANDARD QUERY SYNTAX
When using the standard Fts3 query syntax, a query usually consists of a
list of terms (words) separated by white-space characters. To match a
query, a row (or column) of an Fts3 table must contain each of the specified
terms. For example, the following query:
<col> MATCH 'hello world'
matches rows (or columns, if <col> is the name of a column name) that
contain at least one instance of the token "hello", and at least one
instance of the token "world". Tokens may be grouped into phrases using
quotation marks. In this case, a matching row or column must contain each
of the tokens in the phrase in the order specified, with no intervening
tokens. For example, the query:
<col> MATCH '"hello world" joe"
matches the first of the following two documents, but not the second or
third:
"'Hello world', said Joe."
"One should always greet the world with a cheery hello, thought Joe."
"How many hello world programs could their be?"
As well as grouping tokens together by phrase, the binary NEAR operator
may be used to search for rows that contain two or more specified tokens
or phrases within a specified proximity of each other. The NEAR operator
must always be specified in upper case. The word "near" in lower or mixed
case is treated as an ordinary token. For example, the following query:
<col> MATCH 'engineering NEAR consultancy'
matches rows that contain both the "engineering" and "consultancy" tokens
in the same column with not more than 10 other words between them. It does
not matter which of the two terms occurs first in the document, only that
they be separated by only 10 tokens or less. The user may also specify
a different required proximity by adding "/N" immediately after the NEAR
operator, where N is an integer. For example:
<col> MATCH 'engineering NEAR/5 consultancy'
searches for a row containing an instance of each specified token separated
by not more than 5 other tokens. More than one NEAR operator can be used
in as sequence. For example this query:
<col> MATCH 'reliable NEAR/2 engineering NEAR/5 consultancy'
searches for a row that contains an instance of the token "reliable"
separated by not more than two tokens from an instance of "engineering",
which is in turn separated by not more than 5 other tokens from an
instance of the term "consultancy". Phrases enclosed in quotes may
also be used as arguments to the NEAR operator.
Similar to the NEAR operator, one or more tokens or phrases may be
separated by OR operators. In this case, only one of the specified tokens
or phrases must appear in the document. For example, the query:
<col> MATCH 'hello OR world'
matches rows that contain either the term "hello", or the term "world",
or both. Note that unlike in many programming languages, the OR operator
has a higher precedence than the AND operators implied between white-space
separated tokens. The following query matches documents that contain the
term 'sqlite' and at least one of the terms 'fantastic' or 'impressive',
not those that contain both 'sqlite' and 'fantastic' or 'impressive':
<col> MATCH 'sqlite fantastic OR impressive'
Any token that is part of an Fts3 query expression, whether or not it is
part of a phrase enclosed in quotes, may have a '*' character appended to
it. In this case, the token matches all terms that begin with the characters
of the token, not just those that exactly match it. For example, the
following query:
<col> MATCH 'sql*'
matches all rows that contain the term "SQLite", as well as those that
contain "SQL".
A token that is not part of a quoted phrase may be preceded by a '-'
character, which indicates that matching rows must not contain the
specified term. For example, the following:
<col> MATCH '"database engine" -sqlite'
matches rows that contain the phrase "database engine" but do not contain
the term "sqlite". If the '-' character occurs inside a quoted phrase,
it is ignored. It is possible to use both the '-' prefix and the '*' postfix
on a single term. At this time, all Fts3 queries must contain at least
one term or phrase that is not preceded by the '-' prefix.
Regardless of whether or not a table name or column name is used on the
left hand side of the MATCH operator, a specific column of the fts3 table
may be associated with each token in a query by preceding a token with
a column name followed by a ':' character. For example, regardless of what
is specified for <col>, the following query requires that column "col1"
of the table contains the term "hello", and that column "col2" of the
table contains the term "world". If the table does not contain columns
named "col1" and "col2", then an error is returned and the query is
not run.
<col> MATCH 'col1:hello col2:world'
It is not possible to associate a specific table column with a quoted
phrase or a term preceded by a '-' operator. A '*' character may be
appended to a term associated with a specific column for prefix matching.
3. ENHANCED QUERY SYNTAX
The enhanced query syntax is quite similar to the standard query syntax,
with the following four differences:
1) Parenthesis are supported. When using the enhanced query syntax,
parenthesis may be used to overcome the built-in precedence of the
supplied binary operators. For example, the following query:
<col> MATCH '(hello world) OR (simple example)'
matches documents that contain both "hello" and "world", and documents
that contain both "simple" and "example". It is not possible to formulate
such a query using the standard syntax.
2) Instead of separating tokens and phrases by whitespace, an AND operator
may be explicitly specified. This does not change query processing at
all, but may be used to improve readability. For example, the following
query is handled identically to the one above:
<col> MATCH '(hello AND world) OR (simple AND example)'
As with the OR and NEAR operators, the AND operator must be specified
in upper case. The word "and" specified in lower or mixed case is
handled as a regular token.
3) The '-' token prefix is not supported. Instead, a new binary operator,
NOT, is included. The NOT operator requires that the query specified
as its left-hand operator matches, but that the query specified as the
right-hand operator does not. For example, to query for all rows that
contain the term "example" but not the term "simple", the following
query could be used:
<col> MATCH 'example NOT simple'
As for all other operators, the NOT operator must be specified in
upper case. Otherwise it will be treated as a regular token.
4) Unlike in the standard syntax, where the OR operator has a higher
precedence than the implicit AND operator, when using the enhanced
syntax implicit and explicit AND operators have a higher precedence
than OR operators. Using the enhanced syntax, the following two
queries are equivalent:
<col> MATCH 'sqlite fantastic OR impressive'
<col> MATCH '(sqlite AND fantastic) OR impressive'
however, when using the standard syntax, the query:
<col> MATCH 'sqlite fantastic OR impressive'
is equivalent to the enhanced syntax query:
<col> MATCH 'sqlite AND (fantastic OR impressive)'
The precedence of all enhanced syntax operators, in order from highest
to lowest, is:
NEAR (highest precedence, tightest grouping)
NOT
AND
OR (lowest precedence, loosest grouping)
Using the advanced syntax, it is possible to specify expressions enclosed
in parenthesis as operands to the NOT, AND and OR operators. However both
the left and right hand side operands of NEAR operators must be either
tokens or phrases. Attempting the following query will return an error:
<col> MATCH 'sqlite NEAR (fantastic OR impressive)'
Queries of this form must be re-written as:
<col> MATCH 'sqlite NEAR fantastic OR sqlite NEAR impressive'
ext/fts3/README.tokenizers
+135
View File
@@ -0,0 +1,135 @@
1. FTS3 Tokenizers
When creating a new full-text table, FTS3 allows the user to select
the text tokenizer implementation to be used when indexing text
by specifying a "tokenize" clause as part of the CREATE VIRTUAL TABLE
statement:
CREATE VIRTUAL TABLE <table-name> USING fts3(
<columns ...> [, tokenize <tokenizer-name> [<tokenizer-args>]]
);
The built-in tokenizers (valid values to pass as <tokenizer name>) are
"simple", "porter" and "unicode".
<tokenizer-args> should consist of zero or more white-space separated
arguments to pass to the selected tokenizer implementation. The
interpretation of the arguments, if any, depends on the individual
tokenizer.
2. Custom Tokenizers
FTS3 allows users to provide custom tokenizer implementations. The
interface used to create a new tokenizer is defined and described in
the fts3_tokenizer.h source file.
Registering a new FTS3 tokenizer is similar to registering a new
virtual table module with SQLite. The user passes a pointer to a
structure containing pointers to various callback functions that
make up the implementation of the new tokenizer type. For tokenizers,
the structure (defined in fts3_tokenizer.h) is called
"sqlite3_tokenizer_module".
FTS3 does not expose a C-function that users call to register new
tokenizer types with a database handle. Instead, the pointer must
be encoded as an SQL blob value and passed to FTS3 through the SQL
engine by evaluating a special scalar function, "fts3_tokenizer()".
The fts3_tokenizer() function may be called with one or two arguments,
as follows:
SELECT fts3_tokenizer(<tokenizer-name>);
SELECT fts3_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module ptr>);
Where <tokenizer-name> is a string identifying the tokenizer and
<sqlite3_tokenizer_module ptr> is a pointer to an sqlite3_tokenizer_module
structure encoded as an SQL blob. If the second argument is present,
it is registered as tokenizer <tokenizer-name> and a copy of it
returned. If only one argument is passed, a pointer to the tokenizer
implementation currently registered as <tokenizer-name> is returned,
encoded as a blob. Or, if no such tokenizer exists, an SQL exception
(error) is raised.
SECURITY: If the fts3 extension is used in an environment where potentially
malicious users may execute arbitrary SQL (i.e. gears), they should be
prevented from invoking the fts3_tokenizer() function. The
fts3_tokenizer() function is disabled by default. It is only enabled
by SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER. Do not enable it in
security sensitive environments.
See "Sample code" below for an example of calling the fts3_tokenizer()
function from C code.
3. ICU Library Tokenizers
If this extension is compiled with the SQLITE_ENABLE_ICU pre-processor
symbol defined, then there exists a built-in tokenizer named "icu"
implemented using the ICU library. The first argument passed to the
xCreate() method (see fts3_tokenizer.h) of this tokenizer may be
an ICU locale identifier. For example "tr_TR" for Turkish as used
in Turkey, or "en_AU" for English as used in Australia. For example:
"CREATE VIRTUAL TABLE thai_text USING fts3(text, tokenizer icu th_TH)"
The ICU tokenizer implementation is very simple. It splits the input
text according to the ICU rules for finding word boundaries and discards
any tokens that consist entirely of white-space. This may be suitable
for some applications in some locales, but not all. If more complex
processing is required, for example to implement stemming or
discard punctuation, this can be done by creating a tokenizer
implementation that uses the ICU tokenizer as part of its implementation.
When using the ICU tokenizer this way, it is safe to overwrite the
contents of the strings returned by the xNext() method (see
fts3_tokenizer.h).
4. Sample code.
The following two code samples illustrate the way C code should invoke
the fts3_tokenizer() scalar function:
int registerTokenizer(
sqlite3 *db,
char *zName,
const sqlite3_tokenizer_module *p
){
int rc;
sqlite3_stmt *pStmt;
const char zSql[] = "SELECT fts3_tokenizer(?, ?)";
rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
if( rc!=SQLITE_OK ){
return rc;
}
sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
sqlite3_bind_blob(pStmt, 2, &p, sizeof(p), SQLITE_STATIC);
sqlite3_step(pStmt);
return sqlite3_finalize(pStmt);
}
int queryTokenizer(
sqlite3 *db,
char *zName,
const sqlite3_tokenizer_module **pp
){
int rc;
sqlite3_stmt *pStmt;
const char zSql[] = "SELECT fts3_tokenizer(?)";
*pp = 0;
rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
if( rc!=SQLITE_OK ){
return rc;
}
sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
if( SQLITE_ROW==sqlite3_step(pStmt) ){
if( sqlite3_column_type(pStmt, 0)==SQLITE_BLOB ){
memcpy(pp, sqlite3_column_blob(pStmt, 0), sizeof(*pp));
}
}
return sqlite3_finalize(pStmt);
}
ext/fts3/README.txt
+4
View File
@@ -0,0 +1,4 @@
This folder contains source code to the second full-text search
extension for SQLite. While the API is the same, this version uses a
substantially different storage schema from fts1, so tables will need
to be rebuilt.
ext/fts3/fts3.c
+6206
View File
File diff suppressed because it is too large. Load diff
ext/fts3/fts3.h
+26
View File
@@ -0,0 +1,26 @@
/*
** 2006 Oct 10
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
** This header file is used by programs that want to link against the
** FTS3 library. All it does is declare the sqlite3Fts3Init() interface.
*/
#include "sqlite3.h"
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
int sqlite3Fts3Init(sqlite3 *db);
#ifdef __cplusplus
} /* extern "C" */
#endif /* __cplusplus */
ext/fts3/fts3Int.h
+684
View File
@@ -0,0 +1,684 @@
/*
** 2009 Nov 12
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
*/
#ifndef _FTSINT_H
#define _FTSINT_H
#include <assert.h>
#include <stdlib.h>
#include <stddef.h>
#include <stdio.h>
#include <string.h>
#include <stdarg.h>
#if !defined(NDEBUG) && !defined(SQLITE_DEBUG)
# define NDEBUG 1
#endif
/* FTS3/FTS4 require virtual tables */
#ifdef SQLITE_OMIT_VIRTUALTABLE
# undef SQLITE_ENABLE_FTS3
# undef SQLITE_ENABLE_FTS4
#endif
/*
** FTS4 is really an extension for FTS3. It is enabled using the
** SQLITE_ENABLE_FTS3 macro. But to avoid confusion we also all
** the SQLITE_ENABLE_FTS4 macro to serve as an alisse for SQLITE_ENABLE_FTS3.
*/
#if defined(SQLITE_ENABLE_FTS4) && !defined(SQLITE_ENABLE_FTS3)
# define SQLITE_ENABLE_FTS3
#endif
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
/* If not building as part of the core, include sqlite3ext.h. */
#ifndef SQLITE_CORE
# include "sqlite3ext.h"
SQLITE_EXTENSION_INIT3
#endif
#include "sqlite3.h"
#include "fts3_tokenizer.h"
#include "fts3_hash.h"
/*
** This constant determines the maximum depth of an FTS expression tree
** that the library will create and use. FTS uses recursion to perform
** various operations on the query tree, so the disadvantage of a large
** limit is that it may allow very large queries to use large amounts
** of stack space (perhaps causing a stack overflow).
*/
#ifndef SQLITE_FTS3_MAX_EXPR_DEPTH
# define SQLITE_FTS3_MAX_EXPR_DEPTH 12
#endif
/*
** This constant controls how often segments are merged. Once there are
** FTS3_MERGE_COUNT segments of level N, they are merged into a single
** segment of level N+1.
*/
#define FTS3_MERGE_COUNT 16
/*
** This is the maximum amount of data (in bytes) to store in the
** Fts3Table.pendingTerms hash table. Normally, the hash table is
** populated as documents are inserted/updated/deleted in a transaction
** and used to create a new segment when the transaction is committed.
** However if this limit is reached midway through a transaction, a new
** segment is created and the hash table cleared immediately.
*/
#define FTS3_MAX_PENDING_DATA (1*1024*1024)
/*
** Macro to return the number of elements in an array. SQLite has a
** similar macro called ArraySize(). Use a different name to avoid
** a collision when building an amalgamation with built-in FTS3.
*/
#define SizeofArray(X) ((int)(sizeof(X)/sizeof(X[0])))
#ifndef MIN
# define MIN(x,y) ((x)<(y)?(x):(y))
#endif
#ifndef MAX
# define MAX(x,y) ((x)>(y)?(x):(y))
#endif
/*
** Maximum length of a varint encoded integer. The varint format is different
** from that used by SQLite, so the maximum length is 10, not 9.
*/
#define FTS3_VARINT_MAX 10
#define FTS3_BUFFER_PADDING 8
/*
** FTS4 virtual tables may maintain multiple indexes - one index of all terms
** in the document set and zero or more prefix indexes. All indexes are stored
** as one or more b+-trees in the %_segments and %_segdir tables.
**
** It is possible to determine which index a b+-tree belongs to based on the
** value stored in the "%_segdir.level" column. Given this value L, the index
** that the b+-tree belongs to is (L<<10). In other words, all b+-trees with
** level values between 0 and 1023 (inclusive) belong to index 0, all levels
** between 1024 and 2047 to index 1, and so on.
**
** It is considered impossible for an index to use more than 1024 levels. In
** theory though this may happen, but only after at least
** (FTS3_MERGE_COUNT^1024) separate flushes of the pending-terms tables.
*/
#define FTS3_SEGDIR_MAXLEVEL 1024
#define FTS3_SEGDIR_MAXLEVEL_STR "1024"
/*
** The testcase() macro is only used by the amalgamation. If undefined,
** make it a no-op.
*/
#ifndef testcase
# define testcase(X)
#endif
/*
** Terminator values for position-lists and column-lists.
*/
#define POS_COLUMN (1) /* Column-list terminator */
#define POS_END (0) /* Position-list terminator */
/*
** The assert_fts3_nc() macro is similar to the assert() macro, except that it
** is used for assert() conditions that are true only if it can be
** guranteed that the database is not corrupt.
*/
#ifdef SQLITE_DEBUG
extern int sqlite3_fts3_may_be_corrupt;
# define assert_fts3_nc(x) assert(sqlite3_fts3_may_be_corrupt || (x))
#else
# define assert_fts3_nc(x) assert(x)
#endif
/*
** This section provides definitions to allow the
** FTS3 extension to be compiled outside of the
** amalgamation.
*/
#ifndef SQLITE_AMALGAMATION
/*
** Macros indicating that conditional expressions are always true or
** false.
*/
#if defined(SQLITE_COVERAGE_TEST) || defined(SQLITE_MUTATION_TEST)
# define SQLITE_OMIT_AUXILIARY_SAFETY_CHECKS 1
#endif
#if defined(SQLITE_OMIT_AUXILIARY_SAFETY_CHECKS)
# define ALWAYS(X) (1)
# define NEVER(X) (0)
#elif !defined(NDEBUG)
# define ALWAYS(X) ((X)?1:(assert(0),0))
# define NEVER(X) ((X)?(assert(0),1):0)
#else
# define ALWAYS(X) (X)
# define NEVER(X) (X)
#endif
/*
** Internal types used by SQLite.
*/
typedef unsigned char u8; /* 1-byte (or larger) unsigned integer */
typedef short int i16; /* 2-byte (or larger) signed integer */
typedef unsigned int u32; /* 4-byte unsigned integer */
typedef sqlite3_uint64 u64; /* 8-byte unsigned integer */
typedef sqlite3_int64 i64; /* 8-byte signed integer */
/*
** Macro used to suppress compiler warnings for unused parameters.
*/
#define UNUSED_PARAMETER(x) (void)(x)
/*
** Activate assert() only if SQLITE_TEST is enabled.
*/
#if !defined(NDEBUG) && !defined(SQLITE_DEBUG)
# define NDEBUG 1
#endif
/*
** The TESTONLY macro is used to enclose variable declarations or
** other bits of code that are needed to support the arguments
** within testcase() and assert() macros.
*/
#if defined(SQLITE_DEBUG) || defined(SQLITE_COVERAGE_TEST)
# define TESTONLY(X) X
#else
# define TESTONLY(X)
#endif
#define LARGEST_INT64 (0xffffffff|(((i64)0x7fffffff)<<32))
#define SMALLEST_INT64 (((i64)-1) - LARGEST_INT64)
#define deliberate_fall_through
/*
** Macros needed to provide flexible arrays in a portable way
*/
#ifndef offsetof
# define offsetof(STRUCTURE,FIELD) ((size_t)((char*)&((STRUCTURE*)0)->FIELD))
#endif
#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)
# define FLEXARRAY
#else
# define FLEXARRAY 1
#endif
#endif /* SQLITE_AMALGAMATION */
#ifdef SQLITE_DEBUG
int sqlite3Fts3Corrupt(void);
# define FTS_CORRUPT_VTAB sqlite3Fts3Corrupt()
#else
# define FTS_CORRUPT_VTAB SQLITE_CORRUPT_VTAB
#endif
typedef struct Fts3Table Fts3Table;
typedef struct Fts3Cursor Fts3Cursor;
typedef struct Fts3Expr Fts3Expr;
typedef struct Fts3Phrase Fts3Phrase;
typedef struct Fts3PhraseToken Fts3PhraseToken;
typedef struct Fts3Doclist Fts3Doclist;
typedef struct Fts3SegFilter Fts3SegFilter;
typedef struct Fts3DeferredToken Fts3DeferredToken;
typedef struct Fts3SegReader Fts3SegReader;
typedef struct Fts3MultiSegReader Fts3MultiSegReader;
typedef struct MatchinfoBuffer MatchinfoBuffer;
/*
** A connection to a fulltext index is an instance of the following
** structure. The xCreate and xConnect methods create an instance
** of this structure and xDestroy and xDisconnect free that instance.
** All other methods receive a pointer to the structure as one of their
** arguments.
*/
struct Fts3Table {
sqlite3_vtab base; /* Base class used by SQLite core */
sqlite3 *db; /* The database connection */
const char *zDb; /* logical database name */
const char *zName; /* virtual table name */
int nColumn; /* number of named columns in virtual table */
char **azColumn; /* column names. malloced */
u8 *abNotindexed; /* True for 'notindexed' columns */
sqlite3_tokenizer *pTokenizer; /* tokenizer for inserts and queries */
char *zContentTbl; /* content=xxx option, or NULL */
char *zLanguageid; /* languageid=xxx option, or NULL */
int nAutoincrmerge; /* Value configured by 'automerge' */
u32 nLeafAdd; /* Number of leaf blocks added this trans */
int bLock; /* Used to prevent recursive content= tbls */
/* Precompiled statements used by the implementation. Each of these
** statements is run and reset within a single virtual table API call.
*/
sqlite3_stmt *aStmt[40];
sqlite3_stmt *pSeekStmt; /* Cache for fts3CursorSeekStmt() */
char *zReadExprlist;
char *zWriteExprlist;
int nNodeSize; /* Soft limit for node size */
u8 bFts4; /* True for FTS4, false for FTS3 */
u8 bHasStat; /* True if %_stat table exists (2==unknown) */
u8 bHasDocsize; /* True if %_docsize table exists */
u8 bDescIdx; /* True if doclists are in reverse order */
u8 bIgnoreSavepoint; /* True to ignore xSavepoint invocations */
int nPgsz; /* Page size for host database */
char *zSegmentsTbl; /* Name of %_segments table */
sqlite3_blob *pSegments; /* Blob handle open on %_segments table */
int iSavepoint;
/*
** The following array of hash tables is used to buffer pending index
** updates during transactions. All pending updates buffered at any one
** time must share a common language-id (see the FTS4 langid= feature).
** The current language id is stored in variable iPrevLangid.
**
** A single FTS4 table may have multiple full-text indexes. For each index
** there is an entry in the aIndex[] array. Index 0 is an index of all the
** terms that appear in the document set. Each subsequent index in aIndex[]
** is an index of prefixes of a specific length.
**
** Variable nPendingData contains an estimate the memory consumed by the
** pending data structures, including hash table overhead, but not including
** malloc overhead. When nPendingData exceeds nMaxPendingData, all hash
** tables are flushed to disk. Variable iPrevDocid is the docid of the most
** recently inserted record.
*/
int nIndex; /* Size of aIndex[] */
struct Fts3Index {
int nPrefix; /* Prefix length (0 for main terms index) */
Fts3Hash hPending; /* Pending terms table for this index */
} *aIndex;
int nMaxPendingData; /* Max pending data before flush to disk */
int nPendingData; /* Current bytes of pending data */
sqlite_int64 iPrevDocid; /* Docid of most recently inserted document */
int iPrevLangid; /* Langid of recently inserted document */
int bPrevDelete; /* True if last operation was a delete */
#if defined(SQLITE_DEBUG) || defined(SQLITE_COVERAGE_TEST)
/* State variables used for validating that the transaction control
** methods of the virtual table are called at appropriate times. These
** values do not contribute to FTS functionality; they are used for
** verifying the operation of the SQLite core.
*/
int inTransaction; /* True after xBegin but before xCommit/xRollback */
int mxSavepoint; /* Largest valid xSavepoint integer */
#endif
#if defined(SQLITE_DEBUG) || defined(SQLITE_TEST)
/* True to disable the incremental doclist optimization. This is controlled
** by special insert command 'test-no-incr-doclist'. */
int bNoIncrDoclist;
/* Number of segments in a level */
int nMergeCount;
#endif
};
/* Macro to find the number of segments to merge */
#if defined(SQLITE_DEBUG) || defined(SQLITE_TEST)
# define MergeCount(P) ((P)->nMergeCount)
#else
# define MergeCount(P) FTS3_MERGE_COUNT
#endif
/*
** When the core wants to read from the virtual table, it creates a
** virtual table cursor (an instance of the following structure) using
** the xOpen method. Cursors are destroyed using the xClose method.
*/
struct Fts3Cursor {
sqlite3_vtab_cursor base; /* Base class used by SQLite core */
i16 eSearch; /* Search strategy (see below) */
u8 isEof; /* True if at End Of Results */
u8 isRequireSeek; /* True if must seek pStmt to %_content row */
u8 bSeekStmt; /* True if pStmt is a seek */
sqlite3_stmt *pStmt; /* Prepared statement in use by the cursor */
Fts3Expr *pExpr; /* Parsed MATCH query string */
int iLangid; /* Language being queried for */
int nPhrase; /* Number of matchable phrases in query */
Fts3DeferredToken *pDeferred; /* Deferred search tokens, if any */
sqlite3_int64 iPrevId; /* Previous id read from aDoclist */
char *pNextId; /* Pointer into the body of aDoclist */
char *aDoclist; /* List of docids for full-text queries */
int nDoclist; /* Size of buffer at aDoclist */
u8 bDesc; /* True to sort in descending order */
int eEvalmode; /* An FTS3_EVAL_XX constant */
int nRowAvg; /* Average size of database rows, in pages */
sqlite3_int64 nDoc; /* Documents in table */
i64 iMinDocid; /* Minimum docid to return */
i64 iMaxDocid; /* Maximum docid to return */
int isMatchinfoNeeded; /* True when aMatchinfo[] needs filling in */
MatchinfoBuffer *pMIBuffer; /* Buffer for matchinfo data */
};
#define FTS3_EVAL_FILTER 0
#define FTS3_EVAL_NEXT 1
#define FTS3_EVAL_MATCHINFO 2
/*
** The Fts3Cursor.eSearch member is always set to one of the following.
** Actually, Fts3Cursor.eSearch can be greater than or equal to
** FTS3_FULLTEXT_SEARCH. If so, then Fts3Cursor.eSearch - 2 is the index
** of the column to be searched. For example, in
**
** CREATE VIRTUAL TABLE ex1 USING fts3(a,b,c,d);
** SELECT docid FROM ex1 WHERE b MATCH 'one two three';
**
** Because the LHS of the MATCH operator is 2nd column "b",
** Fts3Cursor.eSearch will be set to FTS3_FULLTEXT_SEARCH+1. (+0 for a,
** +1 for b, +2 for c, +3 for d.) If the LHS of MATCH were "ex1"
** indicating that all columns should be searched,
** then eSearch would be set to FTS3_FULLTEXT_SEARCH+4.
*/
#define FTS3_FULLSCAN_SEARCH 0 /* Linear scan of %_content table */
#define FTS3_DOCID_SEARCH 1 /* Lookup by rowid on %_content table */
#define FTS3_FULLTEXT_SEARCH 2 /* Full-text index search */
/*
** The lower 16-bits of the sqlite3_index_info.idxNum value set by
** the xBestIndex() method contains the Fts3Cursor.eSearch value described
** above. The upper 16-bits contain a combination of the following
** bits, used to describe extra constraints on full-text searches.
*/
#define FTS3_HAVE_LANGID 0x00010000 /* languageid=? */
#define FTS3_HAVE_DOCID_GE 0x00020000 /* docid>=? */
#define FTS3_HAVE_DOCID_LE 0x00040000 /* docid<=? */
struct Fts3Doclist {
char *aAll; /* Array containing doclist (or NULL) */
int nAll; /* Size of a[] in bytes */
char *pNextDocid; /* Pointer to next docid */
sqlite3_int64 iDocid; /* Current docid (if pList!=0) */
int bFreeList; /* True if pList should be sqlite3_free()d */
char *pList; /* Pointer to position list following iDocid */
int nList; /* Length of position list */
};
/*
** A "phrase" is a sequence of one or more tokens that must match in
** sequence. A single token is the base case and the most common case.
** For a sequence of tokens contained in double-quotes (i.e. "one two three")
** nToken will be the number of tokens in the string.
*/
struct Fts3PhraseToken {
char *z; /* Text of the token */
int n; /* Number of bytes in buffer z */
int isPrefix; /* True if token ends with a "*" character */
int bFirst; /* True if token must appear at position 0 */
/* Variables above this point are populated when the expression is
** parsed (by code in fts3_expr.c). Below this point the variables are
** used when evaluating the expression. */
Fts3DeferredToken *pDeferred; /* Deferred token object for this token */
Fts3MultiSegReader *pSegcsr; /* Segment-reader for this token */
};
struct Fts3Phrase {
/* Cache of doclist for this phrase. */
Fts3Doclist doclist;
int bIncr; /* True if doclist is loaded incrementally */
int iDoclistToken;
/* Used by sqlite3Fts3EvalPhrasePoslist() if this is a descendent of an
** OR condition. */
char *pOrPoslist;
i64 iOrDocid;
/* Variables below this point are populated by fts3_expr.c when parsing
** a MATCH expression. Everything above is part of the evaluation phase.
*/
int nToken; /* Number of tokens in the phrase */
int iColumn; /* Index of column this phrase must match */
Fts3PhraseToken aToken[FLEXARRAY]; /* One for each token in the phrase */
};
/* Size (in bytes) of an Fts3Phrase object large enough to hold N tokens */
#define SZ_FTS3PHRASE(N) \
(offsetof(Fts3Phrase,aToken)+(N)*sizeof(Fts3PhraseToken))
/*
** A tree of these objects forms the RHS of a MATCH operator.
**
** If Fts3Expr.eType is FTSQUERY_PHRASE and isLoaded is true, then aDoclist
** points to a malloced buffer, size nDoclist bytes, containing the results
** of this phrase query in FTS3 doclist format. As usual, the initial
** "Length" field found in doclists stored on disk is omitted from this
** buffer.
**
** Variable aMI is used only for FTSQUERY_NEAR nodes to store the global
** matchinfo data. If it is not NULL, it points to an array of size nCol*3,
** where nCol is the number of columns in the queried FTS table. The array
** is populated as follows:
**
** aMI[iCol*3 + 0] = Undefined
** aMI[iCol*3 + 1] = Number of occurrences
** aMI[iCol*3 + 2] = Number of rows containing at least one instance
**
** The aMI array is allocated using sqlite3_malloc(). It should be freed
** when the expression node is.
*/
struct Fts3Expr {
int eType; /* One of the FTSQUERY_XXX values defined below */
int nNear; /* Valid if eType==FTSQUERY_NEAR */
Fts3Expr *pParent; /* pParent->pLeft==this or pParent->pRight==this */
Fts3Expr *pLeft; /* Left operand */
Fts3Expr *pRight; /* Right operand */
Fts3Phrase *pPhrase; /* Valid if eType==FTSQUERY_PHRASE */
/* The following are used by the fts3_eval.c module. */
sqlite3_int64 iDocid; /* Current docid */
u8 bEof; /* True this expression is at EOF already */
u8 bStart; /* True if iDocid is valid */
u8 bDeferred; /* True if this expression is entirely deferred */
/* The following are used by the fts3_snippet.c module. */
int iPhrase; /* Index of this phrase in matchinfo() results */
u32 *aMI; /* See above */
};
/*
** Candidate values for Fts3Query.eType. Note that the order of the first
** four values is in order of precedence when parsing expressions. For
** example, the following:
**
** "a OR b AND c NOT d NEAR e"
**
** is equivalent to:
**
** "a OR (b AND (c NOT (d NEAR e)))"
*/
#define FTSQUERY_NEAR 1
#define FTSQUERY_NOT 2
#define FTSQUERY_AND 3
#define FTSQUERY_OR 4
#define FTSQUERY_PHRASE 5
/* fts3_write.c */
int sqlite3Fts3UpdateMethod(sqlite3_vtab*,int,sqlite3_value**,sqlite3_int64*);
int sqlite3Fts3PendingTermsFlush(Fts3Table *);
void sqlite3Fts3PendingTermsClear(Fts3Table *);
int sqlite3Fts3Optimize(Fts3Table *);
int sqlite3Fts3SegReaderNew(int, int, sqlite3_int64,
sqlite3_int64, sqlite3_int64, const char *, int, Fts3SegReader**);
int sqlite3Fts3SegReaderPending(
Fts3Table*,int,const char*,int,int,Fts3SegReader**);
void sqlite3Fts3SegReaderFree(Fts3SegReader *);
int sqlite3Fts3AllSegdirs(Fts3Table*, int, int, int, sqlite3_stmt **);
int sqlite3Fts3ReadBlock(Fts3Table*, sqlite3_int64, char **, int*, int*);
int sqlite3Fts3SelectDoctotal(Fts3Table *, sqlite3_stmt **);
int sqlite3Fts3SelectDocsize(Fts3Table *, sqlite3_int64, sqlite3_stmt **);
#ifndef SQLITE_DISABLE_FTS4_DEFERRED
void sqlite3Fts3FreeDeferredTokens(Fts3Cursor *);
int sqlite3Fts3DeferToken(Fts3Cursor *, Fts3PhraseToken *, int);
int sqlite3Fts3CacheDeferredDoclists(Fts3Cursor *);
void sqlite3Fts3FreeDeferredDoclists(Fts3Cursor *);
int sqlite3Fts3DeferredTokenList(Fts3DeferredToken *, char **, int *);
#else
# define sqlite3Fts3FreeDeferredTokens(x)
# define sqlite3Fts3DeferToken(x,y,z) SQLITE_OK
# define sqlite3Fts3CacheDeferredDoclists(x) SQLITE_OK
# define sqlite3Fts3FreeDeferredDoclists(x)
# define sqlite3Fts3DeferredTokenList(x,y,z) SQLITE_OK
#endif
void sqlite3Fts3SegmentsClose(Fts3Table *);
int sqlite3Fts3MaxLevel(Fts3Table *, int *);
/* Special values interpreted by sqlite3SegReaderCursor() */
#define FTS3_SEGCURSOR_PENDING -1
#define FTS3_SEGCURSOR_ALL -2
int sqlite3Fts3SegReaderStart(Fts3Table*, Fts3MultiSegReader*, Fts3SegFilter*);
int sqlite3Fts3SegReaderStep(Fts3Table *, Fts3MultiSegReader *);
void sqlite3Fts3SegReaderFinish(Fts3MultiSegReader *);
int sqlite3Fts3SegReaderCursor(Fts3Table *,
int, int, int, const char *, int, int, int, Fts3MultiSegReader *);
/* Flags allowed as part of the 4th argument to SegmentReaderIterate() */
#define FTS3_SEGMENT_REQUIRE_POS 0x00000001
#define FTS3_SEGMENT_IGNORE_EMPTY 0x00000002
#define FTS3_SEGMENT_COLUMN_FILTER 0x00000004
#define FTS3_SEGMENT_PREFIX 0x00000008
#define FTS3_SEGMENT_SCAN 0x00000010
#define FTS3_SEGMENT_FIRST 0x00000020
/* Type passed as 4th argument to SegmentReaderIterate() */
struct Fts3SegFilter {
const char *zTerm;
int nTerm;
int iCol;
int flags;
};
struct Fts3MultiSegReader {
/* Used internally by sqlite3Fts3SegReaderXXX() calls */
Fts3SegReader **apSegment; /* Array of Fts3SegReader objects */
int nSegment; /* Size of apSegment array */
int nAdvance; /* How many seg-readers to advance */
Fts3SegFilter *pFilter; /* Pointer to filter object */
char *aBuffer; /* Buffer to merge doclists in */
i64 nBuffer; /* Allocated size of aBuffer[] in bytes */
int iColFilter; /* If >=0, filter for this column */
int bRestart;
/* Used by fts3.c only. */
int nCost; /* Cost of running iterator */
int bLookup; /* True if a lookup of a single entry. */
/* Output values. Valid only after Fts3SegReaderStep() returns SQLITE_ROW. */
char *zTerm; /* Pointer to term buffer */
int nTerm; /* Size of zTerm in bytes */
char *aDoclist; /* Pointer to doclist buffer */
int nDoclist; /* Size of aDoclist[] in bytes */
};
int sqlite3Fts3Incrmerge(Fts3Table*,int,int);
#define fts3GetVarint32(p, piVal) ( \
(*(u8*)(p)&0x80) ? sqlite3Fts3GetVarint32(p, piVal) : (*piVal=*(u8*)(p), 1) \
)
/* fts3.c */
void sqlite3Fts3ErrMsg(char**,const char*,...);
int sqlite3Fts3PutVarint(char *, sqlite3_int64);
int sqlite3Fts3GetVarint(const char *, sqlite_int64 *);
int sqlite3Fts3GetVarintU(const char *, sqlite_uint64 *);
int sqlite3Fts3GetVarintBounded(const char*,const char*,sqlite3_int64*);
int sqlite3Fts3GetVarint32(const char *, int *);
int sqlite3Fts3VarintLen(sqlite3_uint64);
void sqlite3Fts3Dequote(char *);
void sqlite3Fts3DoclistPrev(int,char*,int,char**,sqlite3_int64*,int*,u8*);
int sqlite3Fts3EvalPhraseStats(Fts3Cursor *, Fts3Expr *, u32 *);
int sqlite3Fts3FirstFilter(sqlite3_int64, char *, int, char *);
void sqlite3Fts3CreateStatTable(int*, Fts3Table*);
int sqlite3Fts3EvalTestDeferred(Fts3Cursor *pCsr, int *pRc);
int sqlite3Fts3ReadInt(const char *z, int *pnOut);
/* fts3_tokenizer.c */
const char *sqlite3Fts3NextToken(const char *, int *);
int sqlite3Fts3InitHashTable(sqlite3 *, Fts3Hash *, const char *);
int sqlite3Fts3InitTokenizer(Fts3Hash *pHash, const char *,
sqlite3_tokenizer **, char **
);
int sqlite3Fts3IsIdChar(char);
/* fts3_snippet.c */
void sqlite3Fts3Offsets(sqlite3_context*, Fts3Cursor*);
void sqlite3Fts3Snippet(sqlite3_context *, Fts3Cursor *, const char *,
const char *, const char *, int, int
);
void sqlite3Fts3Matchinfo(sqlite3_context *, Fts3Cursor *, const char *);
void sqlite3Fts3MIBufferFree(MatchinfoBuffer *p);
/* fts3_expr.c */
int sqlite3Fts3ExprParse(sqlite3_tokenizer *, int,
char **, int, int, int, const char *, int, Fts3Expr **, char **
);
void sqlite3Fts3ExprFree(Fts3Expr *);
#ifdef SQLITE_TEST
int sqlite3Fts3ExprInitTestInterface(sqlite3 *db, Fts3Hash*);
int sqlite3Fts3InitTerm(sqlite3 *db);
#endif
void *sqlite3Fts3MallocZero(i64 nByte);
int sqlite3Fts3OpenTokenizer(sqlite3_tokenizer *, int, const char *, int,
sqlite3_tokenizer_cursor **
);
/* fts3_aux.c */
int sqlite3Fts3InitAux(sqlite3 *db);
void sqlite3Fts3EvalPhraseCleanup(Fts3Phrase *);
int sqlite3Fts3MsrIncrStart(
Fts3Table*, Fts3MultiSegReader*, int, const char*, int);
int sqlite3Fts3MsrIncrNext(
Fts3Table *, Fts3MultiSegReader *, sqlite3_int64 *, char **, int *);
int sqlite3Fts3EvalPhrasePoslist(Fts3Cursor *, Fts3Expr *, int iCol, char **);
int sqlite3Fts3MsrOvfl(Fts3Cursor *, Fts3MultiSegReader *, int *);
int sqlite3Fts3MsrIncrRestart(Fts3MultiSegReader *pCsr);
int sqlite3Fts3MsrCancel(Fts3Cursor*, Fts3Expr*);
/* fts3_tokenize_vtab.c */
int sqlite3Fts3InitTok(sqlite3*, Fts3Hash *, void(*xDestroy)(void*));
/* fts3_unicode2.c (functions generated by parsing unicode text files) */
#ifndef SQLITE_DISABLE_FTS3_UNICODE
int sqlite3FtsUnicodeFold(int, int);
int sqlite3FtsUnicodeIsalnum(int);
int sqlite3FtsUnicodeIsdiacritic(int);
#endif
int sqlite3Fts3ExprIterate(Fts3Expr*, int (*x)(Fts3Expr*,int,void*), void*);
int sqlite3Fts3IntegrityCheck(Fts3Table *p, int *pbOk);
#endif /* !SQLITE_CORE || SQLITE_ENABLE_FTS3 */
#endif /* _FTSINT_H */
ext/fts3/fts3_aux.c
+557
View File
@@ -0,0 +1,557 @@
/*
** 2011 Jan 27
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
*/
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#include <string.h>
#include <assert.h>
typedef struct Fts3auxTable Fts3auxTable;
typedef struct Fts3auxCursor Fts3auxCursor;
struct Fts3auxTable {
sqlite3_vtab base; /* Base class used by SQLite core */
Fts3Table *pFts3Tab;
};
struct Fts3auxCursor {
sqlite3_vtab_cursor base; /* Base class used by SQLite core */
Fts3MultiSegReader csr; /* Must be right after "base" */
Fts3SegFilter filter;
char *zStop;
int nStop; /* Byte-length of string zStop */
int iLangid; /* Language id to query */
int isEof; /* True if cursor is at EOF */
sqlite3_int64 iRowid; /* Current rowid */
int iCol; /* Current value of 'col' column */
int nStat; /* Size of aStat[] array */
struct Fts3auxColstats {
sqlite3_int64 nDoc; /* 'documents' values for current csr row */
sqlite3_int64 nOcc; /* 'occurrences' values for current csr row */
} *aStat;
};
/*
** Schema of the terms table.
*/
#define FTS3_AUX_SCHEMA \
"CREATE TABLE x(term, col, documents, occurrences, languageid HIDDEN)"
/*
** This function does all the work for both the xConnect and xCreate methods.
** These tables have no persistent representation of their own, so xConnect
** and xCreate are identical operations.
*/
static int fts3auxConnectMethod(
sqlite3 *db, /* Database connection */
void *pUnused, /* Unused */
int argc, /* Number of elements in argv array */
const char * const *argv, /* xCreate/xConnect argument array */
sqlite3_vtab **ppVtab, /* OUT: New sqlite3_vtab object */
char **pzErr /* OUT: sqlite3_malloc'd error message */
){
char const *zDb; /* Name of database (e.g. "main") */
char const *zFts3; /* Name of fts3 table */
int nDb; /* Result of strlen(zDb) */
int nFts3; /* Result of strlen(zFts3) */
sqlite3_int64 nByte; /* Bytes of space to allocate here */
int rc; /* value returned by declare_vtab() */
Fts3auxTable *p; /* Virtual table object to return */
UNUSED_PARAMETER(pUnused);
/* The user should invoke this in one of two forms:
**
** CREATE VIRTUAL TABLE xxx USING fts4aux(fts4-table);
** CREATE VIRTUAL TABLE xxx USING fts4aux(fts4-table-db, fts4-table);
*/
if( argc!=4 && argc!=5 ) goto bad_args;
zDb = argv[1];
nDb = (int)strlen(zDb);
if( argc==5 ){
if( nDb==4 && 0==sqlite3_strnicmp("temp", zDb, 4) ){
zDb = argv[3];
nDb = (int)strlen(zDb);
zFts3 = argv[4];
}else{
goto bad_args;
}
}else{
zFts3 = argv[3];
}
nFts3 = (int)strlen(zFts3);
rc = sqlite3_declare_vtab(db, FTS3_AUX_SCHEMA);
if( rc!=SQLITE_OK ) return rc;
nByte = sizeof(Fts3auxTable) + sizeof(Fts3Table) + nDb + nFts3 + 2;
p = (Fts3auxTable *)sqlite3_malloc64(nByte);
if( !p ) return SQLITE_NOMEM;
memset(p, 0, nByte);
p->pFts3Tab = (Fts3Table *)&p[1];
p->pFts3Tab->zDb = (char *)&p->pFts3Tab[1];
p->pFts3Tab->zName = &p->pFts3Tab->zDb[nDb+1];
p->pFts3Tab->db = db;
p->pFts3Tab->nIndex = 1;
memcpy((char *)p->pFts3Tab->zDb, zDb, nDb);
memcpy((char *)p->pFts3Tab->zName, zFts3, nFts3);
sqlite3Fts3Dequote((char *)p->pFts3Tab->zName);
*ppVtab = (sqlite3_vtab *)p;
return SQLITE_OK;
bad_args:
sqlite3Fts3ErrMsg(pzErr, "invalid arguments to fts4aux constructor");
return SQLITE_ERROR;
}
/*
** This function does the work for both the xDisconnect and xDestroy methods.
** These tables have no persistent representation of their own, so xDisconnect
** and xDestroy are identical operations.
*/
static int fts3auxDisconnectMethod(sqlite3_vtab *pVtab){
Fts3auxTable *p = (Fts3auxTable *)pVtab;
Fts3Table *pFts3 = p->pFts3Tab;
int i;
/* Free any prepared statements held */
for(i=0; i<SizeofArray(pFts3->aStmt); i++){
sqlite3_finalize(pFts3->aStmt[i]);
}
sqlite3_free(pFts3->zSegmentsTbl);
sqlite3_free(p);
return SQLITE_OK;
}
#define FTS4AUX_EQ_CONSTRAINT 1
#define FTS4AUX_GE_CONSTRAINT 2
#define FTS4AUX_LE_CONSTRAINT 4
/*
** xBestIndex - Analyze a WHERE and ORDER BY clause.
*/
static int fts3auxBestIndexMethod(
sqlite3_vtab *pVTab,
sqlite3_index_info *pInfo
){
int i;
int iEq = -1;
int iGe = -1;
int iLe = -1;
int iLangid = -1;
int iNext = 1; /* Next free argvIndex value */
UNUSED_PARAMETER(pVTab);
/* This vtab delivers always results in "ORDER BY term ASC" order. */
if( pInfo->nOrderBy==1
&& pInfo->aOrderBy[0].iColumn==0
&& pInfo->aOrderBy[0].desc==0
){
pInfo->orderByConsumed = 1;
}
/* Search for equality and range constraints on the "term" column.
** And equality constraints on the hidden "languageid" column. */
for(i=0; i<pInfo->nConstraint; i++){
if( pInfo->aConstraint[i].usable ){
int op = pInfo->aConstraint[i].op;
int iCol = pInfo->aConstraint[i].iColumn;
if( iCol==0 ){
if( op==SQLITE_INDEX_CONSTRAINT_EQ ) iEq = i;
if( op==SQLITE_INDEX_CONSTRAINT_LT ) iLe = i;
if( op==SQLITE_INDEX_CONSTRAINT_LE ) iLe = i;
if( op==SQLITE_INDEX_CONSTRAINT_GT ) iGe = i;
if( op==SQLITE_INDEX_CONSTRAINT_GE ) iGe = i;
}
if( iCol==4 ){
if( op==SQLITE_INDEX_CONSTRAINT_EQ ) iLangid = i;
}
}
}
if( iEq>=0 ){
pInfo->idxNum = FTS4AUX_EQ_CONSTRAINT;
pInfo->aConstraintUsage[iEq].argvIndex = iNext++;
pInfo->estimatedCost = 5;
}else{
pInfo->idxNum = 0;
pInfo->estimatedCost = 20000;
if( iGe>=0 ){
pInfo->idxNum += FTS4AUX_GE_CONSTRAINT;
pInfo->aConstraintUsage[iGe].argvIndex = iNext++;
pInfo->estimatedCost /= 2;
}
if( iLe>=0 ){
pInfo->idxNum += FTS4AUX_LE_CONSTRAINT;
pInfo->aConstraintUsage[iLe].argvIndex = iNext++;
pInfo->estimatedCost /= 2;
}
}
if( iLangid>=0 ){
pInfo->aConstraintUsage[iLangid].argvIndex = iNext++;
pInfo->estimatedCost--;
}
return SQLITE_OK;
}
/*
** xOpen - Open a cursor.
*/
static int fts3auxOpenMethod(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCsr){
Fts3auxCursor *pCsr; /* Pointer to cursor object to return */
UNUSED_PARAMETER(pVTab);
pCsr = (Fts3auxCursor *)sqlite3_malloc(sizeof(Fts3auxCursor));
if( !pCsr ) return SQLITE_NOMEM;
memset(pCsr, 0, sizeof(Fts3auxCursor));
*ppCsr = (sqlite3_vtab_cursor *)pCsr;
return SQLITE_OK;
}
/*
** xClose - Close a cursor.
*/
static int fts3auxCloseMethod(sqlite3_vtab_cursor *pCursor){
Fts3Table *pFts3 = ((Fts3auxTable *)pCursor->pVtab)->pFts3Tab;
Fts3auxCursor *pCsr = (Fts3auxCursor *)pCursor;
sqlite3Fts3SegmentsClose(pFts3);
sqlite3Fts3SegReaderFinish(&pCsr->csr);
sqlite3_free((void *)pCsr->filter.zTerm);
sqlite3_free(pCsr->zStop);
sqlite3_free(pCsr->aStat);
sqlite3_free(pCsr);
return SQLITE_OK;
}
static int fts3auxGrowStatArray(Fts3auxCursor *pCsr, int nSize){
if( nSize>pCsr->nStat ){
struct Fts3auxColstats *aNew;
aNew = (struct Fts3auxColstats *)sqlite3_realloc64(pCsr->aStat,
sizeof(struct Fts3auxColstats) * nSize
);
if( aNew==0 ) return SQLITE_NOMEM;
memset(&aNew[pCsr->nStat], 0,
sizeof(struct Fts3auxColstats) * (nSize - pCsr->nStat)
);
pCsr->aStat = aNew;
pCsr->nStat = nSize;
}
return SQLITE_OK;
}
/*
** xNext - Advance the cursor to the next row, if any.
*/
static int fts3auxNextMethod(sqlite3_vtab_cursor *pCursor){
Fts3auxCursor *pCsr = (Fts3auxCursor *)pCursor;
Fts3Table *pFts3 = ((Fts3auxTable *)pCursor->pVtab)->pFts3Tab;
int rc;
/* Increment our pretend rowid value. */
pCsr->iRowid++;
for(pCsr->iCol++; pCsr->iCol<pCsr->nStat; pCsr->iCol++){
if( pCsr->aStat[pCsr->iCol].nDoc>0 ) return SQLITE_OK;
}
rc = sqlite3Fts3SegReaderStep(pFts3, &pCsr->csr);
if( rc==SQLITE_ROW ){
int i = 0;
int nDoclist = pCsr->csr.nDoclist;
char *aDoclist = pCsr->csr.aDoclist;
int iCol;
int eState = 0;
if( pCsr->zStop ){
int n = (pCsr->nStop<pCsr->csr.nTerm) ? pCsr->nStop : pCsr->csr.nTerm;
int mc = memcmp(pCsr->zStop, pCsr->csr.zTerm, n);
if( mc<0 || (mc==0 && pCsr->csr.nTerm>pCsr->nStop) ){
pCsr->isEof = 1;
return SQLITE_OK;
}
}
if( fts3auxGrowStatArray(pCsr, 2) ) return SQLITE_NOMEM;
memset(pCsr->aStat, 0, sizeof(struct Fts3auxColstats) * pCsr->nStat);
iCol = 0;
rc = SQLITE_OK;
while( i<nDoclist ){
sqlite3_int64 v = 0;
i += sqlite3Fts3GetVarint(&aDoclist[i], &v);
switch( eState ){
/* State 0. In this state the integer just read was a docid. */
case 0:
pCsr->aStat[0].nDoc++;
eState = 1;
iCol = 0;
break;
/* State 1. In this state we are expecting either a 1, indicating
** that the following integer will be a column number, or the
** start of a position list for column 0.
**
** The only difference between state 1 and state 2 is that if the
** integer encountered in state 1 is not 0 or 1, then we need to
** increment the column 0 "nDoc" count for this term.
*/
case 1:
assert( iCol==0 );
if( v>1 ){
pCsr->aStat[1].nDoc++;
}
eState = 2;
/* fall through */
case 2:
if( v==0 ){ /* 0x00. Next integer will be a docid. */
eState = 0;
}else if( v==1 ){ /* 0x01. Next integer will be a column number. */
eState = 3;
}else{ /* 2 or greater. A position. */
pCsr->aStat[iCol+1].nOcc++;
pCsr->aStat[0].nOcc++;
}
break;
/* State 3. The integer just read is a column number. */
default: assert( eState==3 );
iCol = (int)v;
if( iCol<1 ){
rc = SQLITE_CORRUPT_VTAB;
break;
}
if( fts3auxGrowStatArray(pCsr, iCol+2) ) return SQLITE_NOMEM;
pCsr->aStat[iCol+1].nDoc++;
eState = 2;
break;
}
}
pCsr->iCol = 0;
}else{
pCsr->isEof = 1;
}
return rc;
}
/*
** xFilter - Initialize a cursor to point at the start of its data.
*/
static int fts3auxFilterMethod(
sqlite3_vtab_cursor *pCursor, /* The cursor used for this query */
int idxNum, /* Strategy index */
const char *idxStr, /* Unused */
int nVal, /* Number of elements in apVal */
sqlite3_value **apVal /* Arguments for the indexing scheme */
){
Fts3auxCursor *pCsr = (Fts3auxCursor *)pCursor;
Fts3Table *pFts3 = ((Fts3auxTable *)pCursor->pVtab)->pFts3Tab;
int rc;
int isScan = 0;
int iLangVal = 0; /* Language id to query */
int iEq = -1; /* Index of term=? value in apVal */
int iGe = -1; /* Index of term>=? value in apVal */
int iLe = -1; /* Index of term<=? value in apVal */
int iLangid = -1; /* Index of languageid=? value in apVal */
int iNext = 0;
UNUSED_PARAMETER(nVal);
UNUSED_PARAMETER(idxStr);
assert( idxStr==0 );
assert( idxNum==FTS4AUX_EQ_CONSTRAINT || idxNum==0
|| idxNum==FTS4AUX_LE_CONSTRAINT || idxNum==FTS4AUX_GE_CONSTRAINT
|| idxNum==(FTS4AUX_LE_CONSTRAINT|FTS4AUX_GE_CONSTRAINT)
);
if( idxNum==FTS4AUX_EQ_CONSTRAINT ){
iEq = iNext++;
}else{
isScan = 1;
if( idxNum & FTS4AUX_GE_CONSTRAINT ){
iGe = iNext++;
}
if( idxNum & FTS4AUX_LE_CONSTRAINT ){
iLe = iNext++;
}
}
if( iNext<nVal ){
iLangid = iNext++;
}
/* In case this cursor is being reused, close and zero it. */
testcase(pCsr->filter.zTerm);
sqlite3Fts3SegReaderFinish(&pCsr->csr);
sqlite3_free((void *)pCsr->filter.zTerm);
sqlite3_free(pCsr->aStat);
sqlite3_free(pCsr->zStop);
memset(&pCsr->csr, 0, ((u8*)&pCsr[1]) - (u8*)&pCsr->csr);
pCsr->filter.flags = FTS3_SEGMENT_REQUIRE_POS|FTS3_SEGMENT_IGNORE_EMPTY;
if( isScan ) pCsr->filter.flags |= FTS3_SEGMENT_SCAN;
if( iEq>=0 || iGe>=0 ){
const unsigned char *zStr = sqlite3_value_text(apVal[0]);
assert( (iEq==0 && iGe==-1) || (iEq==-1 && iGe==0) );
if( zStr ){
pCsr->filter.zTerm = sqlite3_mprintf("%s", zStr);
if( pCsr->filter.zTerm==0 ) return SQLITE_NOMEM;
pCsr->filter.nTerm = (int)strlen(pCsr->filter.zTerm);
}
}
if( iLe>=0 ){
pCsr->zStop = sqlite3_mprintf("%s", sqlite3_value_text(apVal[iLe]));
if( pCsr->zStop==0 ) return SQLITE_NOMEM;
pCsr->nStop = (int)strlen(pCsr->zStop);
}
if( iLangid>=0 ){
iLangVal = sqlite3_value_int(apVal[iLangid]);
/* If the user specified a negative value for the languageid, use zero
** instead. This works, as the "languageid=?" constraint will also
** be tested by the VDBE layer. The test will always be false (since
** this module will not return a row with a negative languageid), and
** so the overall query will return zero rows. */
if( iLangVal<0 ) iLangVal = 0;
}
pCsr->iLangid = iLangVal;
rc = sqlite3Fts3SegReaderCursor(pFts3, iLangVal, 0, FTS3_SEGCURSOR_ALL,
pCsr->filter.zTerm, pCsr->filter.nTerm, 0, isScan, &pCsr->csr
);
if( rc==SQLITE_OK ){
rc = sqlite3Fts3SegReaderStart(pFts3, &pCsr->csr, &pCsr->filter);
}
if( rc==SQLITE_OK ) rc = fts3auxNextMethod(pCursor);
return rc;
}
/*
** xEof - Return true if the cursor is at EOF, or false otherwise.
*/
static int fts3auxEofMethod(sqlite3_vtab_cursor *pCursor){
Fts3auxCursor *pCsr = (Fts3auxCursor *)pCursor;
return pCsr->isEof;
}
/*
** xColumn - Return a column value.
*/
static int fts3auxColumnMethod(
sqlite3_vtab_cursor *pCursor, /* Cursor to retrieve value from */
sqlite3_context *pCtx, /* Context for sqlite3_result_xxx() calls */
int iCol /* Index of column to read value from */
){
Fts3auxCursor *p = (Fts3auxCursor *)pCursor;
assert( p->isEof==0 );
switch( iCol ){
case 0: /* term */
sqlite3_result_text(pCtx, p->csr.zTerm, p->csr.nTerm, SQLITE_TRANSIENT);
break;
case 1: /* col */
if( p->iCol ){
sqlite3_result_int(pCtx, p->iCol-1);
}else{
sqlite3_result_text(pCtx, "*", -1, SQLITE_STATIC);
}
break;
case 2: /* documents */
sqlite3_result_int64(pCtx, p->aStat[p->iCol].nDoc);
break;
case 3: /* occurrences */
sqlite3_result_int64(pCtx, p->aStat[p->iCol].nOcc);
break;
default: /* languageid */
assert( iCol==4 );
sqlite3_result_int(pCtx, p->iLangid);
break;
}
return SQLITE_OK;
}
/*
** xRowid - Return the current rowid for the cursor.
*/
static int fts3auxRowidMethod(
sqlite3_vtab_cursor *pCursor, /* Cursor to retrieve value from */
sqlite_int64 *pRowid /* OUT: Rowid value */
){
Fts3auxCursor *pCsr = (Fts3auxCursor *)pCursor;
*pRowid = pCsr->iRowid;
return SQLITE_OK;
}
/*
** Register the fts3aux module with database connection db. Return SQLITE_OK
** if successful or an error code if sqlite3_create_module() fails.
*/
int sqlite3Fts3InitAux(sqlite3 *db){
static const sqlite3_module fts3aux_module = {
0, /* iVersion */
fts3auxConnectMethod, /* xCreate */
fts3auxConnectMethod, /* xConnect */
fts3auxBestIndexMethod, /* xBestIndex */
fts3auxDisconnectMethod, /* xDisconnect */
fts3auxDisconnectMethod, /* xDestroy */
fts3auxOpenMethod, /* xOpen */
fts3auxCloseMethod, /* xClose */
fts3auxFilterMethod, /* xFilter */
fts3auxNextMethod, /* xNext */
fts3auxEofMethod, /* xEof */
fts3auxColumnMethod, /* xColumn */
fts3auxRowidMethod, /* xRowid */
0, /* xUpdate */
0, /* xBegin */
0, /* xSync */
0, /* xCommit */
0, /* xRollback */
0, /* xFindFunction */
0, /* xRename */
0, /* xSavepoint */
0, /* xRelease */
0, /* xRollbackTo */
0, /* xShadowName */
0 /* xIntegrity */
};
int rc; /* Return code */
rc = sqlite3_create_module(db, "fts4aux", &fts3aux_module, 0);
return rc;
}
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
ext/fts3/fts3_expr.c
+1316
View File
File diff suppressed because it is too large. Load diff
ext/fts3/fts3_hash.c
+383
View File
@@ -0,0 +1,383 @@
/*
** 2001 September 22
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
** This is the implementation of generic hash-tables used in SQLite.
** We've modified it slightly to serve as a standalone hash table
** implementation for the full-text indexing module.
*/
/*
** The code in this file is only compiled if:
**
** * The FTS3 module is being built as an extension
** (in which case SQLITE_CORE is not defined), or
**
** * The FTS3 module is being built into the core of
** SQLite (in which case SQLITE_ENABLE_FTS3 is defined).
*/
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include "fts3_hash.h"
/*
** Malloc and Free functions
*/
static void *fts3HashMalloc(sqlite3_int64 n){
void *p = sqlite3_malloc64(n);
if( p ){
memset(p, 0, n);
}
return p;
}
static void fts3HashFree(void *p){
sqlite3_free(p);
}
/* Turn bulk memory into a hash table object by initializing the
** fields of the Hash structure.
**
** "pNew" is a pointer to the hash table that is to be initialized.
** keyClass is one of the constants
** FTS3_HASH_BINARY or FTS3_HASH_STRING. The value of keyClass
** determines what kind of key the hash table will use. "copyKey" is
** true if the hash table should make its own private copy of keys and
** false if it should just use the supplied pointer.
*/
void sqlite3Fts3HashInit(Fts3Hash *pNew, char keyClass, char copyKey){
assert( pNew!=0 );
assert( keyClass>=FTS3_HASH_STRING && keyClass<=FTS3_HASH_BINARY );
pNew->keyClass = keyClass;
pNew->copyKey = copyKey;
pNew->first = 0;
pNew->count = 0;
pNew->htsize = 0;
pNew->ht = 0;
}
/* Remove all entries from a hash table. Reclaim all memory.
** Call this routine to delete a hash table or to reset a hash table
** to the empty state.
*/
void sqlite3Fts3HashClear(Fts3Hash *pH){
Fts3HashElem *elem; /* For looping over all elements of the table */
assert( pH!=0 );
elem = pH->first;
pH->first = 0;
fts3HashFree(pH->ht);
pH->ht = 0;
pH->htsize = 0;
while( elem ){
Fts3HashElem *next_elem = elem->next;
if( pH->copyKey && elem->pKey ){
fts3HashFree(elem->pKey);
}
fts3HashFree(elem);
elem = next_elem;
}
pH->count = 0;
}
/*
** Hash and comparison functions when the mode is FTS3_HASH_STRING
*/
static int fts3StrHash(const void *pKey, int nKey){
const char *z = (const char *)pKey;
unsigned h = 0;
if( nKey<=0 ) nKey = (int) strlen(z);
while( nKey > 0 ){
h = (h<<3) ^ h ^ *z++;
nKey--;
}
return (int)(h & 0x7fffffff);
}
static int fts3StrCompare(const void *pKey1, int n1, const void *pKey2, int n2){
if( n1!=n2 ) return 1;
return strncmp((const char*)pKey1,(const char*)pKey2,n1);
}
/*
** Hash and comparison functions when the mode is FTS3_HASH_BINARY
*/
static int fts3BinHash(const void *pKey, int nKey){
int h = 0;
const char *z = (const char *)pKey;
while( nKey-- > 0 ){
h = (h<<3) ^ h ^ *(z++);
}
return h & 0x7fffffff;
}
static int fts3BinCompare(const void *pKey1, int n1, const void *pKey2, int n2){
if( n1!=n2 ) return 1;
return memcmp(pKey1,pKey2,n1);
}
/*
** Return a pointer to the appropriate hash function given the key class.
**
** The C syntax in this function definition may be unfamilar to some
** programmers, so we provide the following additional explanation:
**
** The name of the function is "ftsHashFunction". The function takes a
** single parameter "keyClass". The return value of ftsHashFunction()
** is a pointer to another function. Specifically, the return value
** of ftsHashFunction() is a pointer to a function that takes two parameters
** with types "const void*" and "int" and returns an "int".
*/
static int (*ftsHashFunction(int keyClass))(const void*,int){
if( keyClass==FTS3_HASH_STRING ){
return &fts3StrHash;
}else{
assert( keyClass==FTS3_HASH_BINARY );
return &fts3BinHash;
}
}
/*
** Return a pointer to the appropriate hash function given the key class.
**
** For help in interpreted the obscure C code in the function definition,
** see the header comment on the previous function.
*/
static int (*ftsCompareFunction(int keyClass))(const void*,int,const void*,int){
if( keyClass==FTS3_HASH_STRING ){
return &fts3StrCompare;
}else{
assert( keyClass==FTS3_HASH_BINARY );
return &fts3BinCompare;
}
}
/* Link an element into the hash table
*/
static void fts3HashInsertElement(
Fts3Hash *pH, /* The complete hash table */
struct _fts3ht *pEntry, /* The entry into which pNew is inserted */
Fts3HashElem *pNew /* The element to be inserted */
){
Fts3HashElem *pHead; /* First element already in pEntry */
pHead = pEntry->chain;
if( pHead ){
pNew->next = pHead;
pNew->prev = pHead->prev;
if( pHead->prev ){ pHead->prev->next = pNew; }
else { pH->first = pNew; }
pHead->prev = pNew;
}else{
pNew->next = pH->first;
if( pH->first ){ pH->first->prev = pNew; }
pNew->prev = 0;
pH->first = pNew;
}
pEntry->count++;
pEntry->chain = pNew;
}
/* Resize the hash table so that it contains "new_size" buckets.
** "new_size" must be a power of 2. The hash table might fail
** to resize if sqliteMalloc() fails.
**
** Return non-zero if a memory allocation error occurs.
*/
static int fts3Rehash(Fts3Hash *pH, int new_size){
struct _fts3ht *new_ht; /* The new hash table */
Fts3HashElem *elem, *next_elem; /* For looping over existing elements */
int (*xHash)(const void*,int); /* The hash function */
assert( (new_size & (new_size-1))==0 );
new_ht = (struct _fts3ht *)fts3HashMalloc( new_size*sizeof(struct _fts3ht) );
if( new_ht==0 ) return 1;
fts3HashFree(pH->ht);
pH->ht = new_ht;
pH->htsize = new_size;
xHash = ftsHashFunction(pH->keyClass);
for(elem=pH->first, pH->first=0; elem; elem = next_elem){
int h = (*xHash)(elem->pKey, elem->nKey) & (new_size-1);
next_elem = elem->next;
fts3HashInsertElement(pH, &new_ht[h], elem);
}
return 0;
}
/* This function (for internal use only) locates an element in an
** hash table that matches the given key. The hash for this key has
** already been computed and is passed as the 4th parameter.
*/
static Fts3HashElem *fts3FindElementByHash(
const Fts3Hash *pH, /* The pH to be searched */
const void *pKey, /* The key we are searching for */
int nKey,
int h /* The hash for this key. */
){
Fts3HashElem *elem; /* Used to loop thru the element list */
int count; /* Number of elements left to test */
int (*xCompare)(const void*,int,const void*,int); /* comparison function */
if( pH->ht ){
struct _fts3ht *pEntry = &pH->ht[h];
elem = pEntry->chain;
count = pEntry->count;
xCompare = ftsCompareFunction(pH->keyClass);
while( count-- && elem ){
if( (*xCompare)(elem->pKey,elem->nKey,pKey,nKey)==0 ){
return elem;
}
elem = elem->next;
}
}
return 0;
}
/* Remove a single entry from the hash table given a pointer to that
** element and a hash on the element's key.
*/
static void fts3RemoveElementByHash(
Fts3Hash *pH, /* The pH containing "elem" */
Fts3HashElem* elem, /* The element to be removed from the pH */
int h /* Hash value for the element */
){
struct _fts3ht *pEntry;
if( elem->prev ){
elem->prev->next = elem->next;
}else{
pH->first = elem->next;
}
if( elem->next ){
elem->next->prev = elem->prev;
}
pEntry = &pH->ht[h];
if( pEntry->chain==elem ){
pEntry->chain = elem->next;
}
pEntry->count--;
if( pEntry->count<=0 ){
pEntry->chain = 0;
}
if( pH->copyKey && elem->pKey ){
fts3HashFree(elem->pKey);
}
fts3HashFree( elem );
pH->count--;
if( pH->count<=0 ){
assert( pH->first==0 );
assert( pH->count==0 );
fts3HashClear(pH);
}
}
Fts3HashElem *sqlite3Fts3HashFindElem(
const Fts3Hash *pH,
const void *pKey,
int nKey
){
int h; /* A hash on key */
int (*xHash)(const void*,int); /* The hash function */
if( pH==0 || pH->ht==0 ) return 0;
xHash = ftsHashFunction(pH->keyClass);
assert( xHash!=0 );
h = (*xHash)(pKey,nKey);
assert( (pH->htsize & (pH->htsize-1))==0 );
return fts3FindElementByHash(pH,pKey,nKey, h & (pH->htsize-1));
}
/*
** Attempt to locate an element of the hash table pH with a key
** that matches pKey,nKey. Return the data for this element if it is
** found, or NULL if there is no match.
*/
void *sqlite3Fts3HashFind(const Fts3Hash *pH, const void *pKey, int nKey){
Fts3HashElem *pElem; /* The element that matches key (if any) */
pElem = sqlite3Fts3HashFindElem(pH, pKey, nKey);
return pElem ? pElem->data : 0;
}
/* Insert an element into the hash table pH. The key is pKey,nKey
** and the data is "data".
**
** If no element exists with a matching key, then a new
** element is created. A copy of the key is made if the copyKey
** flag is set. NULL is returned.
**
** If another element already exists with the same key, then the
** new data replaces the old data and the old data is returned.
** The key is not copied in this instance. If a malloc fails, then
** the new data is returned and the hash table is unchanged.
**
** If the "data" parameter to this function is NULL, then the
** element corresponding to "key" is removed from the hash table.
*/
void *sqlite3Fts3HashInsert(
Fts3Hash *pH, /* The hash table to insert into */
const void *pKey, /* The key */
int nKey, /* Number of bytes in the key */
void *data /* The data */
){
int hraw; /* Raw hash value of the key */
int h; /* the hash of the key modulo hash table size */
Fts3HashElem *elem; /* Used to loop thru the element list */
Fts3HashElem *new_elem; /* New element added to the pH */
int (*xHash)(const void*,int); /* The hash function */
assert( pH!=0 );
xHash = ftsHashFunction(pH->keyClass);
assert( xHash!=0 );
hraw = (*xHash)(pKey, nKey);
assert( (pH->htsize & (pH->htsize-1))==0 );
h = hraw & (pH->htsize-1);
elem = fts3FindElementByHash(pH,pKey,nKey,h);
if( elem ){
void *old_data = elem->data;
if( data==0 ){
fts3RemoveElementByHash(pH,elem,h);
}else{
elem->data = data;
}
return old_data;
}
if( data==0 ) return 0;
if( (pH->htsize==0 && fts3Rehash(pH,8))
|| (pH->count>=pH->htsize && fts3Rehash(pH, pH->htsize*2))
){
pH->count = 0;
return data;
}
assert( pH->htsize>0 );
new_elem = (Fts3HashElem*)fts3HashMalloc( sizeof(Fts3HashElem) );
if( new_elem==0 ) return data;
if( pH->copyKey && pKey!=0 ){
new_elem->pKey = fts3HashMalloc( nKey );
if( new_elem->pKey==0 ){
fts3HashFree(new_elem);
return data;
}
memcpy((void*)new_elem->pKey, pKey, nKey);
}else{
new_elem->pKey = (void*)pKey;
}
new_elem->nKey = nKey;
pH->count++;
assert( pH->htsize>0 );
assert( (pH->htsize & (pH->htsize-1))==0 );
h = hraw & (pH->htsize-1);
fts3HashInsertElement(pH, &pH->ht[h], new_elem);
new_elem->data = data;
return 0;
}
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
ext/fts3/fts3_hash.h
+112
View File
@@ -0,0 +1,112 @@
/*
** 2001 September 22
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
** This is the header file for the generic hash-table implementation
** used in SQLite. We've modified it slightly to serve as a standalone
** hash table implementation for the full-text indexing module.
**
*/
#ifndef _FTS3_HASH_H_
#define _FTS3_HASH_H_
/* Forward declarations of structures. */
typedef struct Fts3Hash Fts3Hash;
typedef struct Fts3HashElem Fts3HashElem;
/* A complete hash table is an instance of the following structure.
** The internals of this structure are intended to be opaque -- client
** code should not attempt to access or modify the fields of this structure
** directly. Change this structure only by using the routines below.
** However, many of the "procedures" and "functions" for modifying and
** accessing this structure are really macros, so we can't really make
** this structure opaque.
*/
struct Fts3Hash {
char keyClass; /* HASH_INT, _POINTER, _STRING, _BINARY */
char copyKey; /* True if copy of key made on insert */
int count; /* Number of entries in this table */
Fts3HashElem *first; /* The first element of the array */
int htsize; /* Number of buckets in the hash table */
struct _fts3ht { /* the hash table */
int count; /* Number of entries with this hash */
Fts3HashElem *chain; /* Pointer to first entry with this hash */
} *ht;
};
/* Each element in the hash table is an instance of the following
** structure. All elements are stored on a single doubly-linked list.
**
** Again, this structure is intended to be opaque, but it can't really
** be opaque because it is used by macros.
*/
struct Fts3HashElem {
Fts3HashElem *next, *prev; /* Next and previous elements in the table */
void *data; /* Data associated with this element */
void *pKey; int nKey; /* Key associated with this element */
};
/*
** There are 2 different modes of operation for a hash table:
**
** FTS3_HASH_STRING pKey points to a string that is nKey bytes long
** (including the null-terminator, if any). Case
** is respected in comparisons.
**
** FTS3_HASH_BINARY pKey points to binary data nKey bytes long.
** memcmp() is used to compare keys.
**
** A copy of the key is made if the copyKey parameter to fts3HashInit is 1.
*/
#define FTS3_HASH_STRING 1
#define FTS3_HASH_BINARY 2
/*
** Access routines. To delete, insert a NULL pointer.
*/
void sqlite3Fts3HashInit(Fts3Hash *pNew, char keyClass, char copyKey);
void *sqlite3Fts3HashInsert(Fts3Hash*, const void *pKey, int nKey, void *pData);
void *sqlite3Fts3HashFind(const Fts3Hash*, const void *pKey, int nKey);
void sqlite3Fts3HashClear(Fts3Hash*);
Fts3HashElem *sqlite3Fts3HashFindElem(const Fts3Hash *, const void *, int);
/*
** Shorthand for the functions above
*/
#define fts3HashInit sqlite3Fts3HashInit
#define fts3HashInsert sqlite3Fts3HashInsert
#define fts3HashFind sqlite3Fts3HashFind
#define fts3HashClear sqlite3Fts3HashClear
#define fts3HashFindElem sqlite3Fts3HashFindElem
/*
** Macros for looping over all elements of a hash table. The idiom is
** like this:
**
** Fts3Hash h;
** Fts3HashElem *p;
** ...
** for(p=fts3HashFirst(&h); p; p=fts3HashNext(p)){
** SomeStructure *pData = fts3HashData(p);
** // do something with pData
** }
*/
#define fts3HashFirst(H) ((H)->first)
#define fts3HashNext(E) ((E)->next)
#define fts3HashData(E) ((E)->data)
#define fts3HashKey(E) ((E)->pKey)
#define fts3HashKeysize(E) ((E)->nKey)
/*
** Number of entries in a hash table
*/
#define fts3HashCount(H) ((H)->count)
#endif /* _FTS3_HASH_H_ */
ext/fts3/fts3_icu.c
+262
View File
@@ -0,0 +1,262 @@
/*
** 2007 June 22
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
** This file implements a tokenizer for fts3 based on the ICU library.
*/
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#ifdef SQLITE_ENABLE_ICU
#include <assert.h>
#include <string.h>
#include "fts3_tokenizer.h"
#include <unicode/ubrk.h>
#include <unicode/ucol.h>
#include <unicode/ustring.h>
#include <unicode/utf16.h>
typedef struct IcuTokenizer IcuTokenizer;
typedef struct IcuCursor IcuCursor;
struct IcuTokenizer {
sqlite3_tokenizer base;
char *zLocale;
};
struct IcuCursor {
sqlite3_tokenizer_cursor base;
UBreakIterator *pIter; /* ICU break-iterator object */
int nChar; /* Number of UChar elements in pInput */
UChar *aChar; /* Copy of input using utf-16 encoding */
int *aOffset; /* Offsets of each character in utf-8 input */
int nBuffer;
char *zBuffer;
int iToken;
};
/*
** Create a new tokenizer instance.
*/
static int icuCreate(
int argc, /* Number of entries in argv[] */
const char * const *argv, /* Tokenizer creation arguments */
sqlite3_tokenizer **ppTokenizer /* OUT: Created tokenizer */
){
IcuTokenizer *p;
int n = 0;
if( argc>0 ){
n = strlen(argv[0])+1;
}
p = (IcuTokenizer *)sqlite3_malloc64(sizeof(IcuTokenizer)+n);
if( !p ){
return SQLITE_NOMEM;
}
memset(p, 0, sizeof(IcuTokenizer));
if( n ){
p->zLocale = (char *)&p[1];
memcpy(p->zLocale, argv[0], n);
}
*ppTokenizer = (sqlite3_tokenizer *)p;
return SQLITE_OK;
}
/*
** Destroy a tokenizer
*/
static int icuDestroy(sqlite3_tokenizer *pTokenizer){
IcuTokenizer *p = (IcuTokenizer *)pTokenizer;
sqlite3_free(p);
return SQLITE_OK;
}
/*
** Prepare to begin tokenizing a particular string. The input
** string to be tokenized is pInput[0..nBytes-1]. A cursor
** used to incrementally tokenize this string is returned in
** *ppCursor.
*/
static int icuOpen(
sqlite3_tokenizer *pTokenizer, /* The tokenizer */
const char *zInput, /* Input string */
int nInput, /* Length of zInput in bytes */
sqlite3_tokenizer_cursor **ppCursor /* OUT: Tokenization cursor */
){
IcuTokenizer *p = (IcuTokenizer *)pTokenizer;
IcuCursor *pCsr;
const int32_t opt = U_FOLD_CASE_DEFAULT;
UErrorCode status = U_ZERO_ERROR;
int nChar;
UChar32 c;
int iInput = 0;
int iOut = 0;
*ppCursor = 0;
if( zInput==0 ){
nInput = 0;
zInput = "";
}else if( nInput<0 ){
nInput = strlen(zInput);
}
nChar = nInput+1;
pCsr = (IcuCursor *)sqlite3_malloc64(
sizeof(IcuCursor) + /* IcuCursor */
((nChar+3)&~3) * sizeof(UChar) + /* IcuCursor.aChar[] */
(nChar+1) * sizeof(int) /* IcuCursor.aOffset[] */
);
if( !pCsr ){
return SQLITE_NOMEM;
}
memset(pCsr, 0, sizeof(IcuCursor));
pCsr->aChar = (UChar *)&pCsr[1];
pCsr->aOffset = (int *)&pCsr->aChar[(nChar+3)&~3];
pCsr->aOffset[iOut] = iInput;
U8_NEXT(zInput, iInput, nInput, c);
while( c>0 ){
int isError = 0;
c = u_foldCase(c, opt);
U16_APPEND(pCsr->aChar, iOut, nChar, c, isError);
if( isError ){
sqlite3_free(pCsr);
return SQLITE_ERROR;
}
pCsr->aOffset[iOut] = iInput;
if( iInput<nInput ){
U8_NEXT(zInput, iInput, nInput, c);
}else{
c = 0;
}
}
pCsr->pIter = ubrk_open(UBRK_WORD, p->zLocale, pCsr->aChar, iOut, &status);
if( !U_SUCCESS(status) ){
sqlite3_free(pCsr);
return SQLITE_ERROR;
}
pCsr->nChar = iOut;
ubrk_first(pCsr->pIter);
*ppCursor = (sqlite3_tokenizer_cursor *)pCsr;
return SQLITE_OK;
}
/*
** Close a tokenization cursor previously opened by a call to icuOpen().
*/
static int icuClose(sqlite3_tokenizer_cursor *pCursor){
IcuCursor *pCsr = (IcuCursor *)pCursor;
ubrk_close(pCsr->pIter);
sqlite3_free(pCsr->zBuffer);
sqlite3_free(pCsr);
return SQLITE_OK;
}
/*
** Extract the next token from a tokenization cursor.
*/
static int icuNext(
sqlite3_tokenizer_cursor *pCursor, /* Cursor returned by simpleOpen */
const char **ppToken, /* OUT: *ppToken is the token text */
int *pnBytes, /* OUT: Number of bytes in token */
int *piStartOffset, /* OUT: Starting offset of token */
int *piEndOffset, /* OUT: Ending offset of token */
int *piPosition /* OUT: Position integer of token */
){
IcuCursor *pCsr = (IcuCursor *)pCursor;
int iStart = 0;
int iEnd = 0;
int nByte = 0;
while( iStart==iEnd ){
UChar32 c;
iStart = ubrk_current(pCsr->pIter);
iEnd = ubrk_next(pCsr->pIter);
if( iEnd==UBRK_DONE ){
return SQLITE_DONE;
}
while( iStart<iEnd ){
int iWhite = iStart;
U16_NEXT(pCsr->aChar, iWhite, pCsr->nChar, c);
if( u_isspace(c) ){
iStart = iWhite;
}else{
break;
}
}
assert(iStart<=iEnd);
}
do {
UErrorCode status = U_ZERO_ERROR;
if( nByte ){
char *zNew = sqlite3_realloc(pCsr->zBuffer, nByte);
if( !zNew ){
return SQLITE_NOMEM;
}
pCsr->zBuffer = zNew;
pCsr->nBuffer = nByte;
}
u_strToUTF8(
pCsr->zBuffer, pCsr->nBuffer, &nByte, /* Output vars */
&pCsr->aChar[iStart], iEnd-iStart, /* Input vars */
&status /* Output success/failure */
);
} while( nByte>pCsr->nBuffer );
*ppToken = pCsr->zBuffer;
*pnBytes = nByte;
*piStartOffset = pCsr->aOffset[iStart];
*piEndOffset = pCsr->aOffset[iEnd];
*piPosition = pCsr->iToken++;
return SQLITE_OK;
}
/*
** The set of routines that implement the simple tokenizer
*/
static const sqlite3_tokenizer_module icuTokenizerModule = {
0, /* iVersion */
icuCreate, /* xCreate */
icuDestroy, /* xCreate */
icuOpen, /* xOpen */
icuClose, /* xClose */
icuNext, /* xNext */
0, /* xLanguageid */
};
/*
** Set *ppModule to point at the implementation of the ICU tokenizer.
*/
void sqlite3Fts3IcuTokenizerModule(
sqlite3_tokenizer_module const**ppModule
){
*ppModule = &icuTokenizerModule;
}
#endif /* defined(SQLITE_ENABLE_ICU) */
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
ext/fts3/fts3_porter.c
+662
View File
@@ -0,0 +1,662 @@
/*
** 2006 September 30
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
** Implementation of the full-text-search tokenizer that implements
** a Porter stemmer.
*/
/*
** The code in this file is only compiled if:
**
** * The FTS3 module is being built as an extension
** (in which case SQLITE_CORE is not defined), or
**
** * The FTS3 module is being built into the core of
** SQLite (in which case SQLITE_ENABLE_FTS3 is defined).
*/
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "fts3_tokenizer.h"
/*
** Class derived from sqlite3_tokenizer
*/
typedef struct porter_tokenizer {
sqlite3_tokenizer base; /* Base class */
} porter_tokenizer;
/*
** Class derived from sqlite3_tokenizer_cursor
*/
typedef struct porter_tokenizer_cursor {
sqlite3_tokenizer_cursor base;
const char *zInput; /* input we are tokenizing */
int nInput; /* size of the input */
int iOffset; /* current position in zInput */
int iToken; /* index of next token to be returned */
char *zToken; /* storage for current token */
int nAllocated; /* space allocated to zToken buffer */
} porter_tokenizer_cursor;
/*
** Create a new tokenizer instance.
*/
static int porterCreate(
int argc, const char * const *argv,
sqlite3_tokenizer **ppTokenizer
){
porter_tokenizer *t;
UNUSED_PARAMETER(argc);
UNUSED_PARAMETER(argv);
t = (porter_tokenizer *) sqlite3_malloc(sizeof(*t));
if( t==NULL ) return SQLITE_NOMEM;
memset(t, 0, sizeof(*t));
*ppTokenizer = &t->base;
return SQLITE_OK;
}
/*
** Destroy a tokenizer
*/
static int porterDestroy(sqlite3_tokenizer *pTokenizer){
sqlite3_free(pTokenizer);
return SQLITE_OK;
}
/*
** Prepare to begin tokenizing a particular string. The input
** string to be tokenized is zInput[0..nInput-1]. A cursor
** used to incrementally tokenize this string is returned in
** *ppCursor.
*/
static int porterOpen(
sqlite3_tokenizer *pTokenizer, /* The tokenizer */
const char *zInput, int nInput, /* String to be tokenized */
sqlite3_tokenizer_cursor **ppCursor /* OUT: Tokenization cursor */
){
porter_tokenizer_cursor *c;
UNUSED_PARAMETER(pTokenizer);
c = (porter_tokenizer_cursor *) sqlite3_malloc(sizeof(*c));
if( c==NULL ) return SQLITE_NOMEM;
c->zInput = zInput;
if( zInput==0 ){
c->nInput = 0;
}else if( nInput<0 ){
c->nInput = (int)strlen(zInput);
}else{
c->nInput = nInput;
}
c->iOffset = 0; /* start tokenizing at the beginning */
c->iToken = 0;
c->zToken = NULL; /* no space allocated, yet. */
c->nAllocated = 0;
*ppCursor = &c->base;
return SQLITE_OK;
}
/*
** Close a tokenization cursor previously opened by a call to
** porterOpen() above.
*/
static int porterClose(sqlite3_tokenizer_cursor *pCursor){
porter_tokenizer_cursor *c = (porter_tokenizer_cursor *) pCursor;
sqlite3_free(c->zToken);
sqlite3_free(c);
return SQLITE_OK;
}
/*
** Vowel or consonant
*/
static const char cType[] = {
0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0,
1, 1, 1, 2, 1
};
/*
** isConsonant() and isVowel() determine if their first character in
** the string they point to is a consonant or a vowel, according
** to Porter ruls.
**
** A consonate is any letter other than 'a', 'e', 'i', 'o', or 'u'.
** 'Y' is a consonant unless it follows another consonant,
** in which case it is a vowel.
**
** In these routine, the letters are in reverse order. So the 'y' rule
** is that 'y' is a consonant unless it is followed by another
** consonent.
*/
static int isVowel(const char*);
static int isConsonant(const char *z){
int j;
char x = *z;
if( x==0 ) return 0;
assert( x>='a' && x<='z' );
j = cType[x-'a'];
if( j<2 ) return j;
return z[1]==0 || isVowel(z + 1);
}
static int isVowel(const char *z){
int j;
char x = *z;
if( x==0 ) return 0;
assert( x>='a' && x<='z' );
j = cType[x-'a'];
if( j<2 ) return 1-j;
return isConsonant(z + 1);
}
/*
** Let any sequence of one or more vowels be represented by V and let
** C be sequence of one or more consonants. Then every word can be
** represented as:
**
** [C] (VC){m} [V]
**
** In prose: A word is an optional consonant followed by zero or
** vowel-consonant pairs followed by an optional vowel. "m" is the
** number of vowel consonant pairs. This routine computes the value
** of m for the first i bytes of a word.
**
** Return true if the m-value for z is 1 or more. In other words,
** return true if z contains at least one vowel that is followed
** by a consonant.
**
** In this routine z[] is in reverse order. So we are really looking
** for an instance of a consonant followed by a vowel.
*/
static int m_gt_0(const char *z){
while( isVowel(z) ){ z++; }
if( *z==0 ) return 0;
while( isConsonant(z) ){ z++; }
return *z!=0;
}
/* Like mgt0 above except we are looking for a value of m which is
** exactly 1
*/
static int m_eq_1(const char *z){
while( isVowel(z) ){ z++; }
if( *z==0 ) return 0;
while( isConsonant(z) ){ z++; }
if( *z==0 ) return 0;
while( isVowel(z) ){ z++; }
if( *z==0 ) return 1;
while( isConsonant(z) ){ z++; }
return *z==0;
}
/* Like mgt0 above except we are looking for a value of m>1 instead
** or m>0
*/
static int m_gt_1(const char *z){
while( isVowel(z) ){ z++; }
if( *z==0 ) return 0;
while( isConsonant(z) ){ z++; }
if( *z==0 ) return 0;
while( isVowel(z) ){ z++; }
if( *z==0 ) return 0;
while( isConsonant(z) ){ z++; }
return *z!=0;
}
/*
** Return TRUE if there is a vowel anywhere within z[0..n-1]
*/
static int hasVowel(const char *z){
while( isConsonant(z) ){ z++; }
return *z!=0;
}
/*
** Return TRUE if the word ends in a double consonant.
**
** The text is reversed here. So we are really looking at
** the first two characters of z[].
*/
static int doubleConsonant(const char *z){
return isConsonant(z) && z[0]==z[1];
}
/*
** Return TRUE if the word ends with three letters which
** are consonant-vowel-consonent and where the final consonant
** is not 'w', 'x', or 'y'.
**
** The word is reversed here. So we are really checking the
** first three letters and the first one cannot be in [wxy].
*/
static int star_oh(const char *z){
return
isConsonant(z) &&
z[0]!='w' && z[0]!='x' && z[0]!='y' &&
isVowel(z+1) &&
isConsonant(z+2);
}
/*
** If the word ends with zFrom and xCond() is true for the stem
** of the word that precedes the zFrom ending, then change the
** ending to zTo.
**
** The input word *pz and zFrom are both in reverse order. zTo
** is in normal order.
**
** Return TRUE if zFrom matches. Return FALSE if zFrom does not
** match. Not that TRUE is returned even if xCond() fails and
** no substitution occurs.
*/
static int stem(
char **pz, /* The word being stemmed (Reversed) */
const char *zFrom, /* If the ending matches this... (Reversed) */
const char *zTo, /* ... change the ending to this (not reversed) */
int (*xCond)(const char*) /* Condition that must be true */
){
char *z = *pz;
while( *zFrom && *zFrom==*z ){ z++; zFrom++; }
if( *zFrom!=0 ) return 0;
if( xCond && !xCond(z) ) return 1;
while( *zTo ){
*(--z) = *(zTo++);
}
*pz = z;
return 1;
}
/*
** This is the fallback stemmer used when the porter stemmer is
** inappropriate. The input word is copied into the output with
** US-ASCII case folding. If the input word is too long (more
** than 20 bytes if it contains no digits or more than 6 bytes if
** it contains digits) then word is truncated to 20 or 6 bytes
** by taking 10 or 3 bytes from the beginning and end.
*/
static void copy_stemmer(const char *zIn, int nIn, char *zOut, int *pnOut){
int i, mx, j;
int hasDigit = 0;
for(i=0; i<nIn; i++){
char c = zIn[i];
if( c>='A' && c<='Z' ){
zOut[i] = c - 'A' + 'a';
}else{
if( c>='0' && c<='9' ) hasDigit = 1;
zOut[i] = c;
}
}
mx = hasDigit ? 3 : 10;
if( nIn>mx*2 ){
for(j=mx, i=nIn-mx; i<nIn; i++, j++){
zOut[j] = zOut[i];
}
i = j;
}
zOut[i] = 0;
*pnOut = i;
}
/*
** Stem the input word zIn[0..nIn-1]. Store the output in zOut.
** zOut is at least big enough to hold nIn bytes. Write the actual
** size of the output word (exclusive of the '\0' terminator) into *pnOut.
**
** Any upper-case characters in the US-ASCII character set ([A-Z])
** are converted to lower case. Upper-case UTF characters are
** unchanged.
**
** Words that are longer than about 20 bytes are stemmed by retaining
** a few bytes from the beginning and the end of the word. If the
** word contains digits, 3 bytes are taken from the beginning and
** 3 bytes from the end. For long words without digits, 10 bytes
** are taken from each end. US-ASCII case folding still applies.
**
** If the input word contains not digits but does characters not
** in [a-zA-Z] then no stemming is attempted and this routine just
** copies the input into the input into the output with US-ASCII
** case folding.
**
** Stemming never increases the length of the word. So there is
** no chance of overflowing the zOut buffer.
*/
static void porter_stemmer(const char *zIn, int nIn, char *zOut, int *pnOut){
int i, j;
char zReverse[28];
char *z, *z2;
if( nIn<3 || nIn>=(int)sizeof(zReverse)-7 ){
/* The word is too big or too small for the porter stemmer.
** Fallback to the copy stemmer */
copy_stemmer(zIn, nIn, zOut, pnOut);
return;
}
for(i=0, j=sizeof(zReverse)-6; i<nIn; i++, j--){
char c = zIn[i];
if( c>='A' && c<='Z' ){
zReverse[j] = c + 'a' - 'A';
}else if( c>='a' && c<='z' ){
zReverse[j] = c;
}else{
/* The use of a character not in [a-zA-Z] means that we fallback
** to the copy stemmer */
copy_stemmer(zIn, nIn, zOut, pnOut);
return;
}
}
memset(&zReverse[sizeof(zReverse)-5], 0, 5);
z = &zReverse[j+1];
/* Step 1a */
if( z[0]=='s' ){
if(
!stem(&z, "sess", "ss", 0) &&
!stem(&z, "sei", "i", 0) &&
!stem(&z, "ss", "ss", 0)
){
z++;
}
}
/* Step 1b */
z2 = z;
if( stem(&z, "dee", "ee", m_gt_0) ){
/* Do nothing. The work was all in the test */
}else if(
(stem(&z, "gni", "", hasVowel) || stem(&z, "de", "", hasVowel))
&& z!=z2
){
if( stem(&z, "ta", "ate", 0) ||
stem(&z, "lb", "ble", 0) ||
stem(&z, "zi", "ize", 0) ){
/* Do nothing. The work was all in the test */
}else if( doubleConsonant(z) && (*z!='l' && *z!='s' && *z!='z') ){
z++;
}else if( m_eq_1(z) && star_oh(z) ){
*(--z) = 'e';
}
}
/* Step 1c */
if( z[0]=='y' && hasVowel(z+1) ){
z[0] = 'i';
}
/* Step 2 */
switch( z[1] ){
case 'a':
if( !stem(&z, "lanoita", "ate", m_gt_0) ){
stem(&z, "lanoit", "tion", m_gt_0);
}
break;
case 'c':
if( !stem(&z, "icne", "ence", m_gt_0) ){
stem(&z, "icna", "ance", m_gt_0);
}
break;
case 'e':
stem(&z, "rezi", "ize", m_gt_0);
break;
case 'g':
stem(&z, "igol", "log", m_gt_0);
break;
case 'l':
if( !stem(&z, "ilb", "ble", m_gt_0)
&& !stem(&z, "illa", "al", m_gt_0)
&& !stem(&z, "iltne", "ent", m_gt_0)
&& !stem(&z, "ile", "e", m_gt_0)
){
stem(&z, "ilsuo", "ous", m_gt_0);
}
break;
case 'o':
if( !stem(&z, "noitazi", "ize", m_gt_0)
&& !stem(&z, "noita", "ate", m_gt_0)
){
stem(&z, "rota", "ate", m_gt_0);
}
break;
case 's':
if( !stem(&z, "msila", "al", m_gt_0)
&& !stem(&z, "ssenevi", "ive", m_gt_0)
&& !stem(&z, "ssenluf", "ful", m_gt_0)
){
stem(&z, "ssensuo", "ous", m_gt_0);
}
break;
case 't':
if( !stem(&z, "itila", "al", m_gt_0)
&& !stem(&z, "itivi", "ive", m_gt_0)
){
stem(&z, "itilib", "ble", m_gt_0);
}
break;
}
/* Step 3 */
switch( z[0] ){
case 'e':
if( !stem(&z, "etaci", "ic", m_gt_0)
&& !stem(&z, "evita", "", m_gt_0)
){
stem(&z, "ezila", "al", m_gt_0);
}
break;
case 'i':
stem(&z, "itici", "ic", m_gt_0);
break;
case 'l':
if( !stem(&z, "laci", "ic", m_gt_0) ){
stem(&z, "luf", "", m_gt_0);
}
break;
case 's':
stem(&z, "ssen", "", m_gt_0);
break;
}
/* Step 4 */
switch( z[1] ){
case 'a':
if( z[0]=='l' && m_gt_1(z+2) ){
z += 2;
}
break;
case 'c':
if( z[0]=='e' && z[2]=='n' && (z[3]=='a' || z[3]=='e') && m_gt_1(z+4) ){
z += 4;
}
break;
case 'e':
if( z[0]=='r' && m_gt_1(z+2) ){
z += 2;
}
break;
case 'i':
if( z[0]=='c' && m_gt_1(z+2) ){
z += 2;
}
break;
case 'l':
if( z[0]=='e' && z[2]=='b' && (z[3]=='a' || z[3]=='i') && m_gt_1(z+4) ){
z += 4;
}
break;
case 'n':
if( z[0]=='t' ){
if( z[2]=='a' ){
if( m_gt_1(z+3) ){
z += 3;
}
}else if( z[2]=='e' ){
if( !stem(&z, "tneme", "", m_gt_1)
&& !stem(&z, "tnem", "", m_gt_1)
){
stem(&z, "tne", "", m_gt_1);
}
}
}
break;
case 'o':
if( z[0]=='u' ){
if( m_gt_1(z+2) ){
z += 2;
}
}else if( z[3]=='s' || z[3]=='t' ){
stem(&z, "noi", "", m_gt_1);
}
break;
case 's':
if( z[0]=='m' && z[2]=='i' && m_gt_1(z+3) ){
z += 3;
}
break;
case 't':
if( !stem(&z, "eta", "", m_gt_1) ){
stem(&z, "iti", "", m_gt_1);
}
break;
case 'u':
if( z[0]=='s' && z[2]=='o' && m_gt_1(z+3) ){
z += 3;
}
break;
case 'v':
case 'z':
if( z[0]=='e' && z[2]=='i' && m_gt_1(z+3) ){
z += 3;
}
break;
}
/* Step 5a */
if( z[0]=='e' ){
if( m_gt_1(z+1) ){
z++;
}else if( m_eq_1(z+1) && !star_oh(z+1) ){
z++;
}
}
/* Step 5b */
if( m_gt_1(z) && z[0]=='l' && z[1]=='l' ){
z++;
}
/* z[] is now the stemmed word in reverse order. Flip it back
** around into forward order and return.
*/
*pnOut = i = (int)strlen(z);
zOut[i] = 0;
while( *z ){
zOut[--i] = *(z++);
}
}
/*
** Characters that can be part of a token. We assume any character
** whose value is greater than 0x80 (any UTF character) can be
** part of a token. In other words, delimiters all must have
** values of 0x7f or lower.
*/
static const char porterIdChar[] = {
/* x0 x1 x2 x3 x4 x5 x6 x7 x8 x9 xA xB xC xD xE xF */
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, /* 3x */
0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, /* 4x */
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, /* 5x */
0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, /* 6x */
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, /* 7x */
};
#define isDelim(C) (((ch=C)&0x80)==0 && (ch<0x30 || !porterIdChar[ch-0x30]))
/*
** Extract the next token from a tokenization cursor. The cursor must
** have been opened by a prior call to porterOpen().
*/
static int porterNext(
sqlite3_tokenizer_cursor *pCursor, /* Cursor returned by porterOpen */
const char **pzToken, /* OUT: *pzToken is the token text */
int *pnBytes, /* OUT: Number of bytes in token */
int *piStartOffset, /* OUT: Starting offset of token */
int *piEndOffset, /* OUT: Ending offset of token */
int *piPosition /* OUT: Position integer of token */
){
porter_tokenizer_cursor *c = (porter_tokenizer_cursor *) pCursor;
const char *z = c->zInput;
while( c->iOffset<c->nInput ){
int iStartOffset, ch;
/* Scan past delimiter characters */
while( c->iOffset<c->nInput && isDelim(z[c->iOffset]) ){
c->iOffset++;
}
/* Count non-delimiter characters. */
iStartOffset = c->iOffset;
while( c->iOffset<c->nInput && !isDelim(z[c->iOffset]) ){
c->iOffset++;
}
if( c->iOffset>iStartOffset ){
int n = c->iOffset-iStartOffset;
if( n>c->nAllocated ){
char *pNew;
c->nAllocated = n+20;
pNew = sqlite3_realloc64(c->zToken, c->nAllocated);
if( !pNew ) return SQLITE_NOMEM;
c->zToken = pNew;
}
porter_stemmer(&z[iStartOffset], n, c->zToken, pnBytes);
*pzToken = c->zToken;
*piStartOffset = iStartOffset;
*piEndOffset = c->iOffset;
*piPosition = c->iToken++;
return SQLITE_OK;
}
}
return SQLITE_DONE;
}
/*
** The set of routines that implement the porter-stemmer tokenizer
*/
static const sqlite3_tokenizer_module porterTokenizerModule = {
0,
porterCreate,
porterDestroy,
porterOpen,
porterClose,
porterNext,
0
};
/*
** Allocate a new porter tokenizer. Return a pointer to the new
** tokenizer in *ppModule
*/
void sqlite3Fts3PorterTokenizerModule(
sqlite3_tokenizer_module const**ppModule
){
*ppModule = &porterTokenizerModule;
}
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
ext/fts3/fts3_snippet.c
+1782
View File
File diff suppressed because it is too large. Load diff
ext/fts3/fts3_term.c
+383
View File
@@ -0,0 +1,383 @@
/*
** 2011 Jan 27
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
** This file is not part of the production FTS code. It is only used for
** testing. It contains a virtual table implementation that provides direct
** access to the full-text index of an FTS table.
*/
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#ifdef SQLITE_TEST
#include <string.h>
#include <assert.h>
#include <stdlib.h>
typedef struct Fts3termTable Fts3termTable;
typedef struct Fts3termCursor Fts3termCursor;
struct Fts3termTable {
sqlite3_vtab base; /* Base class used by SQLite core */
int iIndex; /* Index for Fts3Table.aIndex[] */
Fts3Table *pFts3Tab;
};
struct Fts3termCursor {
sqlite3_vtab_cursor base; /* Base class used by SQLite core */
Fts3MultiSegReader csr; /* Must be right after "base" */
Fts3SegFilter filter;
int isEof; /* True if cursor is at EOF */
char *pNext;
sqlite3_int64 iRowid; /* Current 'rowid' value */
sqlite3_int64 iDocid; /* Current 'docid' value */
int iCol; /* Current 'col' value */
int iPos; /* Current 'pos' value */
};
/*
** Schema of the terms table.
*/
#define FTS3_TERMS_SCHEMA "CREATE TABLE x(term, docid, col, pos)"
/*
** This function does all the work for both the xConnect and xCreate methods.
** These tables have no persistent representation of their own, so xConnect
** and xCreate are identical operations.
*/
static int fts3termConnectMethod(
sqlite3 *db, /* Database connection */
void *pCtx, /* Non-zero for an fts4prefix table */
int argc, /* Number of elements in argv array */
const char * const *argv, /* xCreate/xConnect argument array */
sqlite3_vtab **ppVtab, /* OUT: New sqlite3_vtab object */
char **pzErr /* OUT: sqlite3_malloc'd error message */
){
char const *zDb; /* Name of database (e.g. "main") */
char const *zFts3; /* Name of fts3 table */
int nDb; /* Result of strlen(zDb) */
int nFts3; /* Result of strlen(zFts3) */
sqlite3_int64 nByte; /* Bytes of space to allocate here */
int rc; /* value returned by declare_vtab() */
Fts3termTable *p; /* Virtual table object to return */
int iIndex = 0;
UNUSED_PARAMETER(pCtx);
if( argc==5 ){
iIndex = atoi(argv[4]);
argc--;
}
*ppVtab = 0;
/* The user should specify a single argument - the name of an fts3 table. */
if( argc!=4 ){
sqlite3Fts3ErrMsg(pzErr,
"wrong number of arguments to fts4term constructor"
);
return SQLITE_ERROR;
}
zDb = argv[1];
nDb = (int)strlen(zDb);
zFts3 = argv[3];
nFts3 = (int)strlen(zFts3);
rc = sqlite3_declare_vtab(db, FTS3_TERMS_SCHEMA);
if( rc!=SQLITE_OK ) return rc;
nByte = sizeof(Fts3termTable);
p = (Fts3termTable *)sqlite3Fts3MallocZero(nByte);
if( !p ) return SQLITE_NOMEM;
p->pFts3Tab = (Fts3Table*)sqlite3Fts3MallocZero(
sizeof(Fts3Table) + nDb + nFts3 + 2
);
if( p->pFts3Tab==0 ){
sqlite3_free(p);
return SQLITE_NOMEM;
}
p->pFts3Tab->zDb = (char *)&p->pFts3Tab[1];
p->pFts3Tab->zName = &p->pFts3Tab->zDb[nDb+1];
p->pFts3Tab->db = db;
p->pFts3Tab->nIndex = iIndex+1;
p->iIndex = iIndex;
memcpy((char *)p->pFts3Tab->zDb, zDb, nDb);
memcpy((char *)p->pFts3Tab->zName, zFts3, nFts3);
sqlite3Fts3Dequote((char *)p->pFts3Tab->zName);
*ppVtab = (sqlite3_vtab *)p;
return SQLITE_OK;
}
/*
** This function does the work for both the xDisconnect and xDestroy methods.
** These tables have no persistent representation of their own, so xDisconnect
** and xDestroy are identical operations.
*/
static int fts3termDisconnectMethod(sqlite3_vtab *pVtab){
Fts3termTable *p = (Fts3termTable *)pVtab;
Fts3Table *pFts3 = p->pFts3Tab;
int i;
/* Free any prepared statements held */
for(i=0; i<SizeofArray(pFts3->aStmt); i++){
sqlite3_finalize(pFts3->aStmt[i]);
}
sqlite3_free(pFts3->zSegmentsTbl);
sqlite3_free(pFts3);
sqlite3_free(p);
return SQLITE_OK;
}
#define FTS4AUX_EQ_CONSTRAINT 1
#define FTS4AUX_GE_CONSTRAINT 2
#define FTS4AUX_LE_CONSTRAINT 4
/*
** xBestIndex - Analyze a WHERE and ORDER BY clause.
*/
static int fts3termBestIndexMethod(
sqlite3_vtab *pVTab,
sqlite3_index_info *pInfo
){
UNUSED_PARAMETER(pVTab);
/* This vtab naturally does "ORDER BY term, docid, col, pos". */
if( pInfo->nOrderBy ){
int i;
for(i=0; i<pInfo->nOrderBy; i++){
if( pInfo->aOrderBy[i].iColumn!=i || pInfo->aOrderBy[i].desc ) break;
}
if( i==pInfo->nOrderBy ){
pInfo->orderByConsumed = 1;
}
}
return SQLITE_OK;
}
/*
** xOpen - Open a cursor.
*/
static int fts3termOpenMethod(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCsr){
Fts3termCursor *pCsr; /* Pointer to cursor object to return */
UNUSED_PARAMETER(pVTab);
pCsr = (Fts3termCursor *)sqlite3_malloc(sizeof(Fts3termCursor));
if( !pCsr ) return SQLITE_NOMEM;
memset(pCsr, 0, sizeof(Fts3termCursor));
*ppCsr = (sqlite3_vtab_cursor *)pCsr;
return SQLITE_OK;
}
/*
** xClose - Close a cursor.
*/
static int fts3termCloseMethod(sqlite3_vtab_cursor *pCursor){
Fts3Table *pFts3 = ((Fts3termTable *)pCursor->pVtab)->pFts3Tab;
Fts3termCursor *pCsr = (Fts3termCursor *)pCursor;
sqlite3Fts3SegmentsClose(pFts3);
sqlite3Fts3SegReaderFinish(&pCsr->csr);
sqlite3_free(pCsr);
return SQLITE_OK;
}
/*
** xNext - Advance the cursor to the next row, if any.
*/
static int fts3termNextMethod(sqlite3_vtab_cursor *pCursor){
Fts3termCursor *pCsr = (Fts3termCursor *)pCursor;
Fts3Table *pFts3 = ((Fts3termTable *)pCursor->pVtab)->pFts3Tab;
int rc;
sqlite3_int64 v;
/* Increment our pretend rowid value. */
pCsr->iRowid++;
/* Advance to the next term in the full-text index. */
if( pCsr->csr.aDoclist==0
|| pCsr->pNext>=&pCsr->csr.aDoclist[pCsr->csr.nDoclist-1]
){
rc = sqlite3Fts3SegReaderStep(pFts3, &pCsr->csr);
if( rc!=SQLITE_ROW ){
pCsr->isEof = 1;
return rc;
}
pCsr->iCol = 0;
pCsr->iPos = 0;
pCsr->iDocid = 0;
pCsr->pNext = pCsr->csr.aDoclist;
/* Read docid */
pCsr->pNext += sqlite3Fts3GetVarint(pCsr->pNext, &pCsr->iDocid);
}
pCsr->pNext += sqlite3Fts3GetVarint(pCsr->pNext, &v);
if( v==0 ){
pCsr->pNext += sqlite3Fts3GetVarint(pCsr->pNext, &v);
pCsr->iDocid += v;
pCsr->pNext += sqlite3Fts3GetVarint(pCsr->pNext, &v);
pCsr->iCol = 0;
pCsr->iPos = 0;
}
if( v==1 ){
pCsr->pNext += sqlite3Fts3GetVarint(pCsr->pNext, &v);
pCsr->iCol += (int)v;
pCsr->iPos = 0;
pCsr->pNext += sqlite3Fts3GetVarint(pCsr->pNext, &v);
}
pCsr->iPos += (int)(v - 2);
return SQLITE_OK;
}
/*
** xFilter - Initialize a cursor to point at the start of its data.
*/
static int fts3termFilterMethod(
sqlite3_vtab_cursor *pCursor, /* The cursor used for this query */
int idxNum, /* Strategy index */
const char *idxStr, /* Unused */
int nVal, /* Number of elements in apVal */
sqlite3_value **apVal /* Arguments for the indexing scheme */
){
Fts3termCursor *pCsr = (Fts3termCursor *)pCursor;
Fts3termTable *p = (Fts3termTable *)pCursor->pVtab;
Fts3Table *pFts3 = p->pFts3Tab;
int rc;
UNUSED_PARAMETER(nVal);
UNUSED_PARAMETER(idxNum);
UNUSED_PARAMETER(idxStr);
UNUSED_PARAMETER(apVal);
assert( idxStr==0 && idxNum==0 );
/* In case this cursor is being reused, close and zero it. */
testcase(pCsr->filter.zTerm);
sqlite3Fts3SegReaderFinish(&pCsr->csr);
memset(&pCsr->csr, 0, ((u8*)&pCsr[1]) - (u8*)&pCsr->csr);
pCsr->filter.flags = FTS3_SEGMENT_REQUIRE_POS|FTS3_SEGMENT_IGNORE_EMPTY;
pCsr->filter.flags |= FTS3_SEGMENT_SCAN;
rc = sqlite3Fts3SegReaderCursor(pFts3, 0, p->iIndex, FTS3_SEGCURSOR_ALL,
pCsr->filter.zTerm, pCsr->filter.nTerm, 0, 1, &pCsr->csr
);
if( rc==SQLITE_OK ){
rc = sqlite3Fts3SegReaderStart(pFts3, &pCsr->csr, &pCsr->filter);
}
if( rc==SQLITE_OK ){
rc = fts3termNextMethod(pCursor);
}
return rc;
}
/*
** xEof - Return true if the cursor is at EOF, or false otherwise.
*/
static int fts3termEofMethod(sqlite3_vtab_cursor *pCursor){
Fts3termCursor *pCsr = (Fts3termCursor *)pCursor;
return pCsr->isEof;
}
/*
** xColumn - Return a column value.
*/
static int fts3termColumnMethod(
sqlite3_vtab_cursor *pCursor, /* Cursor to retrieve value from */
sqlite3_context *pCtx, /* Context for sqlite3_result_xxx() calls */
int iCol /* Index of column to read value from */
){
Fts3termCursor *p = (Fts3termCursor *)pCursor;
assert( iCol>=0 && iCol<=3 );
switch( iCol ){
case 0:
sqlite3_result_text(pCtx, p->csr.zTerm, p->csr.nTerm, SQLITE_TRANSIENT);
break;
case 1:
sqlite3_result_int64(pCtx, p->iDocid);
break;
case 2:
sqlite3_result_int64(pCtx, p->iCol);
break;
default:
sqlite3_result_int64(pCtx, p->iPos);
break;
}
return SQLITE_OK;
}
/*
** xRowid - Return the current rowid for the cursor.
*/
static int fts3termRowidMethod(
sqlite3_vtab_cursor *pCursor, /* Cursor to retrieve value from */
sqlite_int64 *pRowid /* OUT: Rowid value */
){
Fts3termCursor *pCsr = (Fts3termCursor *)pCursor;
*pRowid = pCsr->iRowid;
return SQLITE_OK;
}
/*
** Register the fts3term module with database connection db. Return SQLITE_OK
** if successful or an error code if sqlite3_create_module() fails.
*/
int sqlite3Fts3InitTerm(sqlite3 *db){
static const sqlite3_module fts3term_module = {
0, /* iVersion */
fts3termConnectMethod, /* xCreate */
fts3termConnectMethod, /* xConnect */
fts3termBestIndexMethod, /* xBestIndex */
fts3termDisconnectMethod, /* xDisconnect */
fts3termDisconnectMethod, /* xDestroy */
fts3termOpenMethod, /* xOpen */
fts3termCloseMethod, /* xClose */
fts3termFilterMethod, /* xFilter */
fts3termNextMethod, /* xNext */
fts3termEofMethod, /* xEof */
fts3termColumnMethod, /* xColumn */
fts3termRowidMethod, /* xRowid */
0, /* xUpdate */
0, /* xBegin */
0, /* xSync */
0, /* xCommit */
0, /* xRollback */
0, /* xFindFunction */
0, /* xRename */
0, /* xSavepoint */
0, /* xRelease */
0, /* xRollbackTo */
0, /* xShadowName */
0 /* xIntegrity */
};
int rc; /* Return code */
rc = sqlite3_create_module(db, "fts4term", &fts3term_module, 0);
return rc;
}
#endif
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
ext/fts3/fts3_test.c
+620
View File
@@ -0,0 +1,620 @@
/*
** 2011 Jun 13
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
** This file is not part of the production FTS code. It is only used for
** testing. It contains a Tcl command that can be used to test if a document
** matches an FTS NEAR expression.
**
** As of March 2012, it also contains a version 1 tokenizer used for testing
** that the sqlite3_tokenizer_module.xLanguage() method is invoked correctly.
*/
#include "tclsqlite.h"
#include <string.h>
#include <assert.h>
#if defined(SQLITE_TEST)
#if defined(SQLITE_ENABLE_FTS3) || defined(SQLITE_ENABLE_FTS4)
/* Required so that the "ifdef SQLITE_ENABLE_FTS3" below works */
#include "fts3Int.h"
#define NM_MAX_TOKEN 12
typedef struct NearPhrase NearPhrase;
typedef struct NearDocument NearDocument;
typedef struct NearToken NearToken;
struct NearDocument {
int nToken; /* Length of token in bytes */
NearToken *aToken; /* Token array */
};
struct NearToken {
int n; /* Length of token in bytes */
const char *z; /* Pointer to token string */
};
struct NearPhrase {
int nNear; /* Preceding NEAR value */
int nToken; /* Number of tokens in this phrase */
NearToken aToken[NM_MAX_TOKEN]; /* Array of tokens in this phrase */
};
static int nm_phrase_match(
NearPhrase *p,
NearToken *aToken
){
int ii;
for(ii=0; ii<p->nToken; ii++){
NearToken *pToken = &p->aToken[ii];
if( pToken->n>0 && pToken->z[pToken->n-1]=='*' ){
if( aToken[ii].n<(pToken->n-1) ) return 0;
if( memcmp(aToken[ii].z, pToken->z, pToken->n-1) ) return 0;
}else{
if( aToken[ii].n!=pToken->n ) return 0;
if( memcmp(aToken[ii].z, pToken->z, pToken->n) ) return 0;
}
}
return 1;
}
static int nm_near_chain(
int iDir, /* Direction to iterate through aPhrase[] */
NearDocument *pDoc, /* Document to match against */
int iPos, /* Position at which iPhrase was found */
int nPhrase, /* Size of phrase array */
NearPhrase *aPhrase, /* Phrase array */
int iPhrase /* Index of phrase found */
){
int iStart;
int iStop;
int ii;
int nNear;
int iPhrase2;
NearPhrase *p;
NearPhrase *pPrev;
assert( iDir==1 || iDir==-1 );
if( iDir==1 ){
if( (iPhrase+1)==nPhrase ) return 1;
nNear = aPhrase[iPhrase+1].nNear;
}else{
if( iPhrase==0 ) return 1;
nNear = aPhrase[iPhrase].nNear;
}
pPrev = &aPhrase[iPhrase];
iPhrase2 = iPhrase+iDir;
p = &aPhrase[iPhrase2];
iStart = iPos - nNear - p->nToken;
iStop = iPos + nNear + pPrev->nToken;
if( iStart<0 ) iStart = 0;
if( iStop > pDoc->nToken - p->nToken ) iStop = pDoc->nToken - p->nToken;
for(ii=iStart; ii<=iStop; ii++){
if( nm_phrase_match(p, &pDoc->aToken[ii]) ){
if( nm_near_chain(iDir, pDoc, ii, nPhrase, aPhrase, iPhrase2) ) return 1;
}
}
return 0;
}
static int nm_match_count(
NearDocument *pDoc, /* Document to match against */
int nPhrase, /* Size of phrase array */
NearPhrase *aPhrase, /* Phrase array */
int iPhrase /* Index of phrase to count matches for */
){
int nOcc = 0;
int ii;
NearPhrase *p = &aPhrase[iPhrase];
for(ii=0; ii<(pDoc->nToken + 1 - p->nToken); ii++){
if( nm_phrase_match(p, &pDoc->aToken[ii]) ){
/* Test forward NEAR chain (i>iPhrase) */
if( 0==nm_near_chain(1, pDoc, ii, nPhrase, aPhrase, iPhrase) ) continue;
/* Test reverse NEAR chain (i<iPhrase) */
if( 0==nm_near_chain(-1, pDoc, ii, nPhrase, aPhrase, iPhrase) ) continue;
/* This is a real match. Increment the counter. */
nOcc++;
}
}
return nOcc;
}
/*
** Tclcmd: fts3_near_match DOCUMENT EXPR ?OPTIONS?
*/
static int SQLITE_TCLAPI fts3_near_match_cmd(
ClientData clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[]
){
int nTotal = 0;
int rc;
int ii;
int nPhrase;
NearPhrase *aPhrase = 0;
NearDocument doc = {0, 0};
Tcl_Obj **apDocToken;
Tcl_Obj *pRet;
Tcl_Obj *pPhrasecount = 0;
Tcl_Obj **apExprToken;
Tcl_Size nExprToken;
Tcl_Size nn;
UNUSED_PARAMETER(clientData);
/* Must have 3 or more arguments. */
if( objc<3 || (objc%2)==0 ){
Tcl_WrongNumArgs(interp, 1, objv, "DOCUMENT EXPR ?OPTION VALUE?...");
rc = TCL_ERROR;
goto near_match_out;
}
for(ii=3; ii<objc; ii+=2){
enum NM_enum { NM_PHRASECOUNTS };
struct TestnmSubcmd {
char *zName;
enum NM_enum eOpt;
} aOpt[] = {
{ "-phrasecountvar", NM_PHRASECOUNTS },
{ 0, 0 }
};
int iOpt;
if( Tcl_GetIndexFromObjStruct(
interp, objv[ii], aOpt, sizeof(aOpt[0]), "option", 0, &iOpt)
){
return TCL_ERROR;
}
switch( aOpt[iOpt].eOpt ){
case NM_PHRASECOUNTS:
pPhrasecount = objv[ii+1];
break;
}
}
rc = Tcl_ListObjGetElements(interp, objv[1], &nn, &apDocToken);
doc.nToken = (int)nn;
if( rc!=TCL_OK ) goto near_match_out;
doc.aToken = (NearToken *)ckalloc(doc.nToken*sizeof(NearToken));
for(ii=0; ii<doc.nToken; ii++){
doc.aToken[ii].z = Tcl_GetStringFromObj(apDocToken[ii], &nn);
doc.aToken[ii].n = (int)nn;
}
rc = Tcl_ListObjGetElements(interp, objv[2], &nExprToken, &apExprToken);
if( rc!=TCL_OK ) goto near_match_out;
nPhrase = (int)(nExprToken + 1) / 2;
aPhrase = (NearPhrase *)ckalloc(nPhrase * sizeof(NearPhrase));
memset(aPhrase, 0, nPhrase * sizeof(NearPhrase));
for(ii=0; ii<nPhrase; ii++){
Tcl_Obj *pPhrase = apExprToken[ii*2];
Tcl_Obj **apToken;
Tcl_Size nToken;
int jj;
rc = Tcl_ListObjGetElements(interp, pPhrase, &nToken, &apToken);
if( rc!=TCL_OK ) goto near_match_out;
if( nToken>NM_MAX_TOKEN ){
Tcl_AppendResult(interp, "Too many tokens in phrase", NULL);
rc = TCL_ERROR;
goto near_match_out;
}
for(jj=0; jj<(int)nToken; jj++){
NearToken *pT = &aPhrase[ii].aToken[jj];
pT->z = Tcl_GetStringFromObj(apToken[jj], &nn);
pT->n = (int)nn;
}
aPhrase[ii].nToken = (int)nToken;
}
for(ii=1; ii<nPhrase; ii++){
Tcl_Obj *pNear = apExprToken[2*ii-1];
int nNear;
rc = Tcl_GetIntFromObj(interp, pNear, &nNear);
if( rc!=TCL_OK ) goto near_match_out;
aPhrase[ii].nNear = nNear;
}
pRet = Tcl_NewObj();
Tcl_IncrRefCount(pRet);
for(ii=0; ii<nPhrase; ii++){
int nOcc = nm_match_count(&doc, nPhrase, aPhrase, ii);
Tcl_ListObjAppendElement(interp, pRet, Tcl_NewIntObj(nOcc));
nTotal += nOcc;
}
if( pPhrasecount ){
Tcl_ObjSetVar2(interp, pPhrasecount, 0, pRet, 0);
}
Tcl_DecrRefCount(pRet);
Tcl_SetObjResult(interp, Tcl_NewBooleanObj(nTotal>0));
near_match_out:
ckfree((char *)aPhrase);
ckfree((char *)doc.aToken);
return rc;
}
/*
** Tclcmd: fts3_configure_incr_load ?CHUNKSIZE THRESHOLD?
**
** Normally, FTS uses hard-coded values to determine the minimum doclist
** size eligible for incremental loading, and the size of the chunks loaded
** when a doclist is incrementally loaded. This command allows the built-in
** values to be overridden for testing purposes.
**
** If present, the first argument is the chunksize in bytes to load doclists
** in. The second argument is the minimum doclist size in bytes to use
** incremental loading with.
**
** Whether or not the arguments are present, this command returns a list of
** two integers - the initial chunksize and threshold when the command is
** invoked. This can be used to restore the default behavior after running
** tests. For example:
**
** # Override incr-load settings for testing:
** set cfg [fts3_configure_incr_load $new_chunksize $new_threshold]
**
** .... run tests ....
**
** # Restore initial incr-load settings:
** eval fts3_configure_incr_load $cfg
*/
static int SQLITE_TCLAPI fts3_configure_incr_load_cmd(
ClientData clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[]
){
#ifdef SQLITE_ENABLE_FTS3
extern int test_fts3_node_chunksize;
extern int test_fts3_node_chunk_threshold;
Tcl_Obj *pRet;
if( objc!=1 && objc!=3 ){
Tcl_WrongNumArgs(interp, 1, objv, "?CHUNKSIZE THRESHOLD?");
return TCL_ERROR;
}
pRet = Tcl_NewObj();
Tcl_IncrRefCount(pRet);
Tcl_ListObjAppendElement(
interp, pRet, Tcl_NewIntObj(test_fts3_node_chunksize));
Tcl_ListObjAppendElement(
interp, pRet, Tcl_NewIntObj(test_fts3_node_chunk_threshold));
if( objc==3 ){
int iArg1;
int iArg2;
if( Tcl_GetIntFromObj(interp, objv[1], &iArg1)
|| Tcl_GetIntFromObj(interp, objv[2], &iArg2)
){
Tcl_DecrRefCount(pRet);
return TCL_ERROR;
}
test_fts3_node_chunksize = iArg1;
test_fts3_node_chunk_threshold = iArg2;
}
Tcl_SetObjResult(interp, pRet);
Tcl_DecrRefCount(pRet);
#endif
UNUSED_PARAMETER(clientData);
return TCL_OK;
}
#ifdef SQLITE_ENABLE_FTS3
/**************************************************************************
** Beginning of test tokenizer code.
**
** For language 0, this tokenizer is similar to the default 'simple'
** tokenizer. For other languages L, the following:
**
** * Odd numbered languages are case-sensitive. Even numbered
** languages are not.
**
** * Language ids 100 or greater are considered an error.
**
** The implementation assumes that the input contains only ASCII characters
** (i.e. those that may be encoded in UTF-8 using a single byte).
*/
typedef struct test_tokenizer {
sqlite3_tokenizer base;
} test_tokenizer;
typedef struct test_tokenizer_cursor {
sqlite3_tokenizer_cursor base;
const char *aInput; /* Input being tokenized */
int nInput; /* Size of the input in bytes */
int iInput; /* Current offset in aInput */
int iToken; /* Index of next token to be returned */
char *aBuffer; /* Buffer containing current token */
int nBuffer; /* Number of bytes allocated at pToken */
int iLangid; /* Configured language id */
} test_tokenizer_cursor;
static int testTokenizerCreate(
int argc, const char * const *argv,
sqlite3_tokenizer **ppTokenizer
){
test_tokenizer *pNew;
UNUSED_PARAMETER(argc);
UNUSED_PARAMETER(argv);
pNew = sqlite3_malloc(sizeof(test_tokenizer));
if( !pNew ) return SQLITE_NOMEM;
memset(pNew, 0, sizeof(test_tokenizer));
*ppTokenizer = (sqlite3_tokenizer *)pNew;
return SQLITE_OK;
}
static int testTokenizerDestroy(sqlite3_tokenizer *pTokenizer){
test_tokenizer *p = (test_tokenizer *)pTokenizer;
sqlite3_free(p);
return SQLITE_OK;
}
static int testTokenizerOpen(
sqlite3_tokenizer *pTokenizer, /* The tokenizer */
const char *pInput, int nBytes, /* String to be tokenized */
sqlite3_tokenizer_cursor **ppCursor /* OUT: Tokenization cursor */
){
int rc = SQLITE_OK; /* Return code */
test_tokenizer_cursor *pCsr; /* New cursor object */
UNUSED_PARAMETER(pTokenizer);
pCsr = (test_tokenizer_cursor *)sqlite3_malloc(sizeof(test_tokenizer_cursor));
if( pCsr==0 ){
rc = SQLITE_NOMEM;
}else{
memset(pCsr, 0, sizeof(test_tokenizer_cursor));
pCsr->aInput = pInput;
if( nBytes<0 ){
pCsr->nInput = (int)strlen(pInput);
}else{
pCsr->nInput = nBytes;
}
}
*ppCursor = (sqlite3_tokenizer_cursor *)pCsr;
return rc;
}
static int testTokenizerClose(sqlite3_tokenizer_cursor *pCursor){
test_tokenizer_cursor *pCsr = (test_tokenizer_cursor *)pCursor;
sqlite3_free(pCsr->aBuffer);
sqlite3_free(pCsr);
return SQLITE_OK;
}
static int testIsTokenChar(char c){
return (c>='a' && c<='z') || (c>='A' && c<='Z');
}
static int testTolower(char c){
char ret = c;
if( ret>='A' && ret<='Z') ret = ret - ('A'-'a');
return ret;
}
static int testTokenizerNext(
sqlite3_tokenizer_cursor *pCursor, /* Cursor returned by testTokenizerOpen */
const char **ppToken, /* OUT: *ppToken is the token text */
int *pnBytes, /* OUT: Number of bytes in token */
int *piStartOffset, /* OUT: Starting offset of token */
int *piEndOffset, /* OUT: Ending offset of token */
int *piPosition /* OUT: Position integer of token */
){
test_tokenizer_cursor *pCsr = (test_tokenizer_cursor *)pCursor;
int rc = SQLITE_OK;
const char *p;
const char *pEnd;
p = &pCsr->aInput[pCsr->iInput];
pEnd = &pCsr->aInput[pCsr->nInput];
/* Skip past any white-space */
assert( p<=pEnd );
while( p<pEnd && testIsTokenChar(*p)==0 ) p++;
if( p==pEnd ){
rc = SQLITE_DONE;
}else{
/* Advance to the end of the token */
const char *pToken = p;
sqlite3_int64 nToken;
while( p<pEnd && testIsTokenChar(*p) ) p++;
nToken = (sqlite3_int64)(p-pToken);
/* Copy the token into the buffer */
if( nToken>pCsr->nBuffer ){
sqlite3_free(pCsr->aBuffer);
pCsr->aBuffer = sqlite3_malloc64(nToken);
}
if( pCsr->aBuffer==0 ){
rc = SQLITE_NOMEM;
}else{
int i;
if( pCsr->iLangid & 0x00000001 ){
for(i=0; i<nToken; i++) pCsr->aBuffer[i] = pToken[i];
}else{
for(i=0; i<nToken; i++) pCsr->aBuffer[i] = (char)testTolower(pToken[i]);
}
pCsr->iToken++;
pCsr->iInput = (int)(p - pCsr->aInput);
*ppToken = pCsr->aBuffer;
*pnBytes = (int)nToken;
*piStartOffset = (int)(pToken - pCsr->aInput);
*piEndOffset = (int)(p - pCsr->aInput);
*piPosition = pCsr->iToken;
}
}
return rc;
}
static int testTokenizerLanguage(
sqlite3_tokenizer_cursor *pCursor,
int iLangid
){
int rc = SQLITE_OK;
test_tokenizer_cursor *pCsr = (test_tokenizer_cursor *)pCursor;
pCsr->iLangid = iLangid;
if( pCsr->iLangid>=100 ){
rc = SQLITE_ERROR;
}
return rc;
}
#endif
static int SQLITE_TCLAPI fts3_test_tokenizer_cmd(
ClientData clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[]
){
#ifdef SQLITE_ENABLE_FTS3
static const sqlite3_tokenizer_module testTokenizerModule = {
1,
testTokenizerCreate,
testTokenizerDestroy,
testTokenizerOpen,
testTokenizerClose,
testTokenizerNext,
testTokenizerLanguage
};
const sqlite3_tokenizer_module *pPtr = &testTokenizerModule;
if( objc!=1 ){
Tcl_WrongNumArgs(interp, 1, objv, "");
return TCL_ERROR;
}
Tcl_SetObjResult(interp, Tcl_NewByteArrayObj(
(const unsigned char *)&pPtr, sizeof(sqlite3_tokenizer_module *)
));
#endif
UNUSED_PARAMETER(clientData);
return TCL_OK;
}
static int SQLITE_TCLAPI fts3_test_varint_cmd(
ClientData clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[]
){
#ifdef SQLITE_ENABLE_FTS3
char aBuf[24];
int rc;
Tcl_WideInt w;
sqlite3_int64 w2;
int nByte, nByte2;
if( objc!=2 ){
Tcl_WrongNumArgs(interp, 1, objv, "INTEGER");
return TCL_ERROR;
}
rc = Tcl_GetWideIntFromObj(interp, objv[1], &w);
if( rc!=TCL_OK ) return rc;
nByte = sqlite3Fts3PutVarint(aBuf, w);
nByte2 = sqlite3Fts3GetVarint(aBuf, &w2);
if( w!=w2 || nByte!=nByte2 ){
char *zErr = sqlite3_mprintf("error testing %lld", w);
Tcl_ResetResult(interp);
Tcl_AppendResult(interp, zErr, NULL);
return TCL_ERROR;
}
if( w<=2147483647 && w>=0 ){
int i;
nByte2 = fts3GetVarint32(aBuf, &i);
if( (int)w!=i || nByte!=nByte2 ){
char *zErr = sqlite3_mprintf("error testing %lld (32-bit)", w);
Tcl_ResetResult(interp);
Tcl_AppendResult(interp, zErr, NULL);
return TCL_ERROR;
}
}
#endif
UNUSED_PARAMETER(clientData);
return TCL_OK;
}
/*
** End of tokenizer code.
**************************************************************************/
/*
** sqlite3_fts3_may_be_corrupt BOOLEAN
**
** Set or clear the global "may-be-corrupt" flag. Return the old value.
*/
static int SQLITE_TCLAPI fts3_may_be_corrupt(
void * clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[]
){
#ifdef SQLITE_DEBUG
int bOld = sqlite3_fts3_may_be_corrupt;
if( objc!=2 && objc!=1 ){
Tcl_WrongNumArgs(interp, 1, objv, "?BOOLEAN?");
return TCL_ERROR;
}
if( objc==2 ){
int bNew;
if( Tcl_GetBooleanFromObj(interp, objv[1], &bNew) ) return TCL_ERROR;
sqlite3_fts3_may_be_corrupt = bNew;
}
Tcl_SetObjResult(interp, Tcl_NewIntObj(bOld));
#endif
return TCL_OK;
}
int Sqlitetestfts3_Init(Tcl_Interp *interp){
Tcl_CreateObjCommand(interp, "fts3_near_match", fts3_near_match_cmd, 0, 0);
Tcl_CreateObjCommand(interp,
"fts3_configure_incr_load", fts3_configure_incr_load_cmd, 0, 0
);
Tcl_CreateObjCommand(
interp, "fts3_test_tokenizer", fts3_test_tokenizer_cmd, 0, 0
);
Tcl_CreateObjCommand(
interp, "fts3_test_varint", fts3_test_varint_cmd, 0, 0
);
Tcl_CreateObjCommand(
interp, "sqlite3_fts3_may_be_corrupt", fts3_may_be_corrupt, 0, 0
);
return TCL_OK;
}
#endif /* SQLITE_ENABLE_FTS3 || SQLITE_ENABLE_FTS4 */
#endif /* ifdef SQLITE_TEST */
ext/fts3/fts3_tokenize_vtab.c
+459
View File
@@ -0,0 +1,459 @@
/*
** 2013 Apr 22
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
** This file contains code for the "fts3tokenize" virtual table module.
** An fts3tokenize virtual table is created as follows:
**
** CREATE VIRTUAL TABLE <tbl> USING fts3tokenize(
** <tokenizer-name>, <arg-1>, ...
** );
**
** The table created has the following schema:
**
** CREATE TABLE <tbl>(input, token, start, end, position)
**
** When queried, the query must include a WHERE clause of type:
**
** input = <string>
**
** The virtual table module tokenizes this <string>, using the FTS3
** tokenizer specified by the arguments to the CREATE VIRTUAL TABLE
** statement and returns one row for each token in the result. With
** fields set as follows:
**
** input: Always set to a copy of <string>
** token: A token from the input.
** start: Byte offset of the token within the input <string>.
** end: Byte offset of the byte immediately following the end of the
** token within the input string.
** pos: Token offset of token within input.
**
*/
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#include <string.h>
#include <assert.h>
typedef struct Fts3tokTable Fts3tokTable;
typedef struct Fts3tokCursor Fts3tokCursor;
/*
** Virtual table structure.
*/
struct Fts3tokTable {
sqlite3_vtab base; /* Base class used by SQLite core */
const sqlite3_tokenizer_module *pMod;
sqlite3_tokenizer *pTok;
};
/*
** Virtual table cursor structure.
*/
struct Fts3tokCursor {
sqlite3_vtab_cursor base; /* Base class used by SQLite core */
char *zInput; /* Input string */
sqlite3_tokenizer_cursor *pCsr; /* Cursor to iterate through zInput */
int iRowid; /* Current 'rowid' value */
const char *zToken; /* Current 'token' value */
int nToken; /* Size of zToken in bytes */
int iStart; /* Current 'start' value */
int iEnd; /* Current 'end' value */
int iPos; /* Current 'pos' value */
};
/*
** Query FTS for the tokenizer implementation named zName.
*/
static int fts3tokQueryTokenizer(
Fts3Hash *pHash,
const char *zName,
const sqlite3_tokenizer_module **pp,
char **pzErr
){
sqlite3_tokenizer_module *p;
int nName = (int)strlen(zName);
p = (sqlite3_tokenizer_module *)sqlite3Fts3HashFind(pHash, zName, nName+1);
if( !p ){
sqlite3Fts3ErrMsg(pzErr, "unknown tokenizer: %s", zName);
return SQLITE_ERROR;
}
*pp = p;
return SQLITE_OK;
}
/*
** The second argument, argv[], is an array of pointers to nul-terminated
** strings. This function makes a copy of the array and strings into a
** single block of memory. It then dequotes any of the strings that appear
** to be quoted.
**
** If successful, output parameter *pazDequote is set to point at the
** array of dequoted strings and SQLITE_OK is returned. The caller is
** responsible for eventually calling sqlite3_free() to free the array
** in this case. Or, if an error occurs, an SQLite error code is returned.
** The final value of *pazDequote is undefined in this case.
*/
static int fts3tokDequoteArray(
int argc, /* Number of elements in argv[] */
const char * const *argv, /* Input array */
char ***pazDequote /* Output array */
){
int rc = SQLITE_OK; /* Return code */
if( argc==0 ){
*pazDequote = 0;
}else{
int i;
int nByte = 0;
char **azDequote;
for(i=0; i<argc; i++){
nByte += (int)(strlen(argv[i]) + 1);
}
*pazDequote = azDequote = sqlite3_malloc64(sizeof(char *)*argc + nByte);
if( azDequote==0 ){
rc = SQLITE_NOMEM;
}else{
char *pSpace = (char *)&azDequote[argc];
for(i=0; i<argc; i++){
int n = (int)strlen(argv[i]);
azDequote[i] = pSpace;
memcpy(pSpace, argv[i], n+1);
sqlite3Fts3Dequote(pSpace);
pSpace += (n+1);
}
}
}
return rc;
}
/*
** Schema of the tokenizer table.
*/
#define FTS3_TOK_SCHEMA "CREATE TABLE x(input, token, start, end, position)"
/*
** This function does all the work for both the xConnect and xCreate methods.
** These tables have no persistent representation of their own, so xConnect
** and xCreate are identical operations.
**
** argv[0]: module name
** argv[1]: database name
** argv[2]: table name
** argv[3]: first argument (tokenizer name)
*/
static int fts3tokConnectMethod(
sqlite3 *db, /* Database connection */
void *pHash, /* Hash table of tokenizers */
int argc, /* Number of elements in argv array */
const char * const *argv, /* xCreate/xConnect argument array */
sqlite3_vtab **ppVtab, /* OUT: New sqlite3_vtab object */
char **pzErr /* OUT: sqlite3_malloc'd error message */
){
Fts3tokTable *pTab = 0;
const sqlite3_tokenizer_module *pMod = 0;
sqlite3_tokenizer *pTok = 0;
int rc;
char **azDequote = 0;
int nDequote;
rc = sqlite3_declare_vtab(db, FTS3_TOK_SCHEMA);
if( rc!=SQLITE_OK ) return rc;
nDequote = argc-3;
rc = fts3tokDequoteArray(nDequote, &argv[3], &azDequote);
if( rc==SQLITE_OK ){
const char *zModule;
if( nDequote<1 ){
zModule = "simple";
}else{
zModule = azDequote[0];
}
rc = fts3tokQueryTokenizer((Fts3Hash*)pHash, zModule, &pMod, pzErr);
}
assert( (rc==SQLITE_OK)==(pMod!=0) );
if( rc==SQLITE_OK ){
const char * const *azArg = 0;
if( nDequote>1 ) azArg = (const char * const *)&azDequote[1];
rc = pMod->xCreate((nDequote>1 ? nDequote-1 : 0), azArg, &pTok);
}
if( rc==SQLITE_OK ){
pTab = (Fts3tokTable *)sqlite3_malloc(sizeof(Fts3tokTable));
if( pTab==0 ){
rc = SQLITE_NOMEM;
}
}
if( rc==SQLITE_OK ){
memset(pTab, 0, sizeof(Fts3tokTable));
pTab->pMod = pMod;
pTab->pTok = pTok;
*ppVtab = &pTab->base;
}else{
if( pTok ){
pMod->xDestroy(pTok);
}
}
sqlite3_free(azDequote);
return rc;
}
/*
** This function does the work for both the xDisconnect and xDestroy methods.
** These tables have no persistent representation of their own, so xDisconnect
** and xDestroy are identical operations.
*/
static int fts3tokDisconnectMethod(sqlite3_vtab *pVtab){
Fts3tokTable *pTab = (Fts3tokTable *)pVtab;
pTab->pMod->xDestroy(pTab->pTok);
sqlite3_free(pTab);
return SQLITE_OK;
}
/*
** xBestIndex - Analyze a WHERE and ORDER BY clause.
*/
static int fts3tokBestIndexMethod(
sqlite3_vtab *pVTab,
sqlite3_index_info *pInfo
){
int i;
UNUSED_PARAMETER(pVTab);
for(i=0; i<pInfo->nConstraint; i++){
if( pInfo->aConstraint[i].usable
&& pInfo->aConstraint[i].iColumn==0
&& pInfo->aConstraint[i].op==SQLITE_INDEX_CONSTRAINT_EQ
){
pInfo->idxNum = 1;
pInfo->aConstraintUsage[i].argvIndex = 1;
pInfo->aConstraintUsage[i].omit = 1;
pInfo->estimatedCost = 1;
return SQLITE_OK;
}
}
pInfo->idxNum = 0;
assert( pInfo->estimatedCost>1000000.0 );
return SQLITE_OK;
}
/*
** xOpen - Open a cursor.
*/
static int fts3tokOpenMethod(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCsr){
Fts3tokCursor *pCsr;
UNUSED_PARAMETER(pVTab);
pCsr = (Fts3tokCursor *)sqlite3_malloc(sizeof(Fts3tokCursor));
if( pCsr==0 ){
return SQLITE_NOMEM;
}
memset(pCsr, 0, sizeof(Fts3tokCursor));
*ppCsr = (sqlite3_vtab_cursor *)pCsr;
return SQLITE_OK;
}
/*
** Reset the tokenizer cursor passed as the only argument. As if it had
** just been returned by fts3tokOpenMethod().
*/
static void fts3tokResetCursor(Fts3tokCursor *pCsr){
if( pCsr->pCsr ){
Fts3tokTable *pTab = (Fts3tokTable *)(pCsr->base.pVtab);
pTab->pMod->xClose(pCsr->pCsr);
pCsr->pCsr = 0;
}
sqlite3_free(pCsr->zInput);
pCsr->zInput = 0;
pCsr->zToken = 0;
pCsr->nToken = 0;
pCsr->iStart = 0;
pCsr->iEnd = 0;
pCsr->iPos = 0;
pCsr->iRowid = 0;
}
/*
** xClose - Close a cursor.
*/
static int fts3tokCloseMethod(sqlite3_vtab_cursor *pCursor){
Fts3tokCursor *pCsr = (Fts3tokCursor *)pCursor;
fts3tokResetCursor(pCsr);
sqlite3_free(pCsr);
return SQLITE_OK;
}
/*
** xNext - Advance the cursor to the next row, if any.
*/
static int fts3tokNextMethod(sqlite3_vtab_cursor *pCursor){
Fts3tokCursor *pCsr = (Fts3tokCursor *)pCursor;
Fts3tokTable *pTab = (Fts3tokTable *)(pCursor->pVtab);
int rc; /* Return code */
pCsr->iRowid++;
rc = pTab->pMod->xNext(pCsr->pCsr,
&pCsr->zToken, &pCsr->nToken,
&pCsr->iStart, &pCsr->iEnd, &pCsr->iPos
);
if( rc!=SQLITE_OK ){
fts3tokResetCursor(pCsr);
if( rc==SQLITE_DONE ) rc = SQLITE_OK;
}
return rc;
}
/*
** xFilter - Initialize a cursor to point at the start of its data.
*/
static int fts3tokFilterMethod(
sqlite3_vtab_cursor *pCursor, /* The cursor used for this query */
int idxNum, /* Strategy index */
const char *idxStr, /* Unused */
int nVal, /* Number of elements in apVal */
sqlite3_value **apVal /* Arguments for the indexing scheme */
){
int rc = SQLITE_ERROR;
Fts3tokCursor *pCsr = (Fts3tokCursor *)pCursor;
Fts3tokTable *pTab = (Fts3tokTable *)(pCursor->pVtab);
UNUSED_PARAMETER(idxStr);
UNUSED_PARAMETER(nVal);
fts3tokResetCursor(pCsr);
if( idxNum==1 ){
const char *zByte = (const char *)sqlite3_value_text(apVal[0]);
sqlite3_int64 nByte = sqlite3_value_bytes(apVal[0]);
pCsr->zInput = sqlite3_malloc64(nByte+1);
if( pCsr->zInput==0 ){
rc = SQLITE_NOMEM;
}else{
if( nByte>0 ) memcpy(pCsr->zInput, zByte, nByte);
pCsr->zInput[nByte] = 0;
rc = pTab->pMod->xOpen(pTab->pTok, pCsr->zInput, nByte, &pCsr->pCsr);
if( rc==SQLITE_OK ){
pCsr->pCsr->pTokenizer = pTab->pTok;
}
}
}
if( rc!=SQLITE_OK ) return rc;
return fts3tokNextMethod(pCursor);
}
/*
** xEof - Return true if the cursor is at EOF, or false otherwise.
*/
static int fts3tokEofMethod(sqlite3_vtab_cursor *pCursor){
Fts3tokCursor *pCsr = (Fts3tokCursor *)pCursor;
return (pCsr->zToken==0);
}
/*
** xColumn - Return a column value.
*/
static int fts3tokColumnMethod(
sqlite3_vtab_cursor *pCursor, /* Cursor to retrieve value from */
sqlite3_context *pCtx, /* Context for sqlite3_result_xxx() calls */
int iCol /* Index of column to read value from */
){
Fts3tokCursor *pCsr = (Fts3tokCursor *)pCursor;
/* CREATE TABLE x(input, token, start, end, position) */
switch( iCol ){
case 0:
sqlite3_result_text(pCtx, pCsr->zInput, -1, SQLITE_TRANSIENT);
break;
case 1:
sqlite3_result_text(pCtx, pCsr->zToken, pCsr->nToken, SQLITE_TRANSIENT);
break;
case 2:
sqlite3_result_int(pCtx, pCsr->iStart);
break;
case 3:
sqlite3_result_int(pCtx, pCsr->iEnd);
break;
default:
assert( iCol==4 );
sqlite3_result_int(pCtx, pCsr->iPos);
break;
}
return SQLITE_OK;
}
/*
** xRowid - Return the current rowid for the cursor.
*/
static int fts3tokRowidMethod(
sqlite3_vtab_cursor *pCursor, /* Cursor to retrieve value from */
sqlite_int64 *pRowid /* OUT: Rowid value */
){
Fts3tokCursor *pCsr = (Fts3tokCursor *)pCursor;
*pRowid = (sqlite3_int64)pCsr->iRowid;
return SQLITE_OK;
}
/*
** Register the fts3tok module with database connection db. Return SQLITE_OK
** if successful or an error code if sqlite3_create_module() fails.
*/
int sqlite3Fts3InitTok(sqlite3 *db, Fts3Hash *pHash, void(*xDestroy)(void*)){
static const sqlite3_module fts3tok_module = {
0, /* iVersion */
fts3tokConnectMethod, /* xCreate */
fts3tokConnectMethod, /* xConnect */
fts3tokBestIndexMethod, /* xBestIndex */
fts3tokDisconnectMethod, /* xDisconnect */
fts3tokDisconnectMethod, /* xDestroy */
fts3tokOpenMethod, /* xOpen */
fts3tokCloseMethod, /* xClose */
fts3tokFilterMethod, /* xFilter */
fts3tokNextMethod, /* xNext */
fts3tokEofMethod, /* xEof */
fts3tokColumnMethod, /* xColumn */
fts3tokRowidMethod, /* xRowid */
0, /* xUpdate */
0, /* xBegin */
0, /* xSync */
0, /* xCommit */
0, /* xRollback */
0, /* xFindFunction */
0, /* xRename */
0, /* xSavepoint */
0, /* xRelease */
0, /* xRollbackTo */
0, /* xShadowName */
0 /* xIntegrity */
};
int rc; /* Return code */
rc = sqlite3_create_module_v2(
db, "fts3tokenize", &fts3tok_module, (void*)pHash, xDestroy
);
return rc;
}
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
ext/fts3/fts3_tokenizer.c
+516
View File
@@ -0,0 +1,516 @@
/*
** 2007 June 22
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
** This is part of an SQLite module implementing full-text search.
** This particular file implements the generic tokenizer interface.
*/
/*
** The code in this file is only compiled if:
**
** * The FTS3 module is being built as an extension
** (in which case SQLITE_CORE is not defined), or
**
** * The FTS3 module is being built into the core of
** SQLite (in which case SQLITE_ENABLE_FTS3 is defined).
*/
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#include <assert.h>
#include <string.h>
/*
** Return true if the two-argument version of fts3_tokenizer()
** has been activated via a prior call to sqlite3_db_config(db,
** SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER, 1, 0);
*/
static int fts3TokenizerEnabled(sqlite3_context *context){
sqlite3 *db = sqlite3_context_db_handle(context);
int isEnabled = 0;
sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER,-1,&isEnabled);
return isEnabled;
}
/*
** Implementation of the SQL scalar function for accessing the underlying
** hash table. This function may be called as follows:
**
** SELECT <function-name>(<key-name>);
** SELECT <function-name>(<key-name>, <pointer>);
**
** where <function-name> is the name passed as the second argument
** to the sqlite3Fts3InitHashTable() function (e.g. 'fts3_tokenizer').
**
** If the <pointer> argument is specified, it must be a blob value
** containing a pointer to be stored as the hash data corresponding
** to the string <key-name>. If <pointer> is not specified, then
** the string <key-name> must already exist in the has table. Otherwise,
** an error is returned.
**
** Whether or not the <pointer> argument is specified, the value returned
** is a blob containing the pointer stored as the hash data corresponding
** to string <key-name> (after the hash-table is updated, if applicable).
*/
static void fts3TokenizerFunc(
sqlite3_context *context,
int argc,
sqlite3_value **argv
){
Fts3Hash *pHash;
void *pPtr = 0;
const unsigned char *zName;
int nName;
assert( argc==1 || argc==2 );
pHash = (Fts3Hash *)sqlite3_user_data(context);
zName = sqlite3_value_text(argv[0]);
nName = sqlite3_value_bytes(argv[0])+1;
if( argc==2 ){
if( fts3TokenizerEnabled(context) || sqlite3_value_frombind(argv[1]) ){
void *pOld;
int n = sqlite3_value_bytes(argv[1]);
if( zName==0 || n!=sizeof(pPtr) ){
sqlite3_result_error(context, "argument type mismatch", -1);
return;
}
pPtr = *(void **)sqlite3_value_blob(argv[1]);
pOld = sqlite3Fts3HashInsert(pHash, (void *)zName, nName, pPtr);
if( pOld==pPtr ){
sqlite3_result_error(context, "out of memory", -1);
}
}else{
sqlite3_result_error(context, "fts3tokenize disabled", -1);
return;
}
}else{
if( zName ){
pPtr = sqlite3Fts3HashFind(pHash, zName, nName);
}
if( !pPtr ){
char *zErr = sqlite3_mprintf("unknown tokenizer: %s", zName);
sqlite3_result_error(context, zErr, -1);
sqlite3_free(zErr);
return;
}
}
if( fts3TokenizerEnabled(context) || sqlite3_value_frombind(argv[0]) ){
sqlite3_result_blob(context, (void *)&pPtr, sizeof(pPtr), SQLITE_TRANSIENT);
}
}
int sqlite3Fts3IsIdChar(char c){
static const char isFtsIdChar[] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0x */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 1x */
0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 2x */
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, /* 3x */
0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, /* 4x */
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, /* 5x */
0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, /* 6x */
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, /* 7x */
};
return (c&0x80 || isFtsIdChar[(int)(c)]);
}
const char *sqlite3Fts3NextToken(const char *zStr, int *pn){
const char *z1;
const char *z2 = 0;
/* Find the start of the next token. */
z1 = zStr;
while( z2==0 ){
char c = *z1;
switch( c ){
case '\0': return 0; /* No more tokens here */
case '\'':
case '"':
case '`': {
z2 = z1;
while( *++z2 && (*z2!=c || *++z2==c) );
break;
}
case '[':
z2 = &z1[1];
while( *z2 && z2[0]!=']' ) z2++;
if( *z2 ) z2++;
break;
default:
if( sqlite3Fts3IsIdChar(*z1) ){
z2 = &z1[1];
while( sqlite3Fts3IsIdChar(*z2) ) z2++;
}else{
z1++;
}
}
}
*pn = (int)(z2-z1);
return z1;
}
int sqlite3Fts3InitTokenizer(
Fts3Hash *pHash, /* Tokenizer hash table */
const char *zArg, /* Tokenizer name */
sqlite3_tokenizer **ppTok, /* OUT: Tokenizer (if applicable) */
char **pzErr /* OUT: Set to malloced error message */
){
int rc;
char *z = (char *)zArg;
int n = 0;
char *zCopy;
char *zEnd; /* Pointer to nul-term of zCopy */
sqlite3_tokenizer_module *m;
zCopy = sqlite3_mprintf("%s", zArg);
if( !zCopy ) return SQLITE_NOMEM;
zEnd = &zCopy[strlen(zCopy)];
z = (char *)sqlite3Fts3NextToken(zCopy, &n);
if( z==0 ){
assert( n==0 );
z = zCopy;
}
z[n] = '\0';
sqlite3Fts3Dequote(z);
m = (sqlite3_tokenizer_module *)sqlite3Fts3HashFind(pHash,z,(int)strlen(z)+1);
if( !m ){
sqlite3Fts3ErrMsg(pzErr, "unknown tokenizer: %s", z);
rc = SQLITE_ERROR;
}else{
char const **aArg = 0;
int iArg = 0;
z = &z[n+1];
while( z<zEnd && (NULL!=(z = (char *)sqlite3Fts3NextToken(z, &n))) ){
sqlite3_int64 nNew = sizeof(char *)*(iArg+1);
char const **aNew = (const char **)sqlite3_realloc64((void *)aArg, nNew);
if( !aNew ){
sqlite3_free(zCopy);
sqlite3_free((void *)aArg);
return SQLITE_NOMEM;
}
aArg = aNew;
aArg[iArg++] = z;
z[n] = '\0';
sqlite3Fts3Dequote(z);
z = &z[n+1];
}
rc = m->xCreate(iArg, aArg, ppTok);
assert( rc!=SQLITE_OK || *ppTok );
if( rc!=SQLITE_OK ){
sqlite3Fts3ErrMsg(pzErr, "unknown tokenizer");
}else{
(*ppTok)->pModule = m;
}
sqlite3_free((void *)aArg);
}
sqlite3_free(zCopy);
return rc;
}
#ifdef SQLITE_TEST
#include "tclsqlite.h"
#include <string.h>
/*
** Implementation of a special SQL scalar function for testing tokenizers
** designed to be used in concert with the Tcl testing framework. This
** function must be called with two or more arguments:
**
** SELECT <function-name>(<key-name>, ..., <input-string>);
**
** where <function-name> is the name passed as the second argument
** to the sqlite3Fts3InitHashTable() function (e.g. 'fts3_tokenizer')
** concatenated with the string '_test' (e.g. 'fts3_tokenizer_test').
**
** The return value is a string that may be interpreted as a Tcl
** list. For each token in the <input-string>, three elements are
** added to the returned list. The first is the token position, the
** second is the token text (folded, stemmed, etc.) and the third is the
** substring of <input-string> associated with the token. For example,
** using the built-in "simple" tokenizer:
**
** SELECT fts_tokenizer_test('simple', 'I don't see how');
**
** will return the string:
**
** "{0 i I 1 dont don't 2 see see 3 how how}"
**
*/
static void testFunc(
sqlite3_context *context,
int argc,
sqlite3_value **argv
){
Fts3Hash *pHash;
sqlite3_tokenizer_module *p;
sqlite3_tokenizer *pTokenizer = 0;
sqlite3_tokenizer_cursor *pCsr = 0;
const char *zErr = 0;
const char *zName;
int nName;
const char *zInput;
int nInput;
const char *azArg[64];
const char *zToken;
int nToken = 0;
int iStart = 0;
int iEnd = 0;
int iPos = 0;
int i;
Tcl_Obj *pRet;
if( argc<2 ){
sqlite3_result_error(context, "insufficient arguments", -1);
return;
}
nName = sqlite3_value_bytes(argv[0]);
zName = (const char *)sqlite3_value_text(argv[0]);
nInput = sqlite3_value_bytes(argv[argc-1]);
zInput = (const char *)sqlite3_value_text(argv[argc-1]);
pHash = (Fts3Hash *)sqlite3_user_data(context);
p = (sqlite3_tokenizer_module *)sqlite3Fts3HashFind(pHash, zName, nName+1);
if( !p ){
char *zErr2 = sqlite3_mprintf("unknown tokenizer: %s", zName);
sqlite3_result_error(context, zErr2, -1);
sqlite3_free(zErr2);
return;
}
pRet = Tcl_NewObj();
Tcl_IncrRefCount(pRet);
for(i=1; i<argc-1; i++){
azArg[i-1] = (const char *)sqlite3_value_text(argv[i]);
}
if( SQLITE_OK!=p->xCreate(argc-2, azArg, &pTokenizer) ){
zErr = "error in xCreate()";
goto finish;
}
pTokenizer->pModule = p;
if( sqlite3Fts3OpenTokenizer(pTokenizer, 0, zInput, nInput, &pCsr) ){
zErr = "error in xOpen()";
goto finish;
}
while( SQLITE_OK==p->xNext(pCsr, &zToken, &nToken, &iStart, &iEnd, &iPos) ){
Tcl_ListObjAppendElement(0, pRet, Tcl_NewIntObj(iPos));
Tcl_ListObjAppendElement(0, pRet, Tcl_NewStringObj(zToken, nToken));
zToken = &zInput[iStart];
nToken = iEnd-iStart;
Tcl_ListObjAppendElement(0, pRet, Tcl_NewStringObj(zToken, nToken));
}
if( SQLITE_OK!=p->xClose(pCsr) ){
zErr = "error in xClose()";
goto finish;
}
if( SQLITE_OK!=p->xDestroy(pTokenizer) ){
zErr = "error in xDestroy()";
goto finish;
}
finish:
if( zErr ){
sqlite3_result_error(context, zErr, -1);
}else{
sqlite3_result_text(context, Tcl_GetString(pRet), -1, SQLITE_TRANSIENT);
}
Tcl_DecrRefCount(pRet);
}
static
int registerTokenizer(
sqlite3 *db,
char *zName,
const sqlite3_tokenizer_module *p
){
int rc;
sqlite3_stmt *pStmt;
const char zSql[] = "SELECT fts3_tokenizer(?, ?)";
rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
if( rc!=SQLITE_OK ){
return rc;
}
sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
sqlite3_bind_blob(pStmt, 2, &p, sizeof(p), SQLITE_STATIC);
sqlite3_step(pStmt);
return sqlite3_finalize(pStmt);
}
static
int queryTokenizer(
sqlite3 *db,
char *zName,
const sqlite3_tokenizer_module **pp
){
int rc;
sqlite3_stmt *pStmt;
const char zSql[] = "SELECT fts3_tokenizer(?)";
*pp = 0;
rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
if( rc!=SQLITE_OK ){
return rc;
}
sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
if( SQLITE_ROW==sqlite3_step(pStmt) ){
if( sqlite3_column_type(pStmt, 0)==SQLITE_BLOB
&& sqlite3_column_bytes(pStmt, 0)==sizeof(*pp)
){
memcpy((void *)pp, sqlite3_column_blob(pStmt, 0), sizeof(*pp));
}
}
return sqlite3_finalize(pStmt);
}
void sqlite3Fts3SimpleTokenizerModule(sqlite3_tokenizer_module const**ppModule);
/*
** Implementation of the scalar function fts3_tokenizer_internal_test().
** This function is used for testing only, it is not included in the
** build unless SQLITE_TEST is defined.
**
** The purpose of this is to test that the fts3_tokenizer() function
** can be used as designed by the C-code in the queryTokenizer and
** registerTokenizer() functions above. These two functions are repeated
** in the README.tokenizer file as an example, so it is important to
** test them.
**
** To run the tests, evaluate the fts3_tokenizer_internal_test() scalar
** function with no arguments. An assert() will fail if a problem is
** detected. i.e.:
**
** SELECT fts3_tokenizer_internal_test();
**
*/
static void intTestFunc(
sqlite3_context *context,
int argc,
sqlite3_value **argv
){
int rc;
const sqlite3_tokenizer_module *p1;
const sqlite3_tokenizer_module *p2;
sqlite3 *db = (sqlite3 *)sqlite3_user_data(context);
UNUSED_PARAMETER(argc);
UNUSED_PARAMETER(argv);
/* Test the query function */
sqlite3Fts3SimpleTokenizerModule(&p1);
rc = queryTokenizer(db, "simple", &p2);
assert( rc==SQLITE_OK );
assert( p1==p2 );
rc = queryTokenizer(db, "nosuchtokenizer", &p2);
assert( rc==SQLITE_ERROR );
assert( p2==0 );
assert( 0==strcmp(sqlite3_errmsg(db), "unknown tokenizer: nosuchtokenizer") );
/* Test the storage function */
if( fts3TokenizerEnabled(context) ){
rc = registerTokenizer(db, "nosuchtokenizer", p1);
assert( rc==SQLITE_OK );
rc = queryTokenizer(db, "nosuchtokenizer", &p2);
assert( rc==SQLITE_OK );
assert( p2==p1 );
}
sqlite3_result_text(context, "ok", -1, SQLITE_STATIC);
}
#endif
/*
** Set up SQL objects in database db used to access the contents of
** the hash table pointed to by argument pHash. The hash table must
** been initialized to use string keys, and to take a private copy
** of the key when a value is inserted. i.e. by a call similar to:
**
** sqlite3Fts3HashInit(pHash, FTS3_HASH_STRING, 1);
**
** This function adds a scalar function (see header comment above
** fts3TokenizerFunc() in this file for details) and, if ENABLE_TABLE is
** defined at compilation time, a temporary virtual table (see header
** comment above struct HashTableVtab) to the database schema. Both
** provide read/write access to the contents of *pHash.
**
** The third argument to this function, zName, is used as the name
** of both the scalar and, if created, the virtual table.
*/
int sqlite3Fts3InitHashTable(
sqlite3 *db,
Fts3Hash *pHash,
const char *zName
){
int rc = SQLITE_OK;
void *p = (void *)pHash;
const int any = SQLITE_UTF8|SQLITE_DIRECTONLY;
#ifdef SQLITE_TEST
char *zTest = 0;
char *zTest2 = 0;
void *pdb = (void *)db;
zTest = sqlite3_mprintf("%s_test", zName);
zTest2 = sqlite3_mprintf("%s_internal_test", zName);
if( !zTest || !zTest2 ){
rc = SQLITE_NOMEM;
}
#endif
if( SQLITE_OK==rc ){
rc = sqlite3_create_function(db, zName, 1, any, p, fts3TokenizerFunc, 0, 0);
}
if( SQLITE_OK==rc ){
rc = sqlite3_create_function(db, zName, 2, any, p, fts3TokenizerFunc, 0, 0);
}
#ifdef SQLITE_TEST
if( SQLITE_OK==rc ){
rc = sqlite3_create_function(db, zTest, -1, any, p, testFunc, 0, 0);
}
if( SQLITE_OK==rc ){
rc = sqlite3_create_function(db, zTest2, 0, any, pdb, intTestFunc, 0, 0);
}
#endif
#ifdef SQLITE_TEST
sqlite3_free(zTest);
sqlite3_free(zTest2);
#endif
return rc;
}
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
ext/fts3/fts3_tokenizer.h
+161
View File
@@ -0,0 +1,161 @@
/*
** 2006 July 10
**
** The author disclaims copyright to this source code.
**
*************************************************************************
** Defines the interface to tokenizers used by fulltext-search. There
** are three basic components:
**
** sqlite3_tokenizer_module is a singleton defining the tokenizer
** interface functions. This is essentially the class structure for
** tokenizers.
**
** sqlite3_tokenizer is used to define a particular tokenizer, perhaps
** including customization information defined at creation time.
**
** sqlite3_tokenizer_cursor is generated by a tokenizer to generate
** tokens from a particular input.
*/
#ifndef _FTS3_TOKENIZER_H_
#define _FTS3_TOKENIZER_H_
/* TODO(shess) Only used for SQLITE_OK and SQLITE_DONE at this time.
** If tokenizers are to be allowed to call sqlite3_*() functions, then
** we will need a way to register the API consistently.
*/
#include "sqlite3.h"
/*
** Structures used by the tokenizer interface. When a new tokenizer
** implementation is registered, the caller provides a pointer to
** an sqlite3_tokenizer_module containing pointers to the callback
** functions that make up an implementation.
**
** When an fts3 table is created, it passes any arguments passed to
** the tokenizer clause of the CREATE VIRTUAL TABLE statement to the
** sqlite3_tokenizer_module.xCreate() function of the requested tokenizer
** implementation. The xCreate() function in turn returns an
** sqlite3_tokenizer structure representing the specific tokenizer to
** be used for the fts3 table (customized by the tokenizer clause arguments).
**
** To tokenize an input buffer, the sqlite3_tokenizer_module.xOpen()
** method is called. It returns an sqlite3_tokenizer_cursor object
** that may be used to tokenize a specific input buffer based on
** the tokenization rules supplied by a specific sqlite3_tokenizer
** object.
*/
typedef struct sqlite3_tokenizer_module sqlite3_tokenizer_module;
typedef struct sqlite3_tokenizer sqlite3_tokenizer;
typedef struct sqlite3_tokenizer_cursor sqlite3_tokenizer_cursor;
struct sqlite3_tokenizer_module {
/*
** Structure version. Should always be set to 0 or 1.
*/
int iVersion;
/*
** Create a new tokenizer. The values in the argv[] array are the
** arguments passed to the "tokenizer" clause of the CREATE VIRTUAL
** TABLE statement that created the fts3 table. For example, if
** the following SQL is executed:
**
** CREATE .. USING fts3( ... , tokenizer <tokenizer-name> arg1 arg2)
**
** then argc is set to 2, and the argv[] array contains pointers
** to the strings "arg1" and "arg2".
**
** This method should return either SQLITE_OK (0), or an SQLite error
** code. If SQLITE_OK is returned, then *ppTokenizer should be set
** to point at the newly created tokenizer structure. The generic
** sqlite3_tokenizer.pModule variable should not be initialized by
** this callback. The caller will do so.
*/
int (*xCreate)(
int argc, /* Size of argv array */
const char *const*argv, /* Tokenizer argument strings */
sqlite3_tokenizer **ppTokenizer /* OUT: Created tokenizer */
);
/*
** Destroy an existing tokenizer. The fts3 module calls this method
** exactly once for each successful call to xCreate().
*/
int (*xDestroy)(sqlite3_tokenizer *pTokenizer);
/*
** Create a tokenizer cursor to tokenize an input buffer. The caller
** is responsible for ensuring that the input buffer remains valid
** until the cursor is closed (using the xClose() method).
*/
int (*xOpen)(
sqlite3_tokenizer *pTokenizer, /* Tokenizer object */
const char *pInput, int nBytes, /* Input buffer */
sqlite3_tokenizer_cursor **ppCursor /* OUT: Created tokenizer cursor */
);
/*
** Destroy an existing tokenizer cursor. The fts3 module calls this
** method exactly once for each successful call to xOpen().
*/
int (*xClose)(sqlite3_tokenizer_cursor *pCursor);
/*
** Retrieve the next token from the tokenizer cursor pCursor. This
** method should either return SQLITE_OK and set the values of the
** "OUT" variables identified below, or SQLITE_DONE to indicate that
** the end of the buffer has been reached, or an SQLite error code.
**
** *ppToken should be set to point at a buffer containing the
** normalized version of the token (i.e. after any case-folding and/or
** stemming has been performed). *pnBytes should be set to the length
** of this buffer in bytes. The input text that generated the token is
** identified by the byte offsets returned in *piStartOffset and
** *piEndOffset. *piStartOffset should be set to the index of the first
** byte of the token in the input buffer. *piEndOffset should be set
** to the index of the first byte just past the end of the token in
** the input buffer.
**
** The buffer *ppToken is set to point at is managed by the tokenizer
** implementation. It is only required to be valid until the next call
** to xNext() or xClose().
*/
/* TODO(shess) current implementation requires pInput to be
** nul-terminated. This should either be fixed, or pInput/nBytes
** should be converted to zInput.
*/
int (*xNext)(
sqlite3_tokenizer_cursor *pCursor, /* Tokenizer cursor */
const char **ppToken, int *pnBytes, /* OUT: Normalized text for token */
int *piStartOffset, /* OUT: Byte offset of token in input buffer */
int *piEndOffset, /* OUT: Byte offset of end of token in input buffer */
int *piPosition /* OUT: Number of tokens returned before this one */
);
/***********************************************************************
** Methods below this point are only available if iVersion>=1.
*/
/*
** Configure the language id of a tokenizer cursor.
*/
int (*xLanguageid)(sqlite3_tokenizer_cursor *pCsr, int iLangid);
};
struct sqlite3_tokenizer {
const sqlite3_tokenizer_module *pModule; /* The module for this tokenizer */
/* Tokenizer implementations will typically add additional fields */
};
struct sqlite3_tokenizer_cursor {
sqlite3_tokenizer *pTokenizer; /* Tokenizer for this cursor. */
/* Tokenizer implementations will typically add additional fields */
};
int fts3_global_term_cnt(int iTerm, int iCol);
int fts3_term_cnt(int iTerm, int iCol);
#endif /* _FTS3_TOKENIZER_H_ */
ext/fts3/fts3_tokenizer1.c
+234
View File
@@ -0,0 +1,234 @@
/*
** 2006 Oct 10
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
** Implementation of the "simple" full-text-search tokenizer.
*/
/*
** The code in this file is only compiled if:
**
** * The FTS3 module is being built as an extension
** (in which case SQLITE_CORE is not defined), or
**
** * The FTS3 module is being built into the core of
** SQLite (in which case SQLITE_ENABLE_FTS3 is defined).
*/
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "fts3_tokenizer.h"
typedef struct simple_tokenizer {
sqlite3_tokenizer base;
char delim[128]; /* flag ASCII delimiters */
} simple_tokenizer;
typedef struct simple_tokenizer_cursor {
sqlite3_tokenizer_cursor base;
const char *pInput; /* input we are tokenizing */
int nBytes; /* size of the input */
int iOffset; /* current position in pInput */
int iToken; /* index of next token to be returned */
char *pToken; /* storage for current token */
int nTokenAllocated; /* space allocated to zToken buffer */
} simple_tokenizer_cursor;
static int simpleDelim(simple_tokenizer *t, unsigned char c){
return c<0x80 && t->delim[c];
}
static int fts3_isalnum(int x){
return (x>='0' && x<='9') || (x>='A' && x<='Z') || (x>='a' && x<='z');
}
/*
** Create a new tokenizer instance.
*/
static int simpleCreate(
int argc, const char * const *argv,
sqlite3_tokenizer **ppTokenizer
){
simple_tokenizer *t;
t = (simple_tokenizer *) sqlite3_malloc(sizeof(*t));
if( t==NULL ) return SQLITE_NOMEM;
memset(t, 0, sizeof(*t));
/* TODO(shess) Delimiters need to remain the same from run to run,
** else we need to reindex. One solution would be a meta-table to
** track such information in the database, then we'd only want this
** information on the initial create.
*/
if( argc>1 ){
int i, n = (int)strlen(argv[1]);
for(i=0; i<n; i++){
unsigned char ch = argv[1][i];
/* We explicitly don't support UTF-8 delimiters for now. */
if( ch>=0x80 ){
sqlite3_free(t);
return SQLITE_ERROR;
}
t->delim[ch] = 1;
}
} else {
/* Mark non-alphanumeric ASCII characters as delimiters */
int i;
for(i=1; i<0x80; i++){
t->delim[i] = !fts3_isalnum(i) ? -1 : 0;
}
}
*ppTokenizer = &t->base;
return SQLITE_OK;
}
/*
** Destroy a tokenizer
*/
static int simpleDestroy(sqlite3_tokenizer *pTokenizer){
sqlite3_free(pTokenizer);
return SQLITE_OK;
}
/*
** Prepare to begin tokenizing a particular string. The input
** string to be tokenized is pInput[0..nBytes-1]. A cursor
** used to incrementally tokenize this string is returned in
** *ppCursor.
*/
static int simpleOpen(
sqlite3_tokenizer *pTokenizer, /* The tokenizer */
const char *pInput, int nBytes, /* String to be tokenized */
sqlite3_tokenizer_cursor **ppCursor /* OUT: Tokenization cursor */
){
simple_tokenizer_cursor *c;
UNUSED_PARAMETER(pTokenizer);
c = (simple_tokenizer_cursor *) sqlite3_malloc(sizeof(*c));
if( c==NULL ) return SQLITE_NOMEM;
c->pInput = pInput;
if( pInput==0 ){
c->nBytes = 0;
}else if( nBytes<0 ){
c->nBytes = (int)strlen(pInput);
}else{
c->nBytes = nBytes;
}
c->iOffset = 0; /* start tokenizing at the beginning */
c->iToken = 0;
c->pToken = NULL; /* no space allocated, yet. */
c->nTokenAllocated = 0;
*ppCursor = &c->base;
return SQLITE_OK;
}
/*
** Close a tokenization cursor previously opened by a call to
** simpleOpen() above.
*/
static int simpleClose(sqlite3_tokenizer_cursor *pCursor){
simple_tokenizer_cursor *c = (simple_tokenizer_cursor *) pCursor;
sqlite3_free(c->pToken);
sqlite3_free(c);
return SQLITE_OK;
}
/*
** Extract the next token from a tokenization cursor. The cursor must
** have been opened by a prior call to simpleOpen().
*/
static int simpleNext(
sqlite3_tokenizer_cursor *pCursor, /* Cursor returned by simpleOpen */
const char **ppToken, /* OUT: *ppToken is the token text */
int *pnBytes, /* OUT: Number of bytes in token */
int *piStartOffset, /* OUT: Starting offset of token */
int *piEndOffset, /* OUT: Ending offset of token */
int *piPosition /* OUT: Position integer of token */
){
simple_tokenizer_cursor *c = (simple_tokenizer_cursor *) pCursor;
simple_tokenizer *t = (simple_tokenizer *) pCursor->pTokenizer;
unsigned char *p = (unsigned char *)c->pInput;
while( c->iOffset<c->nBytes ){
int iStartOffset;
/* Scan past delimiter characters */
while( c->iOffset<c->nBytes && simpleDelim(t, p[c->iOffset]) ){
c->iOffset++;
}
/* Count non-delimiter characters. */
iStartOffset = c->iOffset;
while( c->iOffset<c->nBytes && !simpleDelim(t, p[c->iOffset]) ){
c->iOffset++;
}
if( c->iOffset>iStartOffset ){
int i, n = c->iOffset-iStartOffset;
if( n>c->nTokenAllocated ){
char *pNew;
c->nTokenAllocated = n+20;
pNew = sqlite3_realloc64(c->pToken, c->nTokenAllocated);
if( !pNew ) return SQLITE_NOMEM;
c->pToken = pNew;
}
for(i=0; i<n; i++){
/* TODO(shess) This needs expansion to handle UTF-8
** case-insensitivity.
*/
unsigned char ch = p[iStartOffset+i];
c->pToken[i] = (char)((ch>='A' && ch<='Z') ? ch-'A'+'a' : ch);
}
*ppToken = c->pToken;
*pnBytes = n;
*piStartOffset = iStartOffset;
*piEndOffset = c->iOffset;
*piPosition = c->iToken++;
return SQLITE_OK;
}
}
return SQLITE_DONE;
}
/*
** The set of routines that implement the simple tokenizer
*/
static const sqlite3_tokenizer_module simpleTokenizerModule = {
0,
simpleCreate,
simpleDestroy,
simpleOpen,
simpleClose,
simpleNext,
0,
};
/*
** Allocate a new simple tokenizer. Return a pointer to the new
** tokenizer in *ppModule
*/
void sqlite3Fts3SimpleTokenizerModule(
sqlite3_tokenizer_module const**ppModule
){
*ppModule = &simpleTokenizerModule;
}
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
ext/fts3/fts3_unicode.c
+397
View File
@@ -0,0 +1,397 @@
/*
** 2012 May 24
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
** Implementation of the "unicode" full-text-search tokenizer.
*/
#ifndef SQLITE_DISABLE_FTS3_UNICODE
#include "fts3Int.h"
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "fts3_tokenizer.h"
/*
** The following two macros - READ_UTF8 and WRITE_UTF8 - have been copied
** from the sqlite3 source file utf.c. If this file is compiled as part
** of the amalgamation, they are not required.
*/
#ifndef SQLITE_AMALGAMATION
static const unsigned char sqlite3Utf8Trans1[] = {
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17,
0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
0x00, 0x01, 0x02, 0x03, 0x00, 0x01, 0x00, 0x00,
};
#define READ_UTF8(zIn, zTerm, c) \
c = *(zIn++); \
if( c>=0xc0 ){ \
c = sqlite3Utf8Trans1[c-0xc0]; \
while( zIn!=zTerm && (*zIn & 0xc0)==0x80 ){ \
c = (c<<6) + (0x3f & *(zIn++)); \
} \
if( c<0x80 \
|| (c&0xFFFFF800)==0xD800 \
|| (c&0xFFFFFFFE)==0xFFFE ){ c = 0xFFFD; } \
}
#define WRITE_UTF8(zOut, c) { \
if( c<0x00080 ){ \
*zOut++ = (u8)(c&0xFF); \
} \
else if( c<0x00800 ){ \
*zOut++ = 0xC0 + (u8)((c>>6)&0x1F); \
*zOut++ = 0x80 + (u8)(c & 0x3F); \
} \
else if( c<0x10000 ){ \
*zOut++ = 0xE0 + (u8)((c>>12)&0x0F); \
*zOut++ = 0x80 + (u8)((c>>6) & 0x3F); \
*zOut++ = 0x80 + (u8)(c & 0x3F); \
}else{ \
*zOut++ = 0xF0 + (u8)((c>>18) & 0x07); \
*zOut++ = 0x80 + (u8)((c>>12) & 0x3F); \
*zOut++ = 0x80 + (u8)((c>>6) & 0x3F); \
*zOut++ = 0x80 + (u8)(c & 0x3F); \
} \
}
#endif /* ifndef SQLITE_AMALGAMATION */
typedef struct unicode_tokenizer unicode_tokenizer;
typedef struct unicode_cursor unicode_cursor;
struct unicode_tokenizer {
sqlite3_tokenizer base;
int eRemoveDiacritic;
int nException;
int *aiException;
};
struct unicode_cursor {
sqlite3_tokenizer_cursor base;
const unsigned char *aInput; /* Input text being tokenized */
int nInput; /* Size of aInput[] in bytes */
int iOff; /* Current offset within aInput[] */
int iToken; /* Index of next token to be returned */
char *zToken; /* storage for current token */
int nAlloc; /* space allocated at zToken */
};
/*
** Destroy a tokenizer allocated by unicodeCreate().
*/
static int unicodeDestroy(sqlite3_tokenizer *pTokenizer){
if( pTokenizer ){
unicode_tokenizer *p = (unicode_tokenizer *)pTokenizer;
sqlite3_free(p->aiException);
sqlite3_free(p);
}
return SQLITE_OK;
}
/*
** As part of a tokenchars= or separators= option, the CREATE VIRTUAL TABLE
** statement has specified that the tokenizer for this table shall consider
** all characters in string zIn/nIn to be separators (if bAlnum==0) or
** token characters (if bAlnum==1).
**
** For each codepoint in the zIn/nIn string, this function checks if the
** sqlite3FtsUnicodeIsalnum() function already returns the desired result.
** If so, no action is taken. Otherwise, the codepoint is added to the
** unicode_tokenizer.aiException[] array. For the purposes of tokenization,
** the return value of sqlite3FtsUnicodeIsalnum() is inverted for all
** codepoints in the aiException[] array.
**
** If a standalone diacritic mark (one that sqlite3FtsUnicodeIsdiacritic()
** identifies as a diacritic) occurs in the zIn/nIn string it is ignored.
** It is not possible to change the behavior of the tokenizer with respect
** to these codepoints.
*/
static int unicodeAddExceptions(
unicode_tokenizer *p, /* Tokenizer to add exceptions to */
int bAlnum, /* Replace Isalnum() return value with this */
const char *zIn, /* Array of characters to make exceptions */
int nIn /* Length of z in bytes */
){
const unsigned char *z = (const unsigned char *)zIn;
const unsigned char *zTerm = &z[nIn];
unsigned int iCode;
int nEntry = 0;
assert( bAlnum==0 || bAlnum==1 );
while( z<zTerm ){
READ_UTF8(z, zTerm, iCode);
assert( (sqlite3FtsUnicodeIsalnum((int)iCode) & 0xFFFFFFFE)==0 );
if( sqlite3FtsUnicodeIsalnum((int)iCode)!=bAlnum
&& sqlite3FtsUnicodeIsdiacritic((int)iCode)==0
){
nEntry++;
}
}
if( nEntry ){
int *aNew; /* New aiException[] array */
int nNew; /* Number of valid entries in array aNew[] */
aNew = sqlite3_realloc64(p->aiException,(p->nException+nEntry)*sizeof(int));
if( aNew==0 ) return SQLITE_NOMEM;
nNew = p->nException;
z = (const unsigned char *)zIn;
while( z<zTerm ){
READ_UTF8(z, zTerm, iCode);
if( sqlite3FtsUnicodeIsalnum((int)iCode)!=bAlnum
&& sqlite3FtsUnicodeIsdiacritic((int)iCode)==0
){
int i, j;
for(i=0; i<nNew && aNew[i]<(int)iCode; i++);
for(j=nNew; j>i; j--) aNew[j] = aNew[j-1];
aNew[i] = (int)iCode;
nNew++;
}
}
p->aiException = aNew;
p->nException = nNew;
}
return SQLITE_OK;
}
/*
** Return true if the p->aiException[] array contains the value iCode.
*/
static int unicodeIsException(unicode_tokenizer *p, int iCode){
if( p->nException>0 ){
int *a = p->aiException;
int iLo = 0;
int iHi = p->nException-1;
while( iHi>=iLo ){
int iTest = (iHi + iLo) / 2;
if( iCode==a[iTest] ){
return 1;
}else if( iCode>a[iTest] ){
iLo = iTest+1;
}else{
iHi = iTest-1;
}
}
}
return 0;
}
/*
** Return true if, for the purposes of tokenization, codepoint iCode is
** considered a token character (not a separator).
*/
static int unicodeIsAlnum(unicode_tokenizer *p, int iCode){
assert( (sqlite3FtsUnicodeIsalnum(iCode) & 0xFFFFFFFE)==0 );
return sqlite3FtsUnicodeIsalnum(iCode) ^ unicodeIsException(p, iCode);
}
/*
** Create a new tokenizer instance.
*/
static int unicodeCreate(
int nArg, /* Size of array argv[] */
const char * const *azArg, /* Tokenizer creation arguments */
sqlite3_tokenizer **pp /* OUT: New tokenizer handle */
){
unicode_tokenizer *pNew; /* New tokenizer object */
int i;
int rc = SQLITE_OK;
pNew = (unicode_tokenizer *) sqlite3_malloc(sizeof(unicode_tokenizer));
if( pNew==NULL ) return SQLITE_NOMEM;
memset(pNew, 0, sizeof(unicode_tokenizer));
pNew->eRemoveDiacritic = 1;
for(i=0; rc==SQLITE_OK && i<nArg; i++){
const char *z = azArg[i];
int n = (int)strlen(z);
if( n==19 && memcmp("remove_diacritics=1", z, 19)==0 ){
pNew->eRemoveDiacritic = 1;
}
else if( n==19 && memcmp("remove_diacritics=0", z, 19)==0 ){
pNew->eRemoveDiacritic = 0;
}
else if( n==19 && memcmp("remove_diacritics=2", z, 19)==0 ){
pNew->eRemoveDiacritic = 2;
}
else if( n>=11 && memcmp("tokenchars=", z, 11)==0 ){
rc = unicodeAddExceptions(pNew, 1, &z[11], n-11);
}
else if( n>=11 && memcmp("separators=", z, 11)==0 ){
rc = unicodeAddExceptions(pNew, 0, &z[11], n-11);
}
else{
/* Unrecognized argument */
rc = SQLITE_ERROR;
}
}
if( rc!=SQLITE_OK ){
unicodeDestroy((sqlite3_tokenizer *)pNew);
pNew = 0;
}
*pp = (sqlite3_tokenizer *)pNew;
return rc;
}
/*
** Prepare to begin tokenizing a particular string. The input
** string to be tokenized is pInput[0..nBytes-1]. A cursor
** used to incrementally tokenize this string is returned in
** *ppCursor.
*/
static int unicodeOpen(
sqlite3_tokenizer *p, /* The tokenizer */
const char *aInput, /* Input string */
int nInput, /* Size of string aInput in bytes */
sqlite3_tokenizer_cursor **pp /* OUT: New cursor object */
){
unicode_cursor *pCsr;
pCsr = (unicode_cursor *)sqlite3_malloc(sizeof(unicode_cursor));
if( pCsr==0 ){
return SQLITE_NOMEM;
}
memset(pCsr, 0, sizeof(unicode_cursor));
pCsr->aInput = (const unsigned char *)aInput;
if( aInput==0 ){
pCsr->nInput = 0;
pCsr->aInput = (const unsigned char*)"";
}else if( nInput<0 ){
pCsr->nInput = (int)strlen(aInput);
}else{
pCsr->nInput = nInput;
}
*pp = &pCsr->base;
UNUSED_PARAMETER(p);
return SQLITE_OK;
}
/*
** Close a tokenization cursor previously opened by a call to
** simpleOpen() above.
*/
static int unicodeClose(sqlite3_tokenizer_cursor *pCursor){
unicode_cursor *pCsr = (unicode_cursor *) pCursor;
sqlite3_free(pCsr->zToken);
sqlite3_free(pCsr);
return SQLITE_OK;
}
/*
** Extract the next token from a tokenization cursor. The cursor must
** have been opened by a prior call to simpleOpen().
*/
static int unicodeNext(
sqlite3_tokenizer_cursor *pC, /* Cursor returned by simpleOpen */
const char **paToken, /* OUT: Token text */
int *pnToken, /* OUT: Number of bytes at *paToken */
int *piStart, /* OUT: Starting offset of token */
int *piEnd, /* OUT: Ending offset of token */
int *piPos /* OUT: Position integer of token */
){
unicode_cursor *pCsr = (unicode_cursor *)pC;
unicode_tokenizer *p = ((unicode_tokenizer *)pCsr->base.pTokenizer);
unsigned int iCode = 0;
char *zOut;
const unsigned char *z = &pCsr->aInput[pCsr->iOff];
const unsigned char *zStart = z;
const unsigned char *zEnd;
const unsigned char *zTerm = &pCsr->aInput[pCsr->nInput];
/* Scan past any delimiter characters before the start of the next token.
** Return SQLITE_DONE early if this takes us all the way to the end of
** the input. */
while( z<zTerm ){
READ_UTF8(z, zTerm, iCode);
if( unicodeIsAlnum(p, (int)iCode) ) break;
zStart = z;
}
if( zStart>=zTerm ) return SQLITE_DONE;
zOut = pCsr->zToken;
do {
int iOut;
/* Grow the output buffer if required. */
if( (zOut-pCsr->zToken)>=(pCsr->nAlloc-4) ){
char *zNew = sqlite3_realloc64(pCsr->zToken, pCsr->nAlloc+64);
if( !zNew ) return SQLITE_NOMEM;
zOut = &zNew[zOut - pCsr->zToken];
pCsr->zToken = zNew;
pCsr->nAlloc += 64;
}
/* Write the folded case of the last character read to the output */
zEnd = z;
iOut = sqlite3FtsUnicodeFold((int)iCode, p->eRemoveDiacritic);
if( iOut ){
WRITE_UTF8(zOut, iOut);
}
/* If the cursor is not at EOF, read the next character */
if( z>=zTerm ) break;
READ_UTF8(z, zTerm, iCode);
}while( unicodeIsAlnum(p, (int)iCode)
|| sqlite3FtsUnicodeIsdiacritic((int)iCode)
);
/* Set the output variables and return. */
pCsr->iOff = (int)(z - pCsr->aInput);
*paToken = pCsr->zToken;
*pnToken = (int)(zOut - pCsr->zToken);
*piStart = (int)(zStart - pCsr->aInput);
*piEnd = (int)(zEnd - pCsr->aInput);
*piPos = pCsr->iToken++;
return SQLITE_OK;
}
/*
** Set *ppModule to a pointer to the sqlite3_tokenizer_module
** structure for the unicode tokenizer.
*/
void sqlite3Fts3UnicodeTokenizer(sqlite3_tokenizer_module const **ppModule){
static const sqlite3_tokenizer_module module = {
0,
unicodeCreate,
unicodeDestroy,
unicodeOpen,
unicodeClose,
unicodeNext,
0,
};
*ppModule = &module;
}
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
#endif /* ifndef SQLITE_DISABLE_FTS3_UNICODE */
ext/fts3/fts3_unicode2.c
+383
View File
@@ -0,0 +1,383 @@
/*
** 2012-05-25
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
*/
/*
** DO NOT EDIT THIS MACHINE GENERATED FILE.
*/
#ifndef SQLITE_DISABLE_FTS3_UNICODE
#if defined(SQLITE_ENABLE_FTS3) || defined(SQLITE_ENABLE_FTS4)
#include <assert.h>
/*
** Return true if the argument corresponds to a unicode codepoint
** classified as either a letter or a number. Otherwise false.
**
** The results are undefined if the value passed to this function
** is less than zero.
*/
int sqlite3FtsUnicodeIsalnum(int c){
/* Each unsigned integer in the following array corresponds to a contiguous
** range of unicode codepoints that are not either letters or numbers (i.e.
** codepoints for which this function should return 0).
**
** The most significant 22 bits in each 32-bit value contain the first
** codepoint in the range. The least significant 10 bits are used to store
** the size of the range (always at least 1). In other words, the value
** ((C<<22) + N) represents a range of N codepoints starting with codepoint
** C. It is not possible to represent a range larger than 1023 codepoints
** using this format.
*/
static const unsigned int aEntry[] = {
0x00000030, 0x0000E807, 0x00016C06, 0x0001EC2F, 0x0002AC07,
0x0002D001, 0x0002D803, 0x0002EC01, 0x0002FC01, 0x00035C01,
0x0003DC01, 0x000B0804, 0x000B480E, 0x000B9407, 0x000BB401,
0x000BBC81, 0x000DD401, 0x000DF801, 0x000E1002, 0x000E1C01,
0x000FD801, 0x00120808, 0x00156806, 0x00162402, 0x00163C01,
0x00164437, 0x0017CC02, 0x00180005, 0x00181816, 0x00187802,
0x00192C15, 0x0019A804, 0x0019C001, 0x001B5001, 0x001B580F,
0x001B9C07, 0x001BF402, 0x001C000E, 0x001C3C01, 0x001C4401,
0x001CC01B, 0x001E980B, 0x001FAC09, 0x001FD804, 0x00205804,
0x00206C09, 0x00209403, 0x0020A405, 0x0020C00F, 0x00216403,
0x00217801, 0x0023901B, 0x00240004, 0x0024E803, 0x0024F812,
0x00254407, 0x00258804, 0x0025C001, 0x00260403, 0x0026F001,
0x0026F807, 0x00271C02, 0x00272C03, 0x00275C01, 0x00278802,
0x0027C802, 0x0027E802, 0x00280403, 0x0028F001, 0x0028F805,
0x00291C02, 0x00292C03, 0x00294401, 0x0029C002, 0x0029D401,
0x002A0403, 0x002AF001, 0x002AF808, 0x002B1C03, 0x002B2C03,
0x002B8802, 0x002BC002, 0x002C0403, 0x002CF001, 0x002CF807,
0x002D1C02, 0x002D2C03, 0x002D5802, 0x002D8802, 0x002DC001,
0x002E0801, 0x002EF805, 0x002F1803, 0x002F2804, 0x002F5C01,
0x002FCC08, 0x00300403, 0x0030F807, 0x00311803, 0x00312804,
0x00315402, 0x00318802, 0x0031FC01, 0x00320802, 0x0032F001,
0x0032F807, 0x00331803, 0x00332804, 0x00335402, 0x00338802,
0x00340802, 0x0034F807, 0x00351803, 0x00352804, 0x00355C01,
0x00358802, 0x0035E401, 0x00360802, 0x00372801, 0x00373C06,
0x00375801, 0x00376008, 0x0037C803, 0x0038C401, 0x0038D007,
0x0038FC01, 0x00391C09, 0x00396802, 0x003AC401, 0x003AD006,
0x003AEC02, 0x003B2006, 0x003C041F, 0x003CD00C, 0x003DC417,
0x003E340B, 0x003E6424, 0x003EF80F, 0x003F380D, 0x0040AC14,
0x00412806, 0x00415804, 0x00417803, 0x00418803, 0x00419C07,
0x0041C404, 0x0042080C, 0x00423C01, 0x00426806, 0x0043EC01,
0x004D740C, 0x004E400A, 0x00500001, 0x0059B402, 0x005A0001,
0x005A6C02, 0x005BAC03, 0x005C4803, 0x005CC805, 0x005D4802,
0x005DC802, 0x005ED023, 0x005F6004, 0x005F7401, 0x0060000F,
0x0062A401, 0x0064800C, 0x0064C00C, 0x00650001, 0x00651002,
0x0066C011, 0x00672002, 0x00677822, 0x00685C05, 0x00687802,
0x0069540A, 0x0069801D, 0x0069FC01, 0x006A8007, 0x006AA006,
0x006C0005, 0x006CD011, 0x006D6823, 0x006E0003, 0x006E840D,
0x006F980E, 0x006FF004, 0x00709014, 0x0070EC05, 0x0071F802,
0x00730008, 0x00734019, 0x0073B401, 0x0073C803, 0x00770027,
0x0077F004, 0x007EF401, 0x007EFC03, 0x007F3403, 0x007F7403,
0x007FB403, 0x007FF402, 0x00800065, 0x0081A806, 0x0081E805,
0x00822805, 0x0082801A, 0x00834021, 0x00840002, 0x00840C04,
0x00842002, 0x00845001, 0x00845803, 0x00847806, 0x00849401,
0x00849C01, 0x0084A401, 0x0084B801, 0x0084E802, 0x00850005,
0x00852804, 0x00853C01, 0x00864264, 0x00900027, 0x0091000B,
0x0092704E, 0x00940200, 0x009C0475, 0x009E53B9, 0x00AD400A,
0x00B39406, 0x00B3BC03, 0x00B3E404, 0x00B3F802, 0x00B5C001,
0x00B5FC01, 0x00B7804F, 0x00B8C00C, 0x00BA001A, 0x00BA6C59,
0x00BC00D6, 0x00BFC00C, 0x00C00005, 0x00C02019, 0x00C0A807,
0x00C0D802, 0x00C0F403, 0x00C26404, 0x00C28001, 0x00C3EC01,
0x00C64002, 0x00C6580A, 0x00C70024, 0x00C8001F, 0x00C8A81E,
0x00C94001, 0x00C98020, 0x00CA2827, 0x00CB003F, 0x00CC0100,
0x01370040, 0x02924037, 0x0293F802, 0x02983403, 0x0299BC10,
0x029A7C01, 0x029BC008, 0x029C0017, 0x029C8002, 0x029E2402,
0x02A00801, 0x02A01801, 0x02A02C01, 0x02A08C09, 0x02A0D804,
0x02A1D004, 0x02A20002, 0x02A2D011, 0x02A33802, 0x02A38012,
0x02A3E003, 0x02A4980A, 0x02A51C0D, 0x02A57C01, 0x02A60004,
0x02A6CC1B, 0x02A77802, 0x02A8A40E, 0x02A90C01, 0x02A93002,
0x02A97004, 0x02A9DC03, 0x02A9EC01, 0x02AAC001, 0x02AAC803,
0x02AADC02, 0x02AAF802, 0x02AB0401, 0x02AB7802, 0x02ABAC07,
0x02ABD402, 0x02AF8C0B, 0x03600001, 0x036DFC02, 0x036FFC02,
0x037FFC01, 0x03EC7801, 0x03ECA401, 0x03EEC810, 0x03F4F802,
0x03F7F002, 0x03F8001A, 0x03F88007, 0x03F8C023, 0x03F95013,
0x03F9A004, 0x03FBFC01, 0x03FC040F, 0x03FC6807, 0x03FCEC06,
0x03FD6C0B, 0x03FF8007, 0x03FFA007, 0x03FFE405, 0x04040003,
0x0404DC09, 0x0405E411, 0x0406400C, 0x0407402E, 0x040E7C01,
0x040F4001, 0x04215C01, 0x04247C01, 0x0424FC01, 0x04280403,
0x04281402, 0x04283004, 0x0428E003, 0x0428FC01, 0x04294009,
0x0429FC01, 0x042CE407, 0x04400003, 0x0440E016, 0x04420003,
0x0442C012, 0x04440003, 0x04449C0E, 0x04450004, 0x04460003,
0x0446CC0E, 0x04471404, 0x045AAC0D, 0x0491C004, 0x05BD442E,
0x05BE3C04, 0x074000F6, 0x07440027, 0x0744A4B5, 0x07480046,
0x074C0057, 0x075B0401, 0x075B6C01, 0x075BEC01, 0x075C5401,
0x075CD401, 0x075D3C01, 0x075DBC01, 0x075E2401, 0x075EA401,
0x075F0C01, 0x07BBC002, 0x07C0002C, 0x07C0C064, 0x07C2800F,
0x07C2C40E, 0x07C3040F, 0x07C3440F, 0x07C4401F, 0x07C4C03C,
0x07C5C02B, 0x07C7981D, 0x07C8402B, 0x07C90009, 0x07C94002,
0x07CC0021, 0x07CCC006, 0x07CCDC46, 0x07CE0014, 0x07CE8025,
0x07CF1805, 0x07CF8011, 0x07D0003F, 0x07D10001, 0x07D108B6,
0x07D3E404, 0x07D4003E, 0x07D50004, 0x07D54018, 0x07D7EC46,
0x07D9140B, 0x07DA0046, 0x07DC0074, 0x38000401, 0x38008060,
0x380400F0,
};
static const unsigned int aAscii[4] = {
0xFFFFFFFF, 0xFC00FFFF, 0xF8000001, 0xF8000001,
};
if( (unsigned int)c<128 ){
return ( (aAscii[c >> 5] & ((unsigned int)1 << (c & 0x001F)))==0 );
}else if( (unsigned int)c<(1<<22) ){
unsigned int key = (((unsigned int)c)<<10) | 0x000003FF;
int iRes = 0;
int iHi = sizeof(aEntry)/sizeof(aEntry[0]) - 1;
int iLo = 0;
while( iHi>=iLo ){
int iTest = (iHi + iLo) / 2;
if( key >= aEntry[iTest] ){
iRes = iTest;
iLo = iTest+1;
}else{
iHi = iTest-1;
}
}
assert( aEntry[0]<key );
assert( key>=aEntry[iRes] );
return (((unsigned int)c) >= ((aEntry[iRes]>>10) + (aEntry[iRes]&0x3FF)));
}
return 1;
}
/*
** If the argument is a codepoint corresponding to a lowercase letter
** in the ASCII range with a diacritic added, return the codepoint
** of the ASCII letter only. For example, if passed 235 - "LATIN
** SMALL LETTER E WITH DIAERESIS" - return 65 ("LATIN SMALL LETTER
** E"). The resuls of passing a codepoint that corresponds to an
** uppercase letter are undefined.
*/
static int remove_diacritic(int c, int bComplex){
unsigned short aDia[] = {
0, 1797, 1848, 1859, 1891, 1928, 1940, 1995,
2024, 2040, 2060, 2110, 2168, 2206, 2264, 2286,
2344, 2383, 2472, 2488, 2516, 2596, 2668, 2732,
2782, 2842, 2894, 2954, 2984, 3000, 3028, 3336,
3456, 3696, 3712, 3728, 3744, 3766, 3832, 3896,
3912, 3928, 3944, 3968, 4008, 4040, 4056, 4106,
4138, 4170, 4202, 4234, 4266, 4296, 4312, 4344,
4408, 4424, 4442, 4472, 4488, 4504, 6148, 6198,
6264, 6280, 6360, 6429, 6505, 6529, 61448, 61468,
61512, 61534, 61592, 61610, 61642, 61672, 61688, 61704,
61726, 61784, 61800, 61816, 61836, 61880, 61896, 61914,
61948, 61998, 62062, 62122, 62154, 62184, 62200, 62218,
62252, 62302, 62364, 62410, 62442, 62478, 62536, 62554,
62584, 62604, 62640, 62648, 62656, 62664, 62730, 62766,
62830, 62890, 62924, 62974, 63032, 63050, 63082, 63118,
63182, 63242, 63274, 63310, 63368, 63390,
};
#define HIBIT ((unsigned char)0x80)
unsigned char aChar[] = {
'\0', 'a', 'c', 'e', 'i', 'n',
'o', 'u', 'y', 'y', 'a', 'c',
'd', 'e', 'e', 'g', 'h', 'i',
'j', 'k', 'l', 'n', 'o', 'r',
's', 't', 'u', 'u', 'w', 'y',
'z', 'o', 'u', 'a', 'i', 'o',
'u', 'u'|HIBIT, 'a'|HIBIT, 'g', 'k', 'o',
'o'|HIBIT, 'j', 'g', 'n', 'a'|HIBIT, 'a',
'e', 'i', 'o', 'r', 'u', 's',
't', 'h', 'a', 'e', 'o'|HIBIT, 'o',
'o'|HIBIT, 'y', '\0', '\0', '\0', '\0',
'\0', '\0', '\0', '\0', 'a', 'b',
'c'|HIBIT, 'd', 'd', 'e'|HIBIT, 'e', 'e'|HIBIT,
'f', 'g', 'h', 'h', 'i', 'i'|HIBIT,
'k', 'l', 'l'|HIBIT, 'l', 'm', 'n',
'o'|HIBIT, 'p', 'r', 'r'|HIBIT, 'r', 's',
's'|HIBIT, 't', 'u', 'u'|HIBIT, 'v', 'w',
'w', 'x', 'y', 'z', 'h', 't',
'w', 'y', 'a', 'a'|HIBIT, 'a'|HIBIT, 'a'|HIBIT,
'e', 'e'|HIBIT, 'e'|HIBIT, 'i', 'o', 'o'|HIBIT,
'o'|HIBIT, 'o'|HIBIT, 'u', 'u'|HIBIT, 'u'|HIBIT, 'y',
};
unsigned int key = (((unsigned int)c)<<3) | 0x00000007;
int iRes = 0;
int iHi = sizeof(aDia)/sizeof(aDia[0]) - 1;
int iLo = 0;
while( iHi>=iLo ){
int iTest = (iHi + iLo) / 2;
if( key >= aDia[iTest] ){
iRes = iTest;
iLo = iTest+1;
}else{
iHi = iTest-1;
}
}
assert( key>=aDia[iRes] );
if( bComplex==0 && (aChar[iRes] & 0x80) ) return c;
return (c > (aDia[iRes]>>3) + (aDia[iRes]&0x07)) ? c : ((int)aChar[iRes] & 0x7F);
}
/*
** Return true if the argument interpreted as a unicode codepoint
** is a diacritical modifier character.
*/
int sqlite3FtsUnicodeIsdiacritic(int c){
unsigned int mask0 = 0x08029FDF;
unsigned int mask1 = 0x000361F8;
if( c<768 || c>817 ) return 0;
return (c < 768+32) ?
(mask0 & ((unsigned int)1 << (c-768))) :
(mask1 & ((unsigned int)1 << (c-768-32)));
}
/*
** Interpret the argument as a unicode codepoint. If the codepoint
** is an upper case character that has a lower case equivalent,
** return the codepoint corresponding to the lower case version.
** Otherwise, return a copy of the argument.
**
** The results are undefined if the value passed to this function
** is less than zero.
*/
int sqlite3FtsUnicodeFold(int c, int eRemoveDiacritic){
/* Each entry in the following array defines a rule for folding a range
** of codepoints to lower case. The rule applies to a range of nRange
** codepoints starting at codepoint iCode.
**
** If the least significant bit in flags is clear, then the rule applies
** to all nRange codepoints (i.e. all nRange codepoints are upper case and
** need to be folded). Or, if it is set, then the rule only applies to
** every second codepoint in the range, starting with codepoint C.
**
** The 7 most significant bits in flags are an index into the aiOff[]
** array. If a specific codepoint C does require folding, then its lower
** case equivalent is ((C + aiOff[flags>>1]) & 0xFFFF).
**
** The contents of this array are generated by parsing the CaseFolding.txt
** file distributed as part of the "Unicode Character Database". See
** http://www.unicode.org for details.
*/
static const struct TableEntry {
unsigned short iCode;
unsigned char flags;
unsigned char nRange;
} aEntry[] = {
{65, 14, 26}, {181, 64, 1}, {192, 14, 23},
{216, 14, 7}, {256, 1, 48}, {306, 1, 6},
{313, 1, 16}, {330, 1, 46}, {376, 116, 1},
{377, 1, 6}, {383, 104, 1}, {385, 50, 1},
{386, 1, 4}, {390, 44, 1}, {391, 0, 1},
{393, 42, 2}, {395, 0, 1}, {398, 32, 1},
{399, 38, 1}, {400, 40, 1}, {401, 0, 1},
{403, 42, 1}, {404, 46, 1}, {406, 52, 1},
{407, 48, 1}, {408, 0, 1}, {412, 52, 1},
{413, 54, 1}, {415, 56, 1}, {416, 1, 6},
{422, 60, 1}, {423, 0, 1}, {425, 60, 1},
{428, 0, 1}, {430, 60, 1}, {431, 0, 1},
{433, 58, 2}, {435, 1, 4}, {439, 62, 1},
{440, 0, 1}, {444, 0, 1}, {452, 2, 1},
{453, 0, 1}, {455, 2, 1}, {456, 0, 1},
{458, 2, 1}, {459, 1, 18}, {478, 1, 18},
{497, 2, 1}, {498, 1, 4}, {502, 122, 1},
{503, 134, 1}, {504, 1, 40}, {544, 110, 1},
{546, 1, 18}, {570, 70, 1}, {571, 0, 1},
{573, 108, 1}, {574, 68, 1}, {577, 0, 1},
{579, 106, 1}, {580, 28, 1}, {581, 30, 1},
{582, 1, 10}, {837, 36, 1}, {880, 1, 4},
{886, 0, 1}, {902, 18, 1}, {904, 16, 3},
{908, 26, 1}, {910, 24, 2}, {913, 14, 17},
{931, 14, 9}, {962, 0, 1}, {975, 4, 1},
{976, 140, 1}, {977, 142, 1}, {981, 146, 1},
{982, 144, 1}, {984, 1, 24}, {1008, 136, 1},
{1009, 138, 1}, {1012, 130, 1}, {1013, 128, 1},
{1015, 0, 1}, {1017, 152, 1}, {1018, 0, 1},
{1021, 110, 3}, {1024, 34, 16}, {1040, 14, 32},
{1120, 1, 34}, {1162, 1, 54}, {1216, 6, 1},
{1217, 1, 14}, {1232, 1, 88}, {1329, 22, 38},
{4256, 66, 38}, {4295, 66, 1}, {4301, 66, 1},
{7680, 1, 150}, {7835, 132, 1}, {7838, 96, 1},
{7840, 1, 96}, {7944, 150, 8}, {7960, 150, 6},
{7976, 150, 8}, {7992, 150, 8}, {8008, 150, 6},
{8025, 151, 8}, {8040, 150, 8}, {8072, 150, 8},
{8088, 150, 8}, {8104, 150, 8}, {8120, 150, 2},
{8122, 126, 2}, {8124, 148, 1}, {8126, 100, 1},
{8136, 124, 4}, {8140, 148, 1}, {8152, 150, 2},
{8154, 120, 2}, {8168, 150, 2}, {8170, 118, 2},
{8172, 152, 1}, {8184, 112, 2}, {8186, 114, 2},
{8188, 148, 1}, {8486, 98, 1}, {8490, 92, 1},
{8491, 94, 1}, {8498, 12, 1}, {8544, 8, 16},
{8579, 0, 1}, {9398, 10, 26}, {11264, 22, 47},
{11360, 0, 1}, {11362, 88, 1}, {11363, 102, 1},
{11364, 90, 1}, {11367, 1, 6}, {11373, 84, 1},
{11374, 86, 1}, {11375, 80, 1}, {11376, 82, 1},
{11378, 0, 1}, {11381, 0, 1}, {11390, 78, 2},
{11392, 1, 100}, {11499, 1, 4}, {11506, 0, 1},
{42560, 1, 46}, {42624, 1, 24}, {42786, 1, 14},
{42802, 1, 62}, {42873, 1, 4}, {42877, 76, 1},
{42878, 1, 10}, {42891, 0, 1}, {42893, 74, 1},
{42896, 1, 4}, {42912, 1, 10}, {42922, 72, 1},
{65313, 14, 26},
};
static const unsigned short aiOff[] = {
1, 2, 8, 15, 16, 26, 28, 32,
37, 38, 40, 48, 63, 64, 69, 71,
79, 80, 116, 202, 203, 205, 206, 207,
209, 210, 211, 213, 214, 217, 218, 219,
775, 7264, 10792, 10795, 23228, 23256, 30204, 54721,
54753, 54754, 54756, 54787, 54793, 54809, 57153, 57274,
57921, 58019, 58363, 61722, 65268, 65341, 65373, 65406,
65408, 65410, 65415, 65424, 65436, 65439, 65450, 65462,
65472, 65476, 65478, 65480, 65482, 65488, 65506, 65511,
65514, 65521, 65527, 65528, 65529,
};
int ret = c;
assert( sizeof(unsigned short)==2 && sizeof(unsigned char)==1 );
if( c<128 ){
if( c>='A' && c<='Z' ) ret = c + ('a' - 'A');
}else if( c<65536 ){
const struct TableEntry *p;
int iHi = sizeof(aEntry)/sizeof(aEntry[0]) - 1;
int iLo = 0;
int iRes = -1;
assert( c>aEntry[0].iCode );
while( iHi>=iLo ){
int iTest = (iHi + iLo) / 2;
int cmp = (c - aEntry[iTest].iCode);
if( cmp>=0 ){
iRes = iTest;
iLo = iTest+1;
}else{
iHi = iTest-1;
}
}
assert( iRes>=0 && c>=aEntry[iRes].iCode );
p = &aEntry[iRes];
if( c<(p->iCode + p->nRange) && 0==(0x01 & p->flags & (p->iCode ^ c)) ){
ret = (c + (aiOff[p->flags>>1])) & 0x0000FFFF;
assert( ret>0 );
}
if( eRemoveDiacritic ){
ret = remove_diacritic(ret, eRemoveDiacritic==2);
}
}
else if( c>=66560 && c<66600 ){
ret = c + 40;
}
return ret;
}
#endif /* defined(SQLITE_ENABLE_FTS3) || defined(SQLITE_ENABLE_FTS4) */
#endif /* !defined(SQLITE_DISABLE_FTS3_UNICODE) */
ext/fts3/fts3_write.c
+5834
View File
File diff suppressed because it is too large. Load diff
ext/fts3/fts3speed.tcl
+122
View File
@@ -0,0 +1,122 @@
#--------------------------------------------------------------------------
# This script contains several sub-programs used to test FTS3/FTS4
# performance. It does not run the queries directly, but generates SQL
# scripts that can be run using the shell tool.
#
# The following cases are tested:
#
# 1. Inserting documents into an FTS3 table.
# 2. Optimizing an FTS3 table (i.e. "INSERT INTO t1 VALUES('optimize')").
# 3. Deleting documents from an FTS3 table.
# 4. Querying FTS3 tables.
#
# Number of tokens in vocabulary. And number of tokens in each document.
#
set VOCAB_SIZE 2000
set DOC_SIZE 100
set NUM_INSERTS 100000
set NUM_SELECTS 1000
# Force everything in this script to be deterministic.
#
expr {srand(0)}
proc usage {} {
puts stderr "Usage: $::argv0 <rows> <selects>"
exit -1
}
proc sql {sql} {
puts $::fd $sql
}
# Return a list of $nWord randomly generated tokens each between 2 and 10
# characters in length.
#
proc build_vocab {nWord} {
set ret [list]
set chars [list a b c d e f g h i j k l m n o p q r s t u v w x y z]
for {set i 0} {$i<$nWord} {incr i} {
set len [expr {int((rand()*9.0)+2)}]
set term ""
for {set j 0} {$j<$len} {incr j} {
append term [lindex $chars [expr {int(rand()*[llength $chars])}]]
}
lappend ret $term
}
set ret
}
proc select_term {} {
set n [llength $::vocab]
set t [expr int(rand()*$n*3)]
if {$t>=2*$n} { set t [expr {($t-2*$n)/100}] }
if {$t>=$n} { set t [expr {($t-$n)/10}] }
lindex $::vocab $t
}
proc select_doc {nTerm} {
set ret [list]
for {set i 0} {$i<$nTerm} {incr i} {
lappend ret [select_term]
}
set ret
}
proc test_1 {nInsert} {
sql "PRAGMA synchronous = OFF;"
sql "DROP TABLE IF EXISTS t1;"
sql "CREATE VIRTUAL TABLE t1 USING fts4;"
for {set i 0} {$i < $nInsert} {incr i} {
set doc [select_doc $::DOC_SIZE]
sql "INSERT INTO t1 VALUES('$doc');"
}
}
proc test_2 {} {
sql "INSERT INTO t1(t1) VALUES('optimize');"
}
proc test_3 {nSelect} {
for {set i 0} {$i < $nSelect} {incr i} {
sql "SELECT count(*) FROM t1 WHERE t1 MATCH '[select_term]';"
}
}
proc test_4 {nSelect} {
for {set i 0} {$i < $nSelect} {incr i} {
sql "SELECT count(*) FROM t1 WHERE t1 MATCH '[select_term] [select_term]';"
}
}
if {[llength $argv]!=0} usage
set ::vocab [build_vocab $::VOCAB_SIZE]
set ::fd [open fts3speed_insert.sql w]
test_1 $NUM_INSERTS
close $::fd
set ::fd [open fts3speed_select.sql w]
test_3 $NUM_SELECTS
close $::fd
set ::fd [open fts3speed_select2.sql w]
test_4 $NUM_SELECTS
close $::fd
set ::fd [open fts3speed_optimize.sql w]
test_2
close $::fd
puts "Success. Created files:"
puts " fts3speed_insert.sql"
puts " fts3speed_select.sql"
puts " fts3speed_select2.sql"
puts " fts3speed_optimize.sql"
ext/fts3/tool/fts3cov.sh
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -e
srcdir=`dirname $(dirname $(dirname $(dirname $0)))`
./testfixture $srcdir/test/fts3.test --output=fts3cov-out.txt
echo ""
for f in `ls $srcdir/ext/fts3/*.c`
do
f=`basename $f`
echo -ne "$f: "
gcov -b $f | grep Taken | sed 's/Taken at least once://'
done
ext/fts3/tool/fts3view.c
+875
View File
@@ -0,0 +1,875 @@
/*
** This program is a debugging and analysis utility that displays
** information about an FTS3 or FTS4 index.
**
** Link this program against the SQLite3 amalgamation with the
** SQLITE_ENABLE_FTS4 compile-time option. Then run it as:
**
** fts3view DATABASE
**
** to get a list of all FTS3/4 tables in DATABASE, or do
**
** fts3view DATABASE TABLE COMMAND ....
**
** to see various aspects of the TABLE table. Type fts3view with no
** arguments for a list of available COMMANDs.
*/
#include <stdio.h>
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include "sqlite3.h"
/*
** Extra command-line arguments:
*/
int nExtra;
char **azExtra;
/*
** Look for a command-line argument.
*/
const char *findOption(const char *zName, int hasArg, const char *zDefault){
int i;
const char *zResult = zDefault;
for(i=0; i<nExtra; i++){
const char *z = azExtra[i];
while( z[0]=='-' ) z++;
if( strcmp(z, zName)==0 ){
int j = 1;
if( hasArg==0 || i==nExtra-1 ) j = 0;
zResult = azExtra[i+j];
while( i+j<nExtra ){
azExtra[i] = azExtra[i+j+1];
i++;
}
break;
}
}
return zResult;
}
/*
** Prepare an SQL query
*/
static sqlite3_stmt *prepare(sqlite3 *db, const char *zFormat, ...){
va_list ap;
char *zSql;
sqlite3_stmt *pStmt;
int rc;
va_start(ap, zFormat);
zSql = sqlite3_vmprintf(zFormat, ap);
va_end(ap);
rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
if( rc ){
fprintf(stderr, "Error: %s\nSQL: %s\n", sqlite3_errmsg(db), zSql);
exit(1);
}
sqlite3_free(zSql);
return pStmt;
}
/*
** Run an SQL statement
*/
static int runSql(sqlite3 *db, const char *zFormat, ...){
va_list ap;
char *zSql;
int rc;
va_start(ap, zFormat);
zSql = sqlite3_vmprintf(zFormat, ap);
rc = sqlite3_exec(db, zSql, 0, 0, 0);
va_end(ap);
return rc;
}
/*
** Show the table schema
*/
static void showSchema(sqlite3 *db, const char *zTab){
sqlite3_stmt *pStmt;
pStmt = prepare(db,
"SELECT sql FROM sqlite_schema"
" WHERE name LIKE '%q%%'"
" ORDER BY 1",
zTab);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
printf("%s;\n", sqlite3_column_text(pStmt, 0));
}
sqlite3_finalize(pStmt);
pStmt = prepare(db, "PRAGMA page_size");
while( sqlite3_step(pStmt)==SQLITE_ROW ){
printf("PRAGMA page_size=%s;\n", sqlite3_column_text(pStmt, 0));
}
sqlite3_finalize(pStmt);
pStmt = prepare(db, "PRAGMA journal_mode");
while( sqlite3_step(pStmt)==SQLITE_ROW ){
printf("PRAGMA journal_mode=%s;\n", sqlite3_column_text(pStmt, 0));
}
sqlite3_finalize(pStmt);
pStmt = prepare(db, "PRAGMA auto_vacuum");
while( sqlite3_step(pStmt)==SQLITE_ROW ){
const char *zType = "???";
switch( sqlite3_column_int(pStmt, 0) ){
case 0: zType = "OFF"; break;
case 1: zType = "FULL"; break;
case 2: zType = "INCREMENTAL"; break;
}
printf("PRAGMA auto_vacuum=%s;\n", zType);
}
sqlite3_finalize(pStmt);
pStmt = prepare(db, "PRAGMA encoding");
while( sqlite3_step(pStmt)==SQLITE_ROW ){
printf("PRAGMA encoding=%s;\n", sqlite3_column_text(pStmt, 0));
}
sqlite3_finalize(pStmt);
}
/*
** Read a 64-bit variable-length integer from memory starting at p[0].
** Return the number of bytes read, or 0 on error.
** The value is stored in *v.
*/
int getVarint(const unsigned char *p, sqlite_int64 *v){
const unsigned char *q = p;
sqlite_uint64 x = 0, y = 1;
while( (*q&0x80)==0x80 && q-(unsigned char *)p<9 ){
x += y * (*q++ & 0x7f);
y <<= 7;
}
x += y * (*q++);
*v = (sqlite_int64) x;
return (int) (q - (unsigned char *)p);
}
/* Show the content of the %_stat table
*/
static void showStat(sqlite3 *db, const char *zTab){
sqlite3_stmt *pStmt;
pStmt = prepare(db, "SELECT id, value FROM '%q_stat'", zTab);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
printf("stat[%d] =", sqlite3_column_int(pStmt, 0));
switch( sqlite3_column_type(pStmt, 1) ){
case SQLITE_INTEGER: {
printf(" %d\n", sqlite3_column_int(pStmt, 1));
break;
}
case SQLITE_BLOB: {
unsigned char *x = (unsigned char*)sqlite3_column_blob(pStmt, 1);
int len = sqlite3_column_bytes(pStmt, 1);
int i = 0;
sqlite3_int64 v;
while( i<len ){
i += getVarint(x, &v);
printf(" %lld", v);
}
printf("\n");
break;
}
}
}
sqlite3_finalize(pStmt);
}
/*
** Report on the vocabulary. This creates an fts4aux table with a random
** name, but deletes it in the end.
*/
static void showVocabulary(sqlite3 *db, const char *zTab){
char *zAux;
sqlite3_uint64 r;
sqlite3_stmt *pStmt;
int nDoc = 0;
int nToken = 0;
int nOccurrence = 0;
int nTop;
int n, i;
sqlite3_randomness(sizeof(r), &r);
zAux = sqlite3_mprintf("viewer_%llx", zTab, r);
runSql(db, "BEGIN");
pStmt = prepare(db, "SELECT count(*) FROM %Q", zTab);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
nDoc = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
printf("Number of documents...................... %9d\n", nDoc);
runSql(db, "CREATE VIRTUAL TABLE %s USING fts4aux(%Q)", zAux, zTab);
pStmt = prepare(db,
"SELECT count(*), sum(occurrences) FROM %s WHERE col='*'",
zAux);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
nToken = sqlite3_column_int(pStmt, 0);
nOccurrence = sqlite3_column_int(pStmt, 1);
}
sqlite3_finalize(pStmt);
printf("Total tokens in all documents............ %9d\n", nOccurrence);
printf("Total number of distinct tokens.......... %9d\n", nToken);
if( nToken==0 ) goto end_vocab;
n = 0;
pStmt = prepare(db, "SELECT count(*) FROM %s"
" WHERE col='*' AND occurrences==1", zAux);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
n = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
printf("Tokens used exactly once................. %9d %5.2f%%\n",
n, n*100.0/nToken);
n = 0;
pStmt = prepare(db, "SELECT count(*) FROM %s"
" WHERE col='*' AND documents==1", zAux);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
n = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
printf("Tokens used in only one document......... %9d %5.2f%%\n",
n, n*100.0/nToken);
if( nDoc>=2000 ){
n = 0;
pStmt = prepare(db, "SELECT count(*) FROM %s"
" WHERE col='*' AND occurrences<=%d", zAux, nDoc/1000);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
n = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
printf("Tokens used in 0.1%% or less of docs...... %9d %5.2f%%\n",
n, n*100.0/nToken);
}
if( nDoc>=200 ){
n = 0;
pStmt = prepare(db, "SELECT count(*) FROM %s"
" WHERE col='*' AND occurrences<=%d", zAux, nDoc/100);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
n = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
printf("Tokens used in 1%% or less of docs........ %9d %5.2f%%\n",
n, n*100.0/nToken);
}
nTop = atoi(findOption("top", 1, "25"));
printf("The %d most common tokens:\n", nTop);
pStmt = prepare(db,
"SELECT term, documents FROM %s"
" WHERE col='*'"
" ORDER BY documents DESC, term"
" LIMIT %d", zAux, nTop);
i = 0;
while( sqlite3_step(pStmt)==SQLITE_ROW ){
i++;
n = sqlite3_column_int(pStmt, 1);
printf(" %2d. %-30s %9d docs %5.2f%%\n", i,
sqlite3_column_text(pStmt, 0), n, n*100.0/nDoc);
}
sqlite3_finalize(pStmt);
end_vocab:
runSql(db, "ROLLBACK");
sqlite3_free(zAux);
}
/*
** Report on the number and sizes of segments
*/
static void showSegmentStats(sqlite3 *db, const char *zTab){
sqlite3_stmt *pStmt;
int nSeg = 0;
sqlite3_int64 szSeg = 0, mxSeg = 0;
int nIdx = 0;
sqlite3_int64 szIdx = 0, mxIdx = 0;
int nRoot = 0;
sqlite3_int64 szRoot = 0, mxRoot = 0;
sqlite3_int64 mx;
int nLeaf;
int n;
int pgsz;
int mxLevel;
int i;
pStmt = prepare(db,
"SELECT count(*), sum(length(block)), max(length(block))"
" FROM '%q_segments'",
zTab);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
nSeg = sqlite3_column_int(pStmt, 0);
szSeg = sqlite3_column_int64(pStmt, 1);
mxSeg = sqlite3_column_int64(pStmt, 2);
}
sqlite3_finalize(pStmt);
pStmt = prepare(db,
"SELECT count(*), sum(length(block)), max(length(block))"
" FROM '%q_segments' a JOIN '%q_segdir' b"
" WHERE a.blockid BETWEEN b.leaves_end_block+1 AND b.end_block",
zTab, zTab);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
nIdx = sqlite3_column_int(pStmt, 0);
szIdx = sqlite3_column_int64(pStmt, 1);
mxIdx = sqlite3_column_int64(pStmt, 2);
}
sqlite3_finalize(pStmt);
pStmt = prepare(db,
"SELECT count(*), sum(length(root)), max(length(root))"
" FROM '%q_segdir'",
zTab);
while( sqlite3_step(pStmt)==SQLITE_ROW ){
nRoot = sqlite3_column_int(pStmt, 0);
szRoot = sqlite3_column_int64(pStmt, 1);
mxRoot = sqlite3_column_int64(pStmt, 2);
}
sqlite3_finalize(pStmt);
printf("Number of segments....................... %9d\n", nSeg+nRoot);
printf("Number of leaf segments.................. %9d\n", nSeg-nIdx);
printf("Number of index segments................. %9d\n", nIdx);
printf("Number of root segments.................. %9d\n", nRoot);
printf("Total size of all segments............... %9lld\n", szSeg+szRoot);
printf("Total size of all leaf segments.......... %9lld\n", szSeg-szIdx);
printf("Total size of all index segments......... %9lld\n", szIdx);
printf("Total size of all root segments.......... %9lld\n", szRoot);
if( nSeg>0 ){
printf("Average size of all segments............. %11.1f\n",
(double)(szSeg+szRoot)/(double)(nSeg+nRoot));
printf("Average size of leaf segments............ %11.1f\n",
(double)(szSeg-szIdx)/(double)(nSeg-nIdx));
}
if( nIdx>0 ){
printf("Average size of index segments........... %11.1f\n",
(double)szIdx/(double)nIdx);
}
if( nRoot>0 ){
printf("Average size of root segments............ %11.1f\n",
(double)szRoot/(double)nRoot);
}
mx = mxSeg;
if( mx<mxRoot ) mx = mxRoot;
printf("Maximum segment size..................... %9lld\n", mx);
printf("Maximum index segment size............... %9lld\n", mxIdx);
printf("Maximum root segment size................ %9lld\n", mxRoot);
pStmt = prepare(db, "PRAGMA page_size");
pgsz = 1024;
while( sqlite3_step(pStmt)==SQLITE_ROW ){
pgsz = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
printf("Database page size....................... %9d\n", pgsz);
pStmt = prepare(db,
"SELECT count(*)"
" FROM '%q_segments' a JOIN '%q_segdir' b"
" WHERE a.blockid BETWEEN b.start_block AND b.leaves_end_block"
" AND length(a.block)>%d",
zTab, zTab, pgsz-45);
n = 0;
while( sqlite3_step(pStmt)==SQLITE_ROW ){
n = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
nLeaf = nSeg - nIdx;
printf("Leaf segments larger than %5d bytes.... %9d %5.2f%%\n",
pgsz-45, n, nLeaf>0 ? n*100.0/nLeaf : 0.0);
pStmt = prepare(db, "SELECT max(level%%1024) FROM '%q_segdir'", zTab);
mxLevel = 0;
while( sqlite3_step(pStmt)==SQLITE_ROW ){
mxLevel = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
for(i=0; i<=mxLevel; i++){
pStmt = prepare(db,
"SELECT count(*), sum(len), avg(len), max(len), sum(len>%d),"
" count(distinct idx)"
" FROM (SELECT length(a.block) AS len, idx"
" FROM '%q_segments' a JOIN '%q_segdir' b"
" WHERE (a.blockid BETWEEN b.start_block"
" AND b.leaves_end_block)"
" AND (b.level%%1024)==%d)",
pgsz-45, zTab, zTab, i);
if( sqlite3_step(pStmt)==SQLITE_ROW
&& (nLeaf = sqlite3_column_int(pStmt, 0))>0
){
sqlite3_int64 sz;
nIdx = sqlite3_column_int(pStmt, 5);
printf("For level %d:\n", i);
printf(" Number of indexes...................... %9d\n", nIdx);
printf(" Number of leaf segments................ %9d\n", nLeaf);
if( nIdx>1 ){
printf(" Average leaf segments per index........ %11.1f\n",
(double)nLeaf/(double)nIdx);
}
printf(" Total size of all leaf segments........ %9lld\n",
(sz = sqlite3_column_int64(pStmt, 1)));
printf(" Average size of leaf segments.......... %11.1f\n",
sqlite3_column_double(pStmt, 2));
if( nIdx>1 ){
printf(" Average leaf segment size per index.... %11.1f\n",
(double)sz/(double)nIdx);
}
printf(" Maximum leaf segment size.............. %9lld\n",
sqlite3_column_int64(pStmt, 3));
n = sqlite3_column_int(pStmt, 4);
printf(" Leaf segments larger than %5d bytes.. %9d %5.2f%%\n",
pgsz-45, n, n*100.0/nLeaf);
}
sqlite3_finalize(pStmt);
}
}
/*
** Print a single "tree" line of the segdir map output.
*/
static void printTreeLine(sqlite3_int64 iLower, sqlite3_int64 iUpper){
printf(" tree %9lld", iLower);
if( iUpper>iLower ){
printf(" thru %9lld (%lld blocks)", iUpper, iUpper-iLower+1);
}
printf("\n");
}
/*
** Check to see if the block of a %_segments entry is NULL.
*/
static int isNullSegment(sqlite3 *db, const char *zTab, sqlite3_int64 iBlockId){
sqlite3_stmt *pStmt;
int rc = 1;
pStmt = prepare(db, "SELECT block IS NULL FROM '%q_segments'"
" WHERE blockid=%lld", zTab, iBlockId);
if( sqlite3_step(pStmt)==SQLITE_ROW ){
rc = sqlite3_column_int(pStmt, 0);
}
sqlite3_finalize(pStmt);
return rc;
}
/*
** Show a map of segments derived from the %_segdir table.
*/
static void showSegdirMap(sqlite3 *db, const char *zTab){
int mxIndex, iIndex;
sqlite3_stmt *pStmt = 0;
sqlite3_stmt *pStmt2 = 0;
int prevLevel;
pStmt = prepare(db, "SELECT max(level/1024) FROM '%q_segdir'", zTab);
if( sqlite3_step(pStmt)==SQLITE_ROW ){
mxIndex = sqlite3_column_int(pStmt, 0);
}else{
mxIndex = 0;
}
sqlite3_finalize(pStmt);
printf("Number of inverted indices............... %3d\n", mxIndex+1);
pStmt = prepare(db,
"SELECT level, idx, start_block, leaves_end_block, end_block, rowid"
" FROM '%q_segdir'"
" WHERE level/1024==?"
" ORDER BY level DESC, idx",
zTab);
pStmt2 = prepare(db,
"SELECT blockid FROM '%q_segments'"
" WHERE blockid BETWEEN ? AND ? ORDER BY blockid",
zTab);
for(iIndex=0; iIndex<=mxIndex; iIndex++){
if( mxIndex>0 ){
printf("**************************** Index %d "
"****************************\n", iIndex);
}
sqlite3_bind_int(pStmt, 1, iIndex);
prevLevel = -1;
while( sqlite3_step(pStmt)==SQLITE_ROW ){
int iLevel = sqlite3_column_int(pStmt, 0)%1024;
int iIdx = sqlite3_column_int(pStmt, 1);
sqlite3_int64 iStart = sqlite3_column_int64(pStmt, 2);
sqlite3_int64 iLEnd = sqlite3_column_int64(pStmt, 3);
sqlite3_int64 iEnd = sqlite3_column_int64(pStmt, 4);
char rtag[20];
if( iLevel!=prevLevel ){
printf("level %2d idx %2d", iLevel, iIdx);
prevLevel = iLevel;
}else{
printf(" idx %2d", iIdx);
}
sqlite3_snprintf(sizeof(rtag), rtag, "r%lld",
sqlite3_column_int64(pStmt,5));
printf(" root %9s\n", rtag);
if( iLEnd>iStart ){
sqlite3_int64 iLower, iPrev = 0, iX;
if( iLEnd+1<=iEnd ){
sqlite3_bind_int64(pStmt2, 1, iLEnd+1);
sqlite3_bind_int64(pStmt2, 2, iEnd);
iLower = -1;
while( sqlite3_step(pStmt2)==SQLITE_ROW ){
iX = sqlite3_column_int64(pStmt2, 0);
if( iLower<0 ){
iLower = iPrev = iX;
}else if( iX==iPrev+1 ){
iPrev = iX;
}else{
printTreeLine(iLower, iPrev);
iLower = iPrev = iX;
}
}
sqlite3_reset(pStmt2);
if( iLower>=0 ){
if( iLower==iPrev && iLower==iEnd
&& isNullSegment(db,zTab,iLower)
){
printf(" null %9lld\n", iLower);
}else{
printTreeLine(iLower, iPrev);
}
}
}
printf(" leaves %9lld thru %9lld (%lld blocks)\n",
iStart, iLEnd, iLEnd - iStart + 1);
}
}
sqlite3_reset(pStmt);
}
sqlite3_finalize(pStmt);
sqlite3_finalize(pStmt2);
}
/*
** Decode a single segment block and display the results on stdout.
*/
static void decodeSegment(
const unsigned char *aData, /* Content to print */
int nData /* Number of bytes of content */
){
sqlite3_int64 iChild = 0;
sqlite3_int64 iPrefix;
sqlite3_int64 nTerm;
sqlite3_int64 n;
sqlite3_int64 iDocsz;
int iHeight;
sqlite3_int64 i = 0;
int cnt = 0;
char zTerm[1000];
i += getVarint(aData, &n);
iHeight = (int)n;
printf("height: %d\n", iHeight);
if( iHeight>0 ){
i += getVarint(aData+i, &iChild);
printf("left-child: %lld\n", iChild);
}
while( i<nData ){
if( (cnt++)>0 ){
i += getVarint(aData+i, &iPrefix);
}else{
iPrefix = 0;
}
i += getVarint(aData+i, &nTerm);
if( iPrefix+nTerm+1 >= sizeof(zTerm) ){
fprintf(stderr, "term to long\n");
exit(1);
}
memcpy(zTerm+iPrefix, aData+i, (size_t)nTerm);
zTerm[iPrefix+nTerm] = 0;
i += nTerm;
if( iHeight==0 ){
i += getVarint(aData+i, &iDocsz);
printf("term: %-25s doclist %7lld bytes offset %lld\n", zTerm, iDocsz, i);
i += iDocsz;
}else{
printf("term: %-25s child %lld\n", zTerm, ++iChild);
}
}
}
/*
** Print a a blob as hex and ascii.
*/
static void printBlob(
const unsigned char *aData, /* Content to print */
int nData /* Number of bytes of content */
){
int i, j;
const char *zOfstFmt;
const int perLine = 16;
if( (nData&~0xfff)==0 ){
zOfstFmt = " %03x: ";
}else if( (nData&~0xffff)==0 ){
zOfstFmt = " %04x: ";
}else if( (nData&~0xfffff)==0 ){
zOfstFmt = " %05x: ";
}else if( (nData&~0xffffff)==0 ){
zOfstFmt = " %06x: ";
}else{
zOfstFmt = " %08x: ";
}
for(i=0; i<nData; i += perLine){
fprintf(stdout, zOfstFmt, i);
for(j=0; j<perLine; j++){
if( i+j>nData ){
fprintf(stdout, " ");
}else{
fprintf(stdout,"%02x ", aData[i+j]);
}
}
for(j=0; j<perLine; j++){
if( i+j>nData ){
fprintf(stdout, " ");
}else{
fprintf(stdout,"%c", isprint(aData[i+j]) ? aData[i+j] : '.');
}
}
fprintf(stdout,"\n");
}
}
/*
** Convert text to a 64-bit integer
*/
static sqlite3_int64 atoi64(const char *z){
sqlite3_int64 v = 0;
while( z[0]>='0' && z[0]<='9' ){
v = v*10 + z[0] - '0';
z++;
}
return v;
}
/*
** Return a prepared statement which, when stepped, will return in its
** first column the blob associated with segment zId. If zId begins with
** 'r' then it is a rowid of a %_segdir entry. Otherwise it is a
** %_segment entry.
*/
static sqlite3_stmt *prepareToGetSegment(
sqlite3 *db, /* The database */
const char *zTab, /* The FTS3/4 table name */
const char *zId /* ID of the segment to open */
){
sqlite3_stmt *pStmt;
if( zId[0]=='r' ){
pStmt = prepare(db, "SELECT root FROM '%q_segdir' WHERE rowid=%lld",
zTab, atoi64(zId+1));
}else{
pStmt = prepare(db, "SELECT block FROM '%q_segments' WHERE blockid=%lld",
zTab, atoi64(zId));
}
return pStmt;
}
/*
** Print the content of a segment or of the root of a segdir. The segment
** or root is identified by azExtra[0]. If the first character of azExtra[0]
** is 'r' then the remainder is the integer rowid of the %_segdir entry.
** If the first character of azExtra[0] is not 'r' then, then all of
** azExtra[0] is an integer which is the block number.
**
** If the --raw option is present in azExtra, then a hex dump is provided.
** Otherwise a decoding is shown.
*/
static void showSegment(sqlite3 *db, const char *zTab){
const unsigned char *aData;
int nData;
sqlite3_stmt *pStmt;
pStmt = prepareToGetSegment(db, zTab, azExtra[0]);
if( sqlite3_step(pStmt)!=SQLITE_ROW ){
sqlite3_finalize(pStmt);
return;
}
nData = sqlite3_column_bytes(pStmt, 0);
aData = sqlite3_column_blob(pStmt, 0);
printf("Segment %s of size %d bytes:\n", azExtra[0], nData);
if( findOption("raw", 0, 0)!=0 ){
printBlob(aData, nData);
}else{
decodeSegment(aData, nData);
}
sqlite3_finalize(pStmt);
}
/*
** Decode a single doclist and display the results on stdout.
*/
static void decodeDoclist(
const unsigned char *aData, /* Content to print */
int nData /* Number of bytes of content */
){
sqlite3_int64 iPrevDocid = 0;
sqlite3_int64 iDocid;
sqlite3_int64 iPos;
sqlite3_int64 iPrevPos = 0;
sqlite3_int64 iCol;
int i = 0;
while( i<nData ){
i += getVarint(aData+i, &iDocid);
printf("docid %lld col0", iDocid+iPrevDocid);
iPrevDocid += iDocid;
iPrevPos = 0;
while( 1 ){
i += getVarint(aData+i, &iPos);
if( iPos==1 ){
i += getVarint(aData+i, &iCol);
printf(" col%lld", iCol);
iPrevPos = 0;
}else if( iPos==0 ){
printf("\n");
break;
}else{
iPrevPos += iPos - 2;
printf(" %lld", iPrevPos);
}
}
}
}
/*
** Print the content of a doclist. The segment or segdir-root is
** identified by azExtra[0]. If the first character of azExtra[0]
** is 'r' then the remainder is the integer rowid of the %_segdir entry.
** If the first character of azExtra[0] is not 'r' then, then all of
** azExtra[0] is an integer which is the block number. The offset
** into the segment is identified by azExtra[1]. The size of the doclist
** is azExtra[2].
**
** If the --raw option is present in azExtra, then a hex dump is provided.
** Otherwise a decoding is shown.
*/
static void showDoclist(sqlite3 *db, const char *zTab){
const unsigned char *aData;
sqlite3_int64 offset;
int nData;
sqlite3_stmt *pStmt;
offset = atoi64(azExtra[1]);
nData = atoi(azExtra[2]);
pStmt = prepareToGetSegment(db, zTab, azExtra[0]);
if( sqlite3_step(pStmt)!=SQLITE_ROW ){
sqlite3_finalize(pStmt);
return;
}
aData = sqlite3_column_blob(pStmt, 0);
printf("Doclist at %s offset %lld of size %d bytes:\n",
azExtra[0], offset, nData);
if( findOption("raw", 0, 0)!=0 ){
printBlob(aData+offset, nData);
}else{
decodeDoclist(aData+offset, nData);
}
sqlite3_finalize(pStmt);
}
/*
** Show the top N largest segments
*/
static void listBigSegments(sqlite3 *db, const char *zTab){
int nTop, i;
sqlite3_stmt *pStmt;
sqlite3_int64 sz;
sqlite3_int64 id;
nTop = atoi(findOption("top", 1, "25"));
printf("The %d largest segments:\n", nTop);
pStmt = prepare(db,
"SELECT blockid, length(block) AS len FROM '%q_segments'"
" ORDER BY 2 DESC, 1"
" LIMIT %d", zTab, nTop);
i = 0;
while( sqlite3_step(pStmt)==SQLITE_ROW ){
i++;
id = sqlite3_column_int64(pStmt, 0);
sz = sqlite3_column_int64(pStmt, 1);
printf(" %2d. %9lld size %lld\n", i, id, sz);
}
sqlite3_finalize(pStmt);
}
static void usage(const char *argv0){
fprintf(stderr, "Usage: %s DATABASE\n"
" or: %s DATABASE FTS3TABLE ARGS...\n", argv0, argv0);
fprintf(stderr,
"ARGS:\n"
" big-segments [--top N] show the largest segments\n"
" doclist BLOCKID OFFSET SIZE [--raw] Decode a doclist\n"
" schema FTS table schema\n"
" segdir directory of segments\n"
" segment BLOCKID [--raw] content of a segment\n"
" segment-stats info on segment sizes\n"
" stat the %%_stat table\n"
" vocabulary [--top N] document vocabulary\n"
);
exit(1);
}
int main(int argc, char **argv){
sqlite3 *db;
int rc;
const char *zTab;
const char *zCmd;
if( argc<2 ) usage(argv[0]);
rc = sqlite3_open(argv[1], &db);
if( rc ){
fprintf(stderr, "Cannot open %s\n", argv[1]);
exit(1);
}
if( argc==2 ){
sqlite3_stmt *pStmt;
int cnt = 0;
pStmt = prepare(db, "SELECT b.sql"
" FROM sqlite_schema a, sqlite_schema b"
" WHERE a.name GLOB '*_segdir'"
" AND b.name=substr(a.name,1,length(a.name)-7)"
" ORDER BY 1");
while( sqlite3_step(pStmt)==SQLITE_ROW ){
cnt++;
printf("%s;\n", sqlite3_column_text(pStmt, 0));
}
sqlite3_finalize(pStmt);
if( cnt==0 ){
printf("/* No FTS3/4 tables found in database %s */\n", argv[1]);
}
return 0;
}
if( argc<4 ) usage(argv[0]);
zTab = argv[2];
zCmd = argv[3];
nExtra = argc-4;
azExtra = argv+4;
if( strcmp(zCmd,"big-segments")==0 ){
listBigSegments(db, zTab);
}else if( strcmp(zCmd,"doclist")==0 ){
if( argc<7 ) usage(argv[0]);
showDoclist(db, zTab);
}else if( strcmp(zCmd,"schema")==0 ){
showSchema(db, zTab);
}else if( strcmp(zCmd,"segdir")==0 ){
showSegdirMap(db, zTab);
}else if( strcmp(zCmd,"segment")==0 ){
if( argc<5 ) usage(argv[0]);
showSegment(db, zTab);
}else if( strcmp(zCmd,"segment-stats")==0 ){
showSegmentStats(db, zTab);
}else if( strcmp(zCmd,"stat")==0 ){
showStat(db, zTab);
}else if( strcmp(zCmd,"vocabulary")==0 ){
showVocabulary(db, zTab);
}else{
usage(argv[0]);
}
return 0;
}
ext/fts3/unicode/CaseFolding.txt
+1224
View File
File diff suppressed because it is too large. Load diff
Loaded 100 of 2208 files, more files were not shown because too many files have changed in this diff. Show more