*** DRAFT ***
Command Line Shell For SQLite

# 1. Getting Started

The SQLite project provides a simple command-line utility named sqlite3 (or sqlite3.exe on Windows) that allows the user to manually enter and execute SQL statements against an SQLite database. This document provides a brief introduction on how to use the sqlite3 program.

To start the sqlite3 program, simply type "sqlite3" at the command prompt. The "sqlite3" command may be optionally followed by the name the file that holds the SQLite database. If the file does not exist, a new database file with the given name will be created automatically. If no database file is specified on the command-line, a temporary database is created, then deleted when the "sqlite3" program exits.

On startup, the sqlite3 program will show a brief banner message then prompt you to enter SQL. Type in SQL statements (terminated by a semicolon), press "Enter" and the SQL will be executed.

For example, to create a new SQLite database named "ex1" with a single table named "tbl1", you might do this:

    $ sqlite3 ex1
    SQLite version 3.8.5 2014-05-29 12:36:14
    Enter ".help" for usage hints.
    sqlite> create table tbl1(one varchar(10), two smallint);
    sqlite> insert into tbl1 values('hello!',10);
    sqlite> insert into tbl1 values('goodbye', 20);
    sqlite> select * from tbl1;
    hello!|10
    goodbye|20
    sqlite>

You can terminate the sqlite3 program by typing your system End-Of-File character (usually a Control-D). Use the interrupt character (usually a Control-C) to stop a long-running SQL statement.

Make sure you type a semicolon at the end of each SQL command! The sqlite3 program looks for a semicolon to know when your SQL command is complete. If you omit the semicolon, sqlite3 will give you a continuation prompt and wait for you to enter more text to be added to the current SQL command. This feature allows you to enter SQL commands that span multiple lines. For example:

    sqlite> CREATE TABLE tbl2 (
       ...>   f1 varchar(30) primary key,
       ...>   f2 text,
       ...>   f3 real
       ...> );
    sqlite>

# 2. Double-click Startup On Windows

Windows users can double-click on the sqlite3.exe icon to cause the command-line shell to pop-up a terminal window running SQLite. However, because double-clicking starts the sqlite3.exe without command-line arguments, no database file will have been specified, so SQLite will use a temporary database that is deleted when the session exits. To use a persistent disk file as the database, enter the ".open" command immediately after the terminal window starts up:

    SQLite version 3.8.5 2014-05-29 12:36:14
    Enter ".help" for usage hints.
    Connected to a transient in-memory database.
    Use ".open FILENAME" to reopen on a persistent database.
    sqlite> .open ex1.db
    sqlite>

The example above causes the database file named "ex1.db" to be opened and used, and created if it does not previously exist. You might want to use a full pathname to ensure that the file is in the directory that you think it is in. Use forward-slashes as the directory separator character. In other words use "c:/work/ex1.db", not "c:\work\ex1.db".

Alternatively, you can create a new database using the default temporary storage, then save that database into a disk file using the ".save" command:

    SQLite version 3.16.0 2016-12-29 19:48:46
    Enter ".help" for usage hints.
    Connected to a transient in-memory database.
    Use ".open FILENAME" to reopen on a persistent database.
    sqlite> ... many SQL commands omitted ...
    sqlite> .save ex1.db
    sqlite>

Be careful when using the ".save" command as it will overwrite any preexisting database files having the same name without prompting for confirmation. As with the ".open" command, you might want to use a full pathname with forward-slash directory separators to avoid ambiguity.

# 3. Special commands to sqlite3 (dot-commands)

Most of the time, sqlite3 just reads lines of input and passes them on to the SQLite library for execution. But input lines that begin with a dot (".") are intercepted and interpreted by the sqlite3 program itself. These "dot commands" are typically used to change the output format of queries, or to execute certain prepackaged query statements.

For a listing of the available dot commands, you can enter ".help" at any time. For example:

    sqlite> .help
    .auth ON|OFF           Show authorizer callbacks
    .backup ?DB? FILE      Backup DB (default "main") to FILE
    .bail on|off           Stop after hitting an error.  Default OFF
    .binary on|off         Turn binary output on or off.  Default OFF
    .cd DIRECTORY          Change the working directory to DIRECTORY
    .changes on|off        Show number of rows changed by SQL
    .check GLOB            Fail if output since .testcase does not match
    .clone NEWDB           Clone data into NEWDB from the existing database
    .databases             List names and files of attached databases
    .dbinfo ?DB?           Show status information about the database
    .dump ?TABLE? ...      Dump the database in an SQL text format
                             If TABLE specified, only dump tables matching
                             LIKE pattern TABLE.
    .echo on|off           Turn command echo on or off
    .eqp on|off|full       Enable or disable automatic EXPLAIN QUERY PLAN
    .exit                  Exit this program
    .fullschema ?--indent? Show schema and the content of sqlite_stat tables
    .headers on|off        Turn display of headers on or off
    .help                  Show this message
    .import FILE TABLE     Import data from FILE into TABLE
    .imposter INDEX TABLE  Create imposter table TABLE on index INDEX
    .indexes ?TABLE?       Show names of all indexes
                             If TABLE specified, only show indexes for tables
                             matching LIKE pattern TABLE.
    .iotrace FILE          Enable I/O diagnostic logging to FILE
    .limit ?LIMIT? ?VAL?   Display or change the value of an SQLITE_LIMIT
    .lint OPTIONS          Report potential schema issues. Options:
                             fkey-indexes     Find missing foreign key indexes
    .load FILE ?ENTRY?     Load an extension library
    .log FILE|off          Turn logging on or off.  FILE can be stderr/stdout
    .mode MODE ?TABLE?     Set output mode where MODE is one of:
                             ascii    Columns/rows delimited by 0x1F and 0x1E
                             csv      Comma-separated values
                             column   Left-aligned columns.  (See .width)
                             html     HTML <table> code
                             insert   SQL insert statements for TABLE
                             line     One value per line
                             list     Values delimited by "|"
                             quote    Escape answers as for SQL
                             tabs     Tab-separated values
                             tcl      TCL list elements
    .nullvalue STRING      Use STRING in place of NULL values
    .once FILENAME         Output for the next SQL command only to FILENAME
    .open ?OPTIONS? ?FILE? Close existing database and reopen FILE
                             The --new option starts with an empty file
    .output ?FILENAME?     Send output to FILENAME or stdout
    .print STRING...       Print literal STRING
    .prompt MAIN CONTINUE  Replace the standard prompts
    .quit                  Exit this program
    .read FILENAME         Execute SQL in FILENAME
    .restore ?DB? FILE     Restore content of DB (default "main") from FILE
    .save FILE             Write in-memory database into FILE
    .scanstats on|off      Turn sqlite3_stmt_scanstatus() metrics on or off
    .schema ?PATTERN?      Show the CREATE statements matching PATTERN
                              Add --indent for pretty-printing
    .selftest ?--init?     Run tests defined in the SELFTEST table
    .separator COL ?ROW?   Change the column separator and optionally the row
                             separator for both the output mode and .import
    .session CMD ...       Create or control sessions
    .sha3sum ?OPTIONS...?  Compute a SHA3 hash of database content
    .shell CMD ARGS...     Run CMD ARGS... in a system shell
    .show                  Show the current values for various settings
    .stats ?on|off?        Show stats or turn stats on or off
    .system CMD ARGS...    Run CMD ARGS... in a system shell
    .tables ?TABLE?        List names of tables
                             If TABLE specified, only list tables matching
                             LIKE pattern TABLE.
    .testcase NAME         Begin redirecting output to 'testcase-out.txt'
    .timeout MS            Try opening locked tables for MS milliseconds
    .timer on|off          Turn SQL timer on or off
    .trace FILE|off        Output each SQL statement as it is run
    .vfsinfo ?AUX?         Information about the top-level VFS
    .vfslist               List all available VFSes
    .vfsname ?AUX?         Print the name of the VFS stack
    .width NUM1 NUM2 ...   Set column widths for "column" mode
                             Negative values right-justify
    sqlite>

# 4. Rules for "dot-commands"

Ordinary SQL statements are free-form, and can be spread across multiple lines, and can have whitespace and comments anywhere. Dot-commands are more restrictive:

- A dot-command must begin with the "." at the left margin with no preceding whitespace.
- The dot-command must be entirely contained on a single input line.
- A dot-command cannot occur in the middle of an ordinary SQL statement. In other words, a dot-command cannot occur at a continuation prompt.
- Dot-commands do not recognize comments.

The dot-commands are interpreted by the sqlite3.exe command-line program, not by SQLite itself. So none of the dot-commands will work as an argument to SQLite interfaces like sqlite3_prepare() or sqlite3_exec().

# 5. Changing Output Formats

The sqlite3 program is able to show the results of a query in eight different formats: "csv", "column", "html", "insert", "line", "list", "quote", "tabs", and "tcl". You can use the ".mode" dot command to switch between these output formats.

The default output mode is "list". In list mode, each row of a query result is written on one line of output and each column within that row is separated by a specific separator string. The default separator is a pipe symbol ("|"). List mode is especially useful when you are going to send the output of a query to another program (such as AWK) for additional processing.

    sqlite> .mode list
    sqlite> select * from tbl1;
    hello|10
    goodbye|20
    sqlite>

Use the ".separator" dot command to change the separator. For example, to change the separator to a comma and a space, you could do this:

    sqlite> .separator ", "
    sqlite> select * from tbl1;
    hello, 10
    goodbye, 20
    sqlite>

The next ".mode" command will reset the ".separator" back to its default. So you will need repeat the ".separator" command whenever you change modes if you want to continue using a non-standard separator.

In "quote" mode, the output is formatted as SQL literals. Strings are enclosed in single-quotes and internal single-quotes are escaped by doubling. Blobs are displayed in hexadecimal blob literal notation (Ex: x'abcd'). Numbers are displayed as ASCII text and NULL values are shown as "NULL". All columns are separated from each other by a comma (or whatever alternative character is selected using ".separator").

    sqlite> .mode quote
    sqlite> select * from tbl1;
    'hello',10
    'goodbye',20
    sqlite>

In "line" mode, each column in a row of the database is shown on a line by itself. Each line consists of the column name, an equal sign and the column data. Successive records are separated by a blank line. Here is an example of line mode output:

    sqlite> .mode line
    sqlite> select * from tbl1;
    one = hello
    two = 10

    one = goodbye
    two = 20
    sqlite>

In column mode, each record is shown on a separate line with the data aligned in columns. For example:

    sqlite> .mode column
    sqlite> select * from tbl1;
    one         two
    ----------  ----------
    hello       10
    goodbye     20
    sqlite>

By default, each column is between 1 and 10 characters wide, depending on the column header name and the width of the first column of data. Data that is too wide to fit in a column is truncated. Use the ".width" dot-command to adjust column widths, like this:

    sqlite> .width 12 6
    sqlite> select * from tbl1;
    one           two
    ------------  ------
    hello         10
    goodbye       20
    sqlite>

The ".width" command in the example above sets the width of the first column to 12 and the width of the second column to 6. All other column widths were unaltered. You can gives as many arguments to ".width" as necessary to specify the widths of as many columns as are in your query results.

If you specify a column a width of 0, then the column width is automatically adjusted to be the maximum of three numbers: 10, the width of the header, and the width of the first row of data. This makes the column width self-adjusting. The default width setting for every column is this auto-adjusting 0 value.

Use a negative column width for right-justified columns.

The column labels that appear on the first two lines of output can be turned on and off using the ".header" dot command. In the examples above, the column labels are on. To turn them off you could do this:

    sqlite> .header off
    sqlite> select * from tbl1;
    hello         10
    goodbye       20
    sqlite>

Another useful output mode is "insert". In insert mode, the output is formatted to look like SQL INSERT statements. Use insert mode to generate text that can later be used to input data into a different database.

When specifying insert mode, you have to give an extra argument which is the name of the table to be inserted into. For example:

    sqlite> .mode insert new_table
    sqlite> select * from tbl1;
    INSERT INTO "new_table" VALUES('hello',10);
    INSERT INTO "new_table" VALUES('goodbye',20);
    sqlite>

The last output mode is "html". In this mode, sqlite3 writes the results of the query as an XHTML table. The beginning <TABLE> and the ending </TABLE> are not written, but all of the intervening <TR>s, <TH>s, and <TD>s are. The html output mode is envisioned as being useful for CGI.

# 6. Writing results to a file

By default, sqlite3 sends query results to standard output. You can change this using the ".output" and ".once" commands. Just put the name of an output file as an argument to .output and all subsequent query results will be written to that file. Or use the .once command instead of .output and output will only be redirected for the single next command before reverting to the console. Use .output with no arguments to begin writing to standard output again. For example:

    sqlite> .mode list
    sqlite> .separator |
    sqlite> .output test_file_1.txt
    sqlite> select * from tbl1;
    sqlite> .exit
    $ cat test_file_1.txt
    hello|10
    goodbye|20
    $

If the first character of the ".output" or ".once" filename is a pipe symbol ("|") then the remaining characters are treated as a command and the output is sent to that command. This makes it easy to pipe the results of a query into some other process. For example, the "open -f" command on a Mac opens a text editor to display the content that it reads from standard input. So to see the results of a query in a text editor, one could type:

    sqlite3> .once '|open -f'
    sqlite3> SELECT * FROM bigTable;

## 6.1. File I/O Functions

The command-line shell adds two application-defined SQL functions that facilitate reading content from a file into a table column, and writing the content of a column into a file, respectively.

The readfile(X) SQL function reads the entire content of the file named X and returns that content as a BLOB. This can be used to load content into a table. For example:

    sqlite> CREATE TABLE images(name TEXT, type TEXT, img BLOB);
    sqlite> INSERT INTO images(name,type,img)
       ...>   VALUES('icon','jpeg',readfile('icon.jpg'));

The writefile(X,Y) SQL function write the blob Y into the file named X and returns the number of bytes written. Use this function to extract the content of a single table column into a file. For example:

    sqlite> SELECT writefile('icon.jpg',img) FROM images WHERE name='icon';

Note that the readfile(X) and writefile(X,Y) functions are extension functions and are not built into the core SQLite library. These routines are available as a loadable extension in the ext/misc/fileio.c source file in the SQLite source code repositories.

# 7. Querying the database schema

The sqlite3 program provides several convenience commands that are useful for looking at the schema of the database. There is nothing that these commands do that cannot be done by some other means. These commands are provided purely as a shortcut.

For example, to see a list of the tables in the database, you can enter ".tables".

    sqlite> .tables
    tbl1
    tbl2
    sqlite>

The ".tables" command is similar to setting list mode then executing the following query:

    SELECT name FROM sqlite_master
    WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'
    ORDER BY 1

But the ".tables" command does more. It queries the sqlite_master table for all attached databases, not just the primary database. And it arranges its output into neat columns.

The ".indexes" command works in a similar way to list all of the indexes. If the ".indexes" command is given an argument which is the name of a table, then it shows just indexes on that table.

The ".schema" command shows the complete schema for the database, or for a single table if an optional tablename argument is provided:

    sqlite> .schema
    create table tbl1(one varchar(10), two smallint)
    CREATE TABLE tbl2 (
      f1 varchar(30) primary key,
      f2 text,
      f3 real
    )
    sqlite> .schema tbl2
    CREATE TABLE tbl2 (
      f1 varchar(30) primary key,
      f2 text,
      f3 real
    )
    sqlite>

The ".schema" command is roughly the same as setting list mode, then entering the following query:

    SELECT sql FROM sqlite_master
    ORDER BY tbl_name, type DESC, name

As with ".tables", the ".schema" command shows the schema for all attached databases. If you only want to see the schema for a single database (perhaps "main") then you can add an argument to ".schema" to restrict its output:

    sqlite> .schema main.*

The ".schema" command can be augmented with the "--indent" option, in which case it tries to reformat the various CREATE statements of the schema so that they are more easily readable by humans.

The ".databases" command shows a list of all databases open in the current connection. There will always be at least 2. The first one is "main", the original database opened. The second is "temp", the database used for temporary tables. There may be additional databases listed for databases attached using the ATTACH statement. The first output column is the name the database is attached with, and the second column is the filename of the external file.

    sqlite> .databases

The ".fullschema" dot-command works like the ".schema" command in that it displays the entire database schema. But ".fullschema" also includes dumps of the statistics tables "sqlite_stat1", "sqlite_stat3", and "sqlite_stat4", if they exist. The ".fullschema" command normally provides all of the information needed to exactly recreate a query plan for a specific query. When reporting suspected problems with the SQLite query planner to the SQLite development team, developers are requested to provide the complete ".fullschema" output as part of the trouble report. Note that the sqlite_stat3 and sqlite_stat4 tables contain samples of index entries and so might contain sensitive data, so do not send the ".fullschema" output of a proprietary database over a public channel.

# 8. CSV Import

Use the ".import" command to import CSV (comma separated value) data into an SQLite table. The ".import" command takes two arguments which are the name of the disk file from which CSV data is to be read and the name of the SQLite table into which the CSV data is to be inserted.

Note that it is important to set the "mode" to "csv" before running the ".import" command. This is necessary to prevent the command-line shell from trying to interpret the input file text as some other format.

    sqlite> .mode csv
    sqlite> .import C:/work/somedata.csv tab1

There are two cases to consider: (1) Table "tab1" does not previously exist and (2) table "tab1" does already exist.

In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file.

For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, that row will be read as data and inserted into the table. To avoid this, make sure that table does not previously exist.

# 9. CSV Export

To export an SQLite table (or part of a table) as CSV, simply set the "mode" to "csv" and then run a query to extract the desired rows of the table.

    sqlite> .header on
    sqlite> .mode csv
    sqlite> .once c:/work/dataout.csv
    sqlite> SELECT * FROM tab1;
    sqlite> .system c:/work/dataout.csv

In the example above, the ".header on" line causes column labels to be printed as the first row of output. This means that the first row of the resulting CSV file will contain column labels. If column labels are not desired, set ".header off" instead. (The ".header off" setting is the default and can be omitted if the headers have not been previously turned on.)

The line ".once FILENAME" causes all query output to go into the named file instead of being printed on the console. In the example above, that line causes the CSV content to be written into a file named "C:/work/dataout.csv".

The final line of the example (the ".system c:/work/dataout.csv") has the same effect as double-clicking on the c:/work/dataout.csv file in windows. This will typically bring up a spreadsheet program to display the CSV file. That command only works as shown on Windows. The equivalent line on a Mac would be ".system open /work/dataout.csv". On Linux and other unix systems you will need to enter something like ".system libreoffice /work/dataout.csv", substituting your preferred CSV viewing program for "libreoffice".

# 10. Converting An Entire Database To An ASCII Text File

Use the ".dump" command to convert the entire contents of a database into a single ASCII text file. This file can be converted back into a database by piping it back into sqlite3.

A good way to make an archival copy of a database is this:

    $ sqlite3 ex1 .dump | gzip -c >ex1.dump.gz

This generates a file named ex1.dump.gz that contains everything you need to reconstruct the database at a later time, or on another machine. To reconstruct the database, just type:

    $ zcat ex1.dump.gz | sqlite3 ex2

The text format is pure SQL so you can also use the .dump command to export an SQLite database into other popular SQL database engines. Like this:

    $ createdb ex2
    $ sqlite3 ex1 .dump | psql ex2

# 11. Loading Extensions

You can add new custom application-defined SQL functions, collating sequences, virtual tables, and VFSes to the command-line shell at run-time using the ".load" command. First, convert the extension in to a DLL or shared library (as described in the Run-Time Loadable Extensions document) then type:

    sqlite> .load /path/to/my_extension

Note that SQLite automatically adds the appropriate extension suffix (".dll" on windows, ".dylib" on Mac, ".so" on most other unixes) to the extension filename. It is generally a good idea to specify the full pathname of the extension.

SQLite computes the entry point for the extension based on the extension filename. To override this choice, simply add the name of the extension as a second argument to the ".load" command.

Source code for several useful extensions can be found in the ext/misc subdirectory of the SQLite source tree. You can use these extensions as-is, or as a basis for creating your own custom extensions to address your own particular needs.

# 12. Cryptographic Hashes Of Database Content

The ".sha3sum" dot-command computes a SHA3 hash of the content of the database. To be clear, the hash is computed over the database content, not its representation on disk. This means, for example, that a VACUUM or similar data-preserving transformation does not change the hash.

The ".sha3sum" command supports options "--sha3-224", "--sha3-256", "--sha3-384", and "--sha3-512" to define which variety of SHA3 to use for the hash. The default is SHA3-256.

The database schema (in the sqlite_master table) is not normally included in the hash, but can be added by the "--schema" option.

The ".sha3sum" command takes a single optional argument which is a LIKE pattern. If this option is present, only tables whose names match the LIKE pattern will be hashed.

The ".sha3sum" command is implemented with the help of the extension function "sha3_query()" that is included with the command-line shell.

# 13. Database Content Self-Tests

The ".selftest" command attempts to verify that a database is intact and is not corrupt. The .selftest command looks for a table in schema named "selftest" and defined as follows:

    CREATE TABLE selftest(
      tno INTEGER PRIMARY KEY,  -- Test number
      op TEXT,                  -- 'run' or 'memo'
      cmd TEXT,                 -- SQL command to run, or text of "memo"
      ans TEXT                  -- Expected result of the SQL command
    );

The .selftest command reads the rows of the selftest table in selftest.tno order. For each 'memo' row, it writes the text in 'cmd' to the output. For each 'run' row, it runs the 'cmd' text as SQL and compares the result to the value in 'ans', and shows an error message if the results differ.

If there is no selftest table, the ".selftest" command runs PRAGMA integrity_check.

The ".selftest --init" command creates the selftest table if it does not already exists, then appends entries that check the SHA3 hash of the content of all tables. Subsequent runs of ".selftest" will verify that the database has not been changed in any way. To generates tests to verify that a subset of the tables are unchanged, simply run ".selftest --init" then DELETE the selftest rows that refer to tables that are not constant.

# 14. Other Dot Commands

There are many other dot-commands available in the command-line shell. See the ".help" command for a complete list for any particular version and build of SQLite.

# 15. Using sqlite3 in a shell script

One way to use sqlite3 in a shell script is to use "echo" or "cat" to generate a sequence of commands in a file, then invoke sqlite3 while redirecting input from the generated command file. This works fine and is appropriate in many circumstances. But as an added convenience, sqlite3 allows a single SQL command to be entered on the command line as a second argument after the database name. When the sqlite3 program is launched with two arguments, the second argument is passed to the SQLite library for processing, the query results are printed on standard output in list mode, and the program exits. This mechanism is designed to make sqlite3 easy to use in conjunction with programs like "awk". For example:

    $ sqlite3 ex1 'select * from tbl1' |
    >  awk '{printf "<tr><td>%s<td>%s\n",$1,$2 }'
    <tr><td>hello<td>10
    <tr><td>goodbye<td>20
    $

# 16. Ending shell commands

SQLite commands are normally terminated by a semicolon. In the shell you can also use the word "GO" (case-insensitive) or a slash character "/" on a line by itself to end a command. These are used by SQL Server and Oracle, respectively. These won't work in sqlite3_exec(), because the shell translates these into a semicolon before passing them to that function.

# 17. Compiling the sqlite3 program from sources

The source code to the sqlite3 command line interface is in a single file named "shell.c" which you can download from the SQLite website. Compile this file (together with the sqlite3 library source code) to generate the executable. For example:

    gcc -o sqlite3 shell.c sqlite3.c -ldl -lpthread

You might want to provide additional compile-time options:

- -DSQLITE_THREADSAFE=0: The command-line shell does no use threads, so you might as well turn thread-safety off, for improved performance and a reduced code footprint. This also eliminates the need for the "-lpthread" switch on the compiler command line.

- -DSQLITE_ENABLE_EXPLAIN_COMMENT: This option adds comment text to the end of each opcode in the output of EXPLAIN. The EXPLAIN output is much easier to read with this option enabled.

- -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION: This option allows EXPLAIN statements to succeed even if the contain references to unknown SQL functions. This makes the shell useful for looking at EXPLAIN or EXPLAIN QUERY PLAN output for queries pasted out of applications that define and use custom SQL functions.

*** DRAFT ***