SQLite a.k.a. SQLite3
(Specifically, this page is about my role in the sqlite ecosystem, not about how to use sqlite or some such.)
sqlite3 is the world's most widely-deployed database engine and is widely believed to contend for first place as the single most widely-deployed piece of software in the world (zlib being the other most likely candidate).
And So it Begins...
Since 2008 i have had the pleasure of working with sqlite's creator, Richard Hipp, via his Fossil SCM project and, in recent years, in much more limited capacities on other pet projects of his such as the althttpd web server and the pikchr "declarative diagram description" tool (for lack of a better description). (Althttpd is, not coincidentally, the web server which runs all of the wanderinghorse.net websites.) More than a quarter of my life has been spent coding in and around that community.
Welcome to the Team!
In late 2021 i was "drafted" into the sqlite team, primarily in an administrative support role, e.g. assisting in administration of its public support forum in order to free Richard and Dan (the core devs) up from some of that workload. (Though i'm quite proficient in C, the relational algebra in sqlite's core code is well above my proverbial paygrade, and i've never had any delusions/aspirations of working on the sqlite internals.)
WebAssembly
In mid-2022 the project began to investigate the feasibility of using WebAssembly, in combination with with the (then-)up-and-coming Origin-Private FileSystem web storage technology, to bring persistent sqlite3 databases to the browsers. (A small number of bleeding edge folks had already been using sqlite3 via WASM for at least 3 years, but without any sort of persistence.) Since WebAssembly is closely associated with JavaScript, and i'm the Fossil SCM's de facto "JavaScript Guy," that new effort fell neatly into my lap with no objection on my part. Indeed, that's the type of thing i do for fun, so welcomed it.
Our first proof-of-concept was sqlite3 fiddle, an HTML interface for the sqlite3 shell app (which is implemented in C), all run in the browser (as opposed to having the database running on the server). The very first iteration of it took roughly half an hour to get running thanks to the deep voodoo of the Emscripten tools, and about 25 minutes of that time was obtaining, installing, and learning how to invoke Emscripten. The ease with which that could be created convinced us that WASM was definitely something we wanted to play more with.
Going in, we'd all heard the word "WASM" but none of us had ever investigated WASM. Within only a few weeks of getting our feet wet in with it, we were convinced that having sqlite3 run browser-side, especially once such databases can be made persistent, will open up a floodgate of new opportunities for web app features and catapult sqlite usage into niches which have historically been off limits to it.
The best part is that the core library requires zero code changes for this: the C code compiles flawlessly to WASM and the friction between its interface and JavaScript is minimal, in particular once one has a bit of JS/WASM "glue code" packed up for reuse.
In October 2022 sqlite.org/wasm went online and in November 2022 sqlite 3.40 became the first release to include the JS/WASM components. The technical goals have been achieved and more than 100 pages of docs have been written, but the work is far from "done" (insofar as maintained software is ever "done"). We're now at the stage where we primarily address user feedback. It's long been my experience that end users tend to come up with the best ideas for new features and improvements.
In January 2023, Google posted about the arrival of SQLite's OPFS support. We were not the first to do so, by the way: to the very best of my knowledge, Roy Hashimoto's wa-sqlite project was the first such implementation (and is admittedly much more geared towards modern JS developers' expectations of how JS should be written and published, whereas we prefer to produce lower-level components with as little "voodoo" as possible between the C code, the JS code, and the user). Google's excitement about it is because this support is now part of the "official" releases of SQLite, raising the profile of OPFS somewhat and providing web developers with a truly compelling use for it. (To be clear, we had frequent contact with folks from Google's Chrome and Emscripten teams during the development, and the OPFS integration came as no suprise to them.)
As of this writing, August 2023, SQLite in the browser is slowly picking up steam. We'll see what happens.
Java Native Interface
In July 2023 we started implementing a Java Native Interface (JNI) binding to the core library at the behest of a corporate backer. Like the JavaScript/WASM bindings, that subproject fell neatly into my lap and is the 8th or 9th non-C/C++ binding i've done for sqlite3. That extension is now feature-complete, but whether or not it will ever amount to much is a mystery, especially now that (A) JDK24 actively aims to dissuade folks from using JNI and (B) the folks who initially asked for it have shown no interest in it. Shrug.
Rebuilding from the Ground Up
A years-long itch of mine has been to replace sqlite's GNU Autotools-based build system with one based on Steve Bennett's Autosetup. Why? Because, frankly, i severely dislike the former (have since the early 2000s) and have had nothing but positive experiences with the latter. We've used Autosetup in the Fossil SCM since 2011 and it's never given us any heartburn (unlike the Autotools, which we avoided touching unless absolutely necessary, because touching it invariably burned our fingers), and i've used it in a number of person projects to good effect. However, i shied from actively scratching that itch because replacing that project's build tools with something new was (A) going to be a many-month project and (B) be an uphill battle on multiple fronts (both within the dev team and without). Thus that itch sat unscratched for several years.
In September 2024 i got the proverbial bee in my bonnet to take a whack at it, simply to see what it might look like, with no real expectation that such an effort would be acceptable for the project. SQLite's unusually wide footprint means that any disruption has the potential to affect millions, even billions, of users, so we generally try to avoid disruptive changes. After about a month of experimentation, Richard (the project lead) agreed that this was the way forward and OK'd the merging of that feature into the SQLite trunk on 2024-10-25. SQLite version 3.48.0 became the first version to use the new build system in the canonical source tree and 3.49.0 became the first to use it in the so-called "autoconf" tree (a highly-simplified version of the source tree suitable for users other than SQLite's own developers, and the one which must downstream folks use).
That caused some downstream disruption, certainly, including no small amount of outrage (for lack of a better word) from a minority of folks. Even so, the build tree is primarily for the developers' benefit, not those of downstream users, and the new tree gave us several significant advantages over the autotools tree. Downstream users are welcome to provide their own build solutions if they can't come to terms with the ones we provide. We provide the source code and "a" build solution, but do not claim to provide The One True Build Solution. Many downstream users do not seem to recognize that distinction, and take it for granted that we are responsible for providing a build solution which works on their favorite environment. As the project's current "Build Guy," i'm eager to have the build run on as many environments as possible, but frequently have to remind users that their specific build requirements may require hand-written (by them) build solutions. Shrug.
As of this writing (2025-02-24), within the project we collectively consider the port to have been well worth the effort, noting that we're still triaging breakage in downstream build automation which trips over changes in the build process. We've taken back control of the build process, something the GNU Autotools like to hide much of from the developers, or to make so cryptic to maintain that developers simply don't touch it unless they absolutely must. Yes, getting the build back to, more or less, feature-parity with the legacy build has required a good deal of effort and more than a few sleepless nights, and is an ongoing process, but (A) this is what i signed up for when Richard agreed to going in this direction and (B) this is what i do for fun (even when it's not all that fun, like when folks who use an exotic platform which we cannot reasonably get access to report that some voodoo libtool (part of the Autotools) used to perform on their behalf is no longer performed on their behalf). Sigh. Oh, well. That's just part of it.