Skip to content

Chapter 6: NoSQL & Specialised Databases

Relational databases are not the only option. This chapter explains why NoSQL appeared and what BASE means, then tours each family with its leading products: key-value stores (Redis, Valkey, DynamoDB, etcd, RocksDB), document databases (MongoDB and alternatives), wide-column stores (Cassandra, ScyllaDB, Bigtable), graph databases and their languages, time-series, search and vector databases. It also covers in-memory, multi-model, ledger and embedded or mobile databases, and finishes with how to choose the right model and combine several in one system.

6.1 Why NoSQL Appeared — and What BASE Means

In the 2000s, giant websites such as Google, Amazon and Facebook had more data and traffic than any single relational server could handle. They built new databases that spread data over thousands of cheap machines, dropped joins and fixed schemas, and sometimes relaxed instant consistency to stay fast and always available. Around 2009 these systems got the catch-all name 'NoSQL', later read as 'Not only SQL'. BASE describes their typical promise: basically available, soft state, eventually consistent.

ACID vs BASE in practice

ACID (classic relational) BASE (classic NoSQL)
First priority Correctness of every transaction Availability and low latency
After a write Everyone sees it immediately Replicas converge soon
Scaling style Traditionally scale up Scale out from day one
Schema Defined up front (on write) Flexible (on read)
Joins Rich joins in the database Data pre-shaped, few joins
Typical use Payments, inventory Feeds, carts, sensor data

These are ends of a spectrum; most modern databases let you mix both.

Think of it like…

A relational database is like a bank branch where every teller checks the central ledger before answering; a BASE system is like a chain of busy cafés that each take orders immediately and reconcile the shared stock list a few seconds later.

Real-world example

The fictional social app ChirpNet stores 2 billion posts. Its relational setup needed constant manual sharding, so it moved timelines to a wide-column store across 120 nodes. A new like might take a second to show for friends on another continent — acceptable for likes, so the team keeps payments and account balances in PostgreSQL, where every read must be exact.

How it works

  1. Data is grouped into 'aggregates' — a whole document, row of columns or value — that the app reads and writes as one unit.
  2. Aggregates are spread across many nodes by key (partitioning) and copied to several nodes (replication).
  3. Writes are accepted by one or a few replicas quickly and propagated to the others in the background.
  4. Readers may briefly see older data (soft state) until replicas converge (eventual consistency); many systems let you tune this per request.
  5. Applications avoid joins by storing data in the shape each query needs, accepting some duplication.

Who uses it

Backend and platform engineers, architects at web-scale companies, and anyone choosing a database for high-volume, flexible or globally distributed data meet NoSQL trade-offs.

Key takeaway

Explain NoSQL as a family of data models (key-value, document, wide-column, graph) chosen for scale, flexibility or access patterns — not as 'no SQL at all'; many NoSQL databases now offer SQL-like languages and ACID transactions.

Going deeper

  • The modern 'NoSQL' label dates from a June 2009 San Francisco meetup about open-source distributed non-relational databases; the hashtag stuck and was later reinterpreted as 'Not only SQL'.
  • Two papers shaped the field: Google's Bigtable (OSDI 2006) introduced the sparse wide-column model, and Amazon's Dynamo (SOSP 2007) showed consistent hashing, quorum replication and eventual consistency in production.
  • BASE was popularised by Dan Pritchett's 2008 ACM Queue article 'BASE: An ACID Alternative', which described trading immediate consistency for availability and partition tolerance at eBay scale.
  • The lines have blurred: MongoDB added multi-document ACID transactions in 4.0 (2018) and DynamoDB added transactions the same year, while PostgreSQL, MySQL and SQL Server gained strong JSON support.
  • NoSQL designs move work into the application: you model data around known queries, duplicate data instead of joining, and handle conflicts or stale reads deliberately.
  • Many NoSQL systems expose tunable consistency — for example Cassandra's ONE, QUORUM and ALL levels or DynamoDB's strongly consistent reads — so each request picks its own trade-off.

Common mistakes & misconceptions

  • Believing NoSQL means 'no schema': the schema still exists in your code and queries; changing it later still needs migrations or careful versioning.
  • Choosing NoSQL for scale you do not have: one PostgreSQL server handles many thousands of transactions per second, far beyond most new apps.
  • Assuming every NoSQL database is eventually consistent: many (MongoDB primaries, HBase, etcd, FoundationDB) are strongly consistent by default.

Try it: Feel eventual consistency vs strong consistency (DynamoDB Local or AWS CLI)

# eventually consistent read (default, half the cost)
aws dynamodb get-item --table-name Posts --key '{"id":{"S":"p1"}}'

# strongly consistent read of the same item
aws dynamodb get-item --table-name Posts --key '{"id":{"S":"p1"}}' --consistent-read

Key databases, concepts, languages, tools & standards

Name What it does Example
Eventual consistency
Distributed systems · Concept · Current · Intermediate · General
Guarantee that if writes stop, all replicas eventually return the same value, though reads may briefly be stale. A user changes their profile photo; friends in Asia see the old photo for about a second.
Soft state
BASE model · Concept · Current · Intermediate · General
Idea that the system's state may change over time without new input, as replicas exchange updates and converge. A follower count drifts from 1,203 to 1,207 as background replication catches up, with no new follows.
Aggregate-oriented model
Martin Fowler and Pramod Sadalage, 'NoSQL Distilled' · Concept · Current · Intermediate · General
Storing related data as one unit (document, row, value) that is read, written and distributed together. An order document holds its line items and address, so loading an order is one read with no joins.
Schema-on-read
NoSQL and data lakes · Concept · Current · Intermediate · General
Data is stored without enforcing a fixed structure; the reading code interprets it, so structure can evolve gradually. Old user documents lack a 'timezone' field, so the app treats a missing value as UTC.
Horizontal scaling by default
NoSQL design principle · Concept · Current · Beginner · General
Designing the database from day one to spread data and load over many commodity servers. ChirpNet grows from 60 to 120 nodes during a viral event without changing application code.
Tunable consistency
Cassandra, DynamoDB, Cosmos DB and others · Feature · Current · Intermediate · General
Choosing consistency per request — faster and possibly stale, or slower and guaranteed current. Likes are read at consistency level ONE, while account settings use QUORUM.
Bigtable paper
Google, OSDI 2006 · Resource · Current · Intermediate · Wide-column
Paper describing Google's sparse, distributed, sorted map — the blueprint for HBase and the wide-column model. A database course assigns the Bigtable paper to explain row keys, column families and tablets.
Dynamo paper
Amazon, SOSP 2007 · Resource · Current · Intermediate · Key-value
Paper on Amazon's highly available key-value store: consistent hashing, quorums, vector clocks and hinted handoff. Cassandra's ring partitioning and Riak's design both trace directly to ideas in the Dynamo paper.
NoSQL Distilled
Pramod Sadalage and Martin Fowler, 2012 · Resource · Current · Beginner · General
Short, readable book introducing NoSQL models, aggregates, distribution and polyglot persistence. A bootcamp student reads it in a weekend before designing a document schema for a project.
NoSQL with ACID
MongoDB, DynamoDB, FoundationDB and others · Concept · Current · Intermediate · General
Trend of non-relational databases adding multi-item ACID transactions, narrowing the gap with relational systems. A game moves gold between two player documents in one MongoDB transaction so coins are never duplicated.
Denormalisation cost
NoSQL data modelling · Concept · Current · Intermediate · General
Duplicating data for fast reads means every copy must be updated when the original changes. When a user renames themselves, ChirpNet must update the name stored inside 30,000 old comments.

6.2 Redis, Valkey, Memcached and the Licence Split

Redis is an in-memory key-value store that answers in well under a millisecond, used for caching, sessions, leaderboards, queues and rate limits. Unlike a simple cache it understands data structures such as lists, hashes, sets and sorted sets. In 2024 Redis changed to non-open-source licences, and the community forked it as Valkey under the Linux Foundation; in 2025 Redis added the open-source AGPL option back. Memcached is the older, simpler pure cache.

Licence changes split one community into two compatible projects.
Redis and Valkey: a licence story. Licence changes split one community into two compatible projects.

Think of it like…

Redis is like a chef's countertop: small compared with the pantry (your main database), but everything on it is within arm's reach, already sorted into bowls, trays and stacks.

Real-world example

The fictional game studio PixelForge keeps its global leaderboard in a sorted set: ZADD after each match and ZREVRANGE to show the top 100 — 50,000 updates a second on one node. Player sessions live in hashes with a 30-minute expiry. When its cloud provider offered Valkey at a lower price, PixelForge switched without changing a line of code.

How it works

  1. Clients send commands over the simple text-based RESP protocol to port 6379.
  2. A main thread executes commands one at a time on data held in RAM, so each command is atomic; extra I/O threads handle networking.
  3. Keys can expire automatically (TTL), and an eviction policy such as allkeys-lru removes keys when memory fills.
  4. Optional persistence saves RDB snapshots and/or an append-only file (AOF) so data survives restarts.
  5. Replicas, Sentinel (automatic failover) or Cluster mode (16,384 hash slots across shards) add availability and scale.

Who uses it

Backend developers, SREs and cloud engineers use Redis or Valkey in almost every web stack; it appears in system-design interviews for caching, rate limiting and leaderboards.

Key takeaway

Know the licence timeline: BSD until 7.2; RSALv2/SSPL from 7.4 (March 2024); Valkey fork (March 2024, BSD); Redis 8 adds AGPLv3 as an option (May 2025).

Going deeper

  • A single Redis or Valkey node commonly serves hundreds of thousands of simple operations per second with sub-millisecond latency; the limit is usually network and memory, not CPU.
  • Cluster mode splits keys into 16,384 hash slots by CRC16 of the key; hash tags such as {user42} force related keys into the same slot so multi-key commands work.
  • Persistence options: RDB point-in-time snapshots (compact, may lose minutes of writes) and AOF with appendfsync everysec (lose at most about one second); many caches disable both.
  • Redis 8 folded the former Redis Stack modules — JSON, the Redis Query Engine with vector search, time series and probabilistic structures — into the core, and added a vector set data type.
  • Valkey is backed by AWS, Google Cloud, Oracle, Ericsson, Snap and others; AWS ElastiCache and MemoryDB, and Google Memorystore, offer Valkey at lower prices than their Redis-branded options.
  • Memcached (2003, written for LiveJournal) is multithreaded, stores only strings with a default maximum item size of 1 MB, and has no persistence or replication — simple and very fast for pure caching.

Common mistakes & misconceptions

  • Using Redis as the only copy of important data without persistence and replication: a restart can wipe everything.
  • Running KEYS * in production: it blocks the single command thread while scanning every key; use SCAN instead.
  • Forgetting maxmemory and an eviction policy: the server can run out of RAM or start rejecting writes.

Try it: Run Valkey (or Redis) locally and build a leaderboard

docker run -d --name kv -p 6379:6379 valkey/valkey:latest
docker exec -it kv valkey-cli

ZADD leaderboard 3200 ana 2950 ben 4100 chen
ZREVRANGE leaderboard 0 2 WITHSCORES
SET session:ana '{"cart":3}' EX 1800
TTL session:ana

Key databases, concepts, languages, tools & standards

Name What it does Example
Redis
Redis Ltd. (RSALv2 / SSPLv1 / AGPLv3) · Database · Current · Beginner · Key-value
In-memory data-structure store used as cache, session store, message broker and lightweight database, with optional persistence. PixelForge keeps its live leaderboard and player sessions in Redis, answering in under a millisecond.
Valkey
Linux Foundation (BSD 3-Clause) · Database · Current · Beginner · Key-value
Open-source fork of Redis 7.2.4 created in 2024 after Redis's licence change; command- and protocol-compatible. PixelForge swaps its endpoint to ElastiCache for Valkey and its existing Redis client libraries keep working.
Redis data structures
Redis / Valkey · Data structure · Current · Beginner · Key-value
Strings, lists, hashes, sets, sorted sets, streams, bitmaps and HyperLogLogs, each with atomic commands. INCR page:views counts visits; LPUSH and BRPOP build a simple job queue.
Sorted set
Redis / Valkey · Data structure · Current · Intermediate · Key-value
Set of unique members each with a numeric score, kept ordered for fast ranking and range queries. ZINCRBY leaderboard 25 ana adds points; ZREVRANK leaderboard ana shows her current position.
TTL and eviction policies
Redis / Valkey · Feature · Current · Beginner · Key-value
Keys can expire after a set time, and policies like allkeys-lru decide what to drop when memory is full. SET otp:ana 482913 EX 300 makes a one-time code vanish automatically after five minutes.
RDB and AOF persistence
Redis / Valkey · Feature · Current · Intermediate · Key-value
Snapshot files and an append-only log of writes that let in-memory data survive restarts. With appendfsync everysec, a crash loses at most about one second of leaderboard updates.
Redis Cluster
Redis / Valkey · Feature · Current · Advanced · Key-value
Built-in sharding over 16,384 hash slots with replicas per shard and automatic failover. A six-node cluster — three primaries and three replicas — spreads 200 GB of sessions across shards.
Redis Sentinel
Redis / Valkey · Tool · Current · Intermediate · Key-value
Monitoring processes that detect a failed primary, promote a replica and tell clients the new address. Three Sentinels agree the primary is down and promote a replica within about ten seconds.
RESP protocol
Redis / Valkey · Protocol · Current · Intermediate · Key-value
REdis Serialization Protocol: a simple text-based request/response format; RESP3 adds richer reply types. Typing PING into a raw TCP connection on port 6379 returns +PONG, a RESP simple string.
Redis Query Engine and vector sets
Redis Ltd. (Redis 8) · Feature · Emerging · Intermediate · Vector
Secondary indexing, full-text and vector similarity search over hashes and JSON, plus a native vector set type. A support bot stores answer embeddings in Redis and finds the closest five in a few milliseconds.
Memcached
Memcached project (BSD) · Database · Current · Beginner · Key-value
Simple, multithreaded, in-memory key-value cache without persistence or data structures, created in 2003. A news site caches rendered article HTML in Memcached for 60 seconds to absorb traffic spikes.
Amazon ElastiCache
Amazon Web Services · Service · Current · Intermediate · Key-value
Managed in-memory caching service for Valkey, Redis OSS and Memcached, including a serverless option. PixelForge runs ElastiCache Serverless for Valkey and pays per gigabyte stored and requests served.
Amazon MemoryDB
Amazon Web Services · Service · Current · Advanced · Key-value
Redis/Valkey-compatible in-memory database with a durable multi-AZ transaction log, usable as a primary database. A game stores player inventories in MemoryDB, getting microsecond reads without losing data on node failure.
Dragonfly
DragonflyDB (BSL) · Database · Emerging · Advanced · Key-value
Multithreaded, Redis- and Memcached-compatible in-memory store designed to use all CPU cores on one large machine. A team replaces a 12-shard Redis cluster with one large Dragonfly instance in a benchmark trial.
Garnet
Microsoft Research (MIT) · Database · Emerging · Advanced · Key-value
Open-source cache-store from Microsoft, written in C#, that speaks the RESP protocol for Redis-client compatibility. An Azure-based team tests Garnet as a drop-in RESP server for its .NET services.

6.3 DynamoDB, etcd, FoundationDB and Other Key-Value Stores

A key-value store keeps data as pairs: a unique key and the value stored under it, like a giant dictionary. Some are huge cloud services (Amazon DynamoDB), some are small, strongly consistent stores for configuration (etcd, ZooKeeper), and some are storage engines that other databases are built on (RocksDB, LevelDB). Simplicity makes them fast and easy to scale, but you can only look things up the ways you planned for.

The partition key's hash picks a partition; each partition lives on three storage nodes in different zones.
How DynamoDB serves a request. The partition key's hash picks a partition; each partition lives on three storage nodes in different zones.

Think of it like…

A key-value store is like a coat check: hand over ticket 417 and you get exactly that coat back instantly — but you cannot ask the attendant for 'all the blue coats'.

Real-world example

The fictional food-delivery app DashBite stores orders in DynamoDB with partition key customer_id and sort key order_time, so 'this customer's last ten orders' is one fast query. On a Friday evening traffic jumps from 2,000 to 40,000 requests per second; on-demand capacity absorbs it with single-digit-millisecond reads. Meanwhile, DashBite's Kubernetes clusters keep all their own configuration in etcd.

How it works

  1. Every item is addressed by a key; the store hashes or orders the key to find the partition or node that owns it.
  2. Distributed stores replicate each partition (DynamoDB keeps three copies across Availability Zones; etcd uses Raft across 3 or 5 members).
  3. Reads and writes by key are fast; range queries work only where keys are ordered (DynamoDB sort keys, RocksDB, FoundationDB).
  4. Secondary indexes or extra tables support other access patterns, at the cost of extra writes.
  5. Embedded engines such as RocksDB run inside another program and store data on local disk using LSM-trees.

Who uses it

Serverless and cloud developers (DynamoDB), Kubernetes and platform engineers (etcd), database builders (RocksDB, FoundationDB) and ad-tech or fraud teams (Aerospike) rely on key-value stores.

Key takeaway

For DynamoDB, memorise: partition key (+ optional sort key), 400 KB maximum item size, RCUs/WCUs or on-demand billing, GSIs for extra access patterns, and Streams for change events.

Going deeper

  • DynamoDB (2012) descends from ideas in the Dynamo paper but uses leader-based Multi-Paxos per partition; each partition handles up to 3,000 read units and 1,000 write units per second and about 10 GB.
  • One RCU is one strongly consistent read per second of up to 4 KB (or two eventually consistent reads); one WCU is one write per second of up to 1 KB; on-demand pricing was cut by about half in late 2024.
  • DynamoDB transactions cover up to 100 items; global tables replicate across Regions, and a multi-Region strong consistency option was added in 2025 alongside the classic last-writer-wins mode.
  • etcd stores Kubernetes' entire cluster state; it uses Raft, keeps a default 2 GB storage quota, and should run on 3 or 5 members with fast SSDs because every write is fsynced.
  • RocksDB, Meta's 2012 fork of Google's LevelDB, is an embedded LSM-tree engine underneath MyRocks, TiKV, YugabyteDB, Kafka Streams state stores and many more systems.
  • FoundationDB, bought by Apple in 2015 and open-sourced in 2018, offers an ordered key-value store with strict serializable ACID transactions and is famous for deterministic simulation testing.

Common mistakes & misconceptions

  • Choosing a low-cardinality DynamoDB partition key (such as status or country): traffic piles onto a few partitions and gets throttled.
  • Using Scan for normal queries: it reads the whole table and costs capacity for every item; design keys or indexes for each access pattern.
  • Storing large blobs in etcd or ZooKeeper: they are for small coordination data; large values slow the whole cluster.

Try it: Use DynamoDB Local and etcd on your machine

docker run -d -p 8000:8000 amazon/dynamodb-local
aws dynamodb create-table --endpoint-url http://localhost:8000 --table-name Orders \
  --attribute-definitions AttributeName=pk,AttributeType=S AttributeName=sk,AttributeType=S \
  --key-schema AttributeName=pk,KeyType=HASH AttributeName=sk,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST

# etcd: put and get a key
etcdctl put /config/featureX on && etcdctl get /config/featureX

Key databases, concepts, languages, tools & standards

Name What it does Example
Amazon DynamoDB
Amazon Web Services · Service · Current · Beginner · Key-value
Serverless key-value and document database with single-digit-millisecond latency at virtually any scale and pay-per-request billing. DashBite stores every order in DynamoDB and handles a 20x Friday spike without capacity planning.
RCUs, WCUs and on-demand mode
Amazon DynamoDB · Metric · Current · Intermediate · Key-value
Capacity units measuring reads and writes; provisioned mode buys them in advance, on-demand bills per request. A steady 500 writes/s workload is cheaper provisioned; a spiky new app starts on-demand.
DynamoDB Streams
Amazon DynamoDB · Feature · Current · Intermediate · Key-value
Time-ordered log of item changes kept for 24 hours, used to trigger Lambda functions or replicate data. Each new order in the stream triggers a Lambda that sends the restaurant a notification.
DynamoDB global tables
Amazon DynamoDB · Feature · Current · Advanced · Key-value
Multi-Region, multi-active replication of a table, with eventual or multi-Region strong consistency options. DashBite serves Europe and the US from local Regions, and either can take over if the other fails.
DynamoDB Accelerator (DAX)
Amazon Web Services · Service · Current · Intermediate · Key-value
In-memory write-through cache in front of DynamoDB, cutting read latency to microseconds for repeated reads. Restaurant menus read millions of times an hour are served from DAX instead of the table.
etcd
CNCF (graduated project) · Database · Current · Intermediate · Key-value
Small, strongly consistent key-value store using Raft, best known for holding Kubernetes cluster state. kubectl apply writes a Deployment object; the API server stores it in etcd under /registry/deployments/...
Apache ZooKeeper
Apache Software Foundation · Tool · Legacy · Intermediate · Key-value
Coordination service for configuration, leader election and locks; Kafka removed its dependency in Kafka 4.0 (2025). An older Hadoop cluster still uses ZooKeeper to elect the active HDFS NameNode.
Consul KV
HashiCorp (IBM) · Tool · Current · Intermediate · Key-value
Key-value store built into HashiCorp Consul for service configuration and feature flags, replicated with Raft. consul kv put app/checkout/timeout 3s changes a timeout that services pick up immediately.
FoundationDB
Apple (open source, Apache 2.0) · Database · Current · Advanced · Key-value
Distributed ordered key-value store with strict serializable ACID transactions, used as a foundation for other data layers. Snowflake stores its metadata in FoundationDB, and Apple uses it behind CloudKit.
Aerospike
Aerospike Inc. · Database · Current · Advanced · Key-value
Distributed key-value database keeping indexes in RAM and data on SSDs for predictable sub-millisecond reads. An ad-tech bidder looks up a user profile in Aerospike within 1 ms before bidding.
Riak KV
Basho (defunct), now community-maintained · Database · Legacy · Advanced · Key-value
Dynamo-inspired, highly available key-value store popular in the 2010s; its company closed in 2017. A legacy messaging system still runs Riak, and the team plans a move to Cassandra.
DynamoDB Local
Amazon Web Services · Tool · Current · Beginner · Key-value
Free downloadable version of DynamoDB for developing and testing on your own machine without an AWS bill. docker run -p 8000:8000 amazon/dynamodb-local lets students practise table design offline.

6.4 Document Databases — MongoDB and Its Alternatives

A document database stores each record as a self-contained document, usually JSON-like, with nested fields and lists. Documents in the same collection can have different fields, so the structure can change as the app grows. MongoDB is the best-known document database; others include Couchbase, Google Firestore, Amazon DocumentDB and PostgreSQL's JSONB column type.

Routers consult config servers to send each query only to shards that hold the data.
A MongoDB sharded cluster. Routers consult config servers to send each query only to shards that hold the data.

Think of it like…

A document database is like a filing cabinet of complete folders: each customer's folder holds their details, addresses and past orders together, so you grab one folder instead of visiting five different drawers.

Real-world example

Elena builds the fictional recipe site CookCircle. Each recipe is one MongoDB document with a title, a list of ingredients, steps and embedded ratings. Loading a recipe page is a single read by _id. An aggregation pipeline computes 'top 10 vegan recipes this week', and a sharded cluster spreads 40 million recipes across three shards.

How it works

  1. The app sends JSON-like documents; the database stores them in a binary form (BSON in MongoDB) inside collections.
  2. Every document has a unique _id; secondary indexes can cover any field, including nested fields and arrays.
  3. Queries filter, project and sort documents; aggregation pipelines chain stages such as $match, $group and $lookup.
  4. A replica set keeps copies on several nodes; one primary takes writes and elects a new primary if it fails.
  5. Sharding splits a collection by a shard key across replica sets, with routers (mongos) directing each query.

Who uses it

Full-stack and mobile developers, content and catalogue platforms, gaming and IoT companies use document databases; MongoDB skills are common in job ads for JavaScript and Python developers.

Key takeaway

The core modelling question is embed or reference: embed data read together and bounded in size; reference data that is shared, large or unbounded (and remember the 16 MB document limit).

Going deeper

  • MongoDB (2009) stores BSON documents up to 16 MB, supports multi-document ACID transactions since 4.0 (2018) and across shards since 4.2; MongoDB 8.0 (October 2024) focused on performance.
  • MongoDB moved to the Server Side Public License in October 2018, so cloud providers cannot offer it as a service without open-sourcing their stack; this led to compatible alternatives such as Amazon DocumentDB.
  • MongoDB Atlas, the company's managed cloud on AWS, Azure and Google Cloud, adds Lucene-based Atlas Search and Atlas Vector Search; MongoDB bought embedding-model company Voyage AI in 2025.
  • Replica sets elect a primary using a Raft-like protocol; write concern (for example w: 'majority') and read concern control how durable and fresh each operation is.
  • Microsoft open-sourced DocumentDB in 2025 — a MongoDB-compatible engine built as PostgreSQL extensions — and it joined the Linux Foundation; FerretDB 2.0 uses it to offer a MongoDB-compatible API on Postgres.
  • PostgreSQL's JSONB (since 9.4, 2014) stores parsed binary JSON with GIN indexing and SQL/JSON path queries, often enough when you need a few flexible fields next to relational data.

Common mistakes & misconceptions

  • Embedding unbounded arrays (every comment ever in one post document): documents grow towards 16 MB and updates slow down; reference instead.
  • Choosing a monotonically increasing shard key such as a timestamp: all inserts hit one shard; use a hashed or compound key.
  • Assuming 'MongoDB-compatible' services behave identically: DocumentDB, Cosmos DB and Firestore support different subsets of operators and features.

Try it: Run MongoDB locally and try an aggregation

docker run -d --name mongo -p 27017:27017 mongo:8
docker exec -it mongo mongosh

use cookcircle
db.recipes.insertMany([{title:'Dal', tags:['vegan'], rating:4.7},{title:'Pesto', tags:['veg'], rating:4.2}])
db.recipes.aggregate([{ $match:{tags:'vegan'} }, { $sort:{rating:-1} }, { $limit:10 }])

Key databases, concepts, languages, tools & standards

Name What it does Example
MongoDB
MongoDB Inc. (SSPL / commercial) · Database · Current · Beginner · Document
The most popular document database: flexible JSON-like documents, rich queries, replication, sharding and ACID transactions. CookCircle stores 40 million recipe documents in MongoDB and serves each page with one read.
BSON
MongoDB Inc. · Format · Current · Intermediate · Document
Binary JSON format adding types such as dates, 64-bit integers, decimals and ObjectIds; documents can be up to 16 MB. createdAt is stored as a BSON Date, so range queries on dates sort correctly.
MQL and the aggregation pipeline
MongoDB Inc. · Language · Current · Intermediate · Document
MongoDB Query Language plus pipelines of stages ($match, $group, $lookup, $project) for analytics and transformations. A pipeline groups this week's ratings by recipe and returns the ten best vegan dishes.
Replica set
MongoDB · Feature · Current · Intermediate · Document
Group of MongoDB nodes holding the same data, with one elected primary for writes and secondaries for failover. When the primary's server dies, a secondary is elected within about 10-12 seconds and writes resume.
Sharded cluster
MongoDB · Feature · Current · Advanced · Document
MongoDB deployment splitting collections across replica-set shards by shard key, with mongos routers and config servers. A hashed _id shard key spreads CookCircle's recipes evenly over three shards.
Multi-document transactions
MongoDB 4.0+ (2018) · Feature · Current · Intermediate · Document
ACID transactions spanning several documents and collections, extended to sharded clusters in 4.2. Moving a recipe between two cookbooks updates both documents atomically or not at all.
MongoDB Atlas
MongoDB Inc. · Service · Current · Beginner · Document
MongoDB's managed cloud service on AWS, Azure and Google Cloud, with search, vector search, triggers and backups. Elena launches a free Atlas cluster for development and upgrades to a dedicated tier at launch.
Atlas Search and Vector Search
MongoDB Inc. · Feature · Current · Intermediate · Search
Full-text (Lucene-based) and vector similarity search integrated into MongoDB queries via $search and $vectorSearch. Searching 'quick spicy noodles' combines keyword and vector results to suggest the most relevant recipes.
Couchbase
Couchbase Inc. (private since 2025) · Database · Current · Intermediate · Document
Distributed document database with a built-in memory cache, SQL++ query language and mobile sync; Capella is its cloud service. An airline serves seat maps from Couchbase's memory-first cache with sub-millisecond reads.
Google Cloud Firestore
Google (Firebase) · Service · Current · Beginner · Document
Serverless document database with real-time listeners and offline sync for mobile and web apps. A chat app's messages appear instantly on all devices through Firestore snapshot listeners.
Amazon DocumentDB
Amazon Web Services · Service · Current · Intermediate · Document
Managed document database implementing the MongoDB API on AWS's own storage engine rather than MongoDB code. A team moves a MongoDB 5-era app to DocumentDB after confirming all its operators are supported.
DocumentDB (open source)
Microsoft, Linux Foundation (MIT) · Database · Emerging · Advanced · Document
Open-source, MongoDB-compatible document engine built as PostgreSQL extensions, released in 2025. A company runs a MongoDB-compatible API on its existing Postgres platform using DocumentDB.
FerretDB
FerretDB Inc. (Apache 2.0) · Tool · Emerging · Intermediate · Document
Open-source proxy that speaks the MongoDB wire protocol and stores data in PostgreSQL with the DocumentDB extension. An app built with the MongoDB driver runs on FerretDB backed by managed Postgres, avoiding SSPL.
PostgreSQL JSONB
PostgreSQL Global Development Group · Feature · Current · Intermediate · Document
Binary JSON column type with GIN indexes and SQL/JSON path queries, a common alternative to a separate document database. A products table keeps fixed columns for price and SKU and a JSONB column for category-specific attributes.
Apache CouchDB
Apache Software Foundation · Database · Current · Intermediate · Document
Document database with an HTTP/JSON API and multi-master replication designed for offline-first sync. Field clinics sync patient visit documents to a central CouchDB whenever they get internet access.

6.5 Wide-Column Stores — Cassandra 5, ScyllaDB, HBase and Bigtable

Wide-column databases store data in tables of rows, but each row can have many columns grouped by a partition key and sorted by clustering columns, and the data is spread across many machines. They are built for huge volumes of fast writes — billions of events, messages or sensor readings — across data centres, with no single point of failure. You design tables around the exact queries you will run.

With three replicas, the client succeeds after two acknowledgements; the third catches up.
A Cassandra write at QUORUM. With three replicas, the client succeeds after two acknowledgements; the third catches up.

Think of it like…

A wide-column store is like a postal sorting centre: every letter is routed by postcode (partition key) to one sorting bay, where it is kept in date order (clustering column), so 'all letters for postcode 1012 this week' is one quick grab.

Real-world example

The fictional smart-meter company GridPulse collects a reading every 15 minutes from 5 million homes — about 480 million rows a day. It stores them in Apache Cassandra 5 with partition key (meter_id, day) and clustering column reading_time. A six-node cluster in each of two data centres absorbs the writes, and fetching one home's day of readings is a single-partition query that takes a few milliseconds.

How it works

  1. Nodes form a ring; each owns token ranges, and the partition key's hash decides which nodes store a row.
  2. A client sends a write to any node, which acts as coordinator and forwards it to the replicas (often 3 per data centre).
  3. Each replica appends to a commit log and an in-memory memtable, later flushed to immutable SSTables on disk.
  4. The consistency level (ONE, QUORUM, LOCAL_QUORUM, ALL) decides how many replicas must answer before success.
  5. Background compaction merges SSTables, drops overwritten data and expired tombstones; repair keeps replicas in sync.

Who uses it

Platform and data engineers at messaging, IoT, streaming, finance and e-commerce companies run wide-column stores; Apple, Netflix and Discord are well-known large users of Cassandra or ScyllaDB.

Key takeaway

Remember 'query-first design': in Cassandra you typically create one table per query pattern, duplicating data, because there are no joins and only limited secondary indexing.

Going deeper

  • Cassandra was built at Facebook in 2008, combining Dynamo-style partitioning and replication with a Bigtable-style storage model, and became an Apache top-level project in 2010.
  • Cassandra 5.0 (September 2024) added Storage-Attached Indexes (SAI), a vector type with approximate nearest-neighbour search, trie-based memtables and SSTables, the Unified Compaction Strategy and dynamic data masking.
  • ScyllaDB rewrote Cassandra in C++ on the Seastar framework with a shard-per-core design; in late 2024 it stopped releasing an open-source edition and moved to a single source-available release.
  • Deletes write tombstones, markers that hide data until compaction removes them after gc_grace_seconds (10 days by default); millions of tombstones in one partition make reads slow or fail.
  • Google Bigtable (paper 2006, cloud service 2015) supports the HBase API and has added GoogleSQL queries; Apache HBase runs a similar model on Hadoop's HDFS with ZooKeeper coordination.
  • IBM acquired DataStax, the main commercial Cassandra company behind Astra DB, in 2025; AWS offers Amazon Keyspaces, a serverless Cassandra-compatible service.

Common mistakes & misconceptions

  • Creating huge partitions (for example all readings for a meter forever): keep partitions to tens of megabytes by adding a time bucket such as day to the key.
  • Using Cassandra like a relational database with ALLOW FILTERING everywhere: queries then scan the cluster; model tables for each query instead.
  • Skipping regular repair: replicas drift apart and deleted data can reappear ('zombie' data) after tombstones expire.

Try it: Run Cassandra 5 and create a time-series table

docker run -d --name cass -p 9042:9042 cassandra:5
docker exec -it cass cqlsh

CREATE KEYSPACE grid WITH replication = {'class':'SimpleStrategy','replication_factor':1};
CREATE TABLE grid.readings (meter_id text, day date, ts timestamp, kwh double,
  PRIMARY KEY ((meter_id, day), ts)) WITH CLUSTERING ORDER BY (ts DESC);

Key databases, concepts, languages, tools & standards

Name What it does Example
Apache Cassandra
Apache Software Foundation · Database · Current · Intermediate · Wide-column
Distributed wide-column database with masterless replication across data centres, tuned for very high write throughput. GridPulse writes 480 million meter readings a day into two six-node Cassandra data centres.
Cassandra 5.0
Apache Software Foundation, Sept 2024 · Database · Current · Intermediate · Wide-column
Major release adding Storage-Attached Indexes, vector search, trie-based storage and the Unified Compaction Strategy. After upgrading to 5.0, GridPulse adds an SAI index on region instead of maintaining a second table.
CQL (Cassandra Query Language)
Apache Cassandra · Language · Current · Beginner · Wide-column
SQL-like language for Cassandra with CREATE TABLE, INSERT and SELECT, but no joins and key-based filtering. SELECT kwh FROM readings WHERE meter_id='M17' AND day='2026-10-08'; reads one partition.
Partition key and clustering columns
Cassandra / ScyllaDB · Concept · Current · Intermediate · Wide-column
The partition key decides which nodes store a row; clustering columns sort rows inside that partition. PRIMARY KEY ((meter_id, day), ts) keeps one meter's day together, sorted by time.
Consistency levels
Cassandra / ScyllaDB · Feature · Current · Intermediate · Wide-column
Per-query setting for how many replicas must respond, such as ONE, QUORUM, LOCAL_QUORUM or ALL. Writes and reads at LOCAL_QUORUM with three replicas give consistent results within one data centre.
Storage-Attached Indexes (SAI)
Apache Cassandra 5 · Feature · Emerging · Advanced · Wide-column
Secondary indexes stored alongside each SSTable, allowing efficient filtering on non-key and vector columns. CREATE INDEX ON readings (region) USING 'sai'; lets analysts filter by region without a new table.
Compaction strategies
Cassandra / ScyllaDB · Algorithm · Current · Advanced · Wide-column
Ways of merging SSTables — size-tiered, leveled, time-window and unified — trading write, read and space costs. Time-window compaction groups meter readings by day so whole old files can be dropped when TTLs expire.
Gossip protocol
Cassandra / ScyllaDB · Protocol · Current · Advanced · Wide-column
Peer-to-peer protocol by which nodes exchange membership and health information every second. When a node fails, gossip spreads the news so coordinators stop sending it requests.
ScyllaDB
ScyllaDB Inc. (source-available since 2024) · Database · Current · Intermediate · Wide-column
Cassandra- and DynamoDB-API-compatible database written in C++ with a shard-per-core architecture for low tail latency. A chat company replaces 72 Cassandra nodes with a much smaller ScyllaDB cluster and lowers p99 latency.
Apache HBase
Apache Software Foundation · Database · Legacy · Advanced · Wide-column
Bigtable-style wide-column database on Hadoop HDFS, still used in established Hadoop estates. A bank's older Hadoop platform stores ten years of transaction history in HBase.
Google Cloud Bigtable
Google Cloud · Service · Current · Intermediate · Wide-column
Google's managed wide-column database behind many Google services, with HBase API compatibility and SQL queries. An ad-tech firm stores 2 PB of user events in Bigtable with single-digit-millisecond lookups.
Amazon Keyspaces
Amazon Web Services · Service · Current · Intermediate · Wide-column
Serverless, Cassandra-compatible managed service where you pay per request and storage. A team migrates a small Cassandra cluster to Keyspaces and stops managing nodes and repairs.
DataStax Astra DB
DataStax (IBM) · Service · Current · Intermediate · Wide-column
Serverless database service based on Cassandra, marketed for generative-AI apps with vector search. A start-up stores document chunks and embeddings in Astra DB for a retrieval chatbot.
nodetool repair
Apache Cassandra · Command · Current · Advanced · Wide-column
Anti-entropy operation that compares replicas with Merkle trees and fixes differences. GridPulse runs incremental repair on each node weekly, well inside the 10-day gc_grace window.

6.6 Graph Databases and Graph Query Languages

A graph database stores things (nodes) and the relationships between them (edges) as first-class data. Questions like 'friends of friends', 'shortest route' or 'which accounts share a device with a known fraudster' become quick walks along edges instead of many expensive joins. There are two main styles: property graphs (Neo4j and most products) and RDF triple stores used for linked data and knowledge graphs.

Alice and Bob never transact directly, but sharing a device links Bob to a flagged card.
A small fraud graph. Alice and Bob never transact directly, but sharing a device links Bob to a flagged card.

Think of it like…

A graph database is like a detective's pinboard with photos joined by red string: to find connections you follow the strings directly instead of re-reading every file in the cabinet.

Real-world example

Investigators at the fictional bank Westmoor use Neo4j to model customers, cards, phones and devices. A Cypher query finds every account within three hops of a confirmed fraudster that shares a device or phone number — a question that needed a 12-table SQL join and 40 seconds before, and now returns in under 100 ms. Twenty-three linked accounts are frozen the same afternoon.

How it works

  1. Model the domain as nodes with labels (Person, Card, Device) and relationships with types (OWNS, USES), each holding properties.
  2. Load data and create indexes on the properties used to find starting nodes, such as account number.
  3. Write a pattern query: find start nodes by index, then traverse relationships hop by hop.
  4. Use graph algorithms (shortest path, PageRank, community detection) for analytics across the whole graph.
  5. For RDF, data is stored as subject-predicate-object triples identified by URIs and queried with SPARQL.

Who uses it

Fraud and anti-money-laundering analysts, recommendation and network engineers, knowledge-graph and semantic-web specialists, and AI teams building GraphRAG use graph databases.

Key takeaway

Know the standard: ISO/IEC 39075 GQL (April 2024) is the first new ISO database query language since SQL, and it is heavily based on Cypher's pattern style.

Going deeper

  • Native graph stores like Neo4j use 'index-free adjacency': each node record points directly to its relationships, so a hop costs roughly the same whatever the total graph size.
  • GQL (ISO/IEC 39075:2024) standardises property-graph queries; SQL/PGQ (part 16 of SQL:2023) lets relational databases query tables as graphs with the same pattern syntax.
  • RDF 1.1 (2014) models facts as triples; SPARQL 1.1 (2013) queries them and supports federation across endpoints; OWL ontologies add reasoning; RDF 1.2 work adds statements about statements.
  • Neo4j moved to calendar versioning in 2025 (2025.x releases) and introduced Cypher 25, aligning Cypher with GQL; the Community Edition is GPLv3 and AuraDB is its managed cloud service.
  • Amazon Neptune supports Gremlin, openCypher and SPARQL; Neptune Analytics (2023) is an in-memory engine for graph algorithms and vector search over large graphs.
  • GraphRAG approaches build a knowledge graph from documents with an LLM, then retrieve connected entities and summaries to answer questions better than plain vector search for some tasks.

Common mistakes & misconceptions

  • Using a graph database for simple tabular data: lists of orders or invoices are better in a relational database; graphs shine when relationships are the question.
  • Unbounded variable-length traversals like -[*]-: they can explode across 'supernodes' with millions of edges; always cap hop counts.
  • Confusing property graphs and RDF: they use different models, languages and tools, so pick based on whether you need linked-data standards.

Try it: Try Neo4j and a Cypher pattern query

docker run -d --name neo4j -p 7474:7474 -p 7687:7687 -e NEO4J_AUTH=neo4j/ChangeMe_123 neo4j:latest
# open http://localhost:7474 and run:
CREATE (a:Person {name:'Alice'})-[:USES]->(d:Device {id:'D7'})<-[:USES]-(b:Person {name:'Bob'});
MATCH (p:Person {name:'Alice'})-[:USES]->(:Device)<-[:USES]-(other) RETURN other.name;

Key databases, concepts, languages, tools & standards

Name What it does Example
Property graph model
Neo4j, GQL standard · Data model · Current · Beginner · Graph
Graph of nodes and relationships, both with labels or types and key-value properties. (:Person {name:'Alice'})-[:OWNS {since:2021}]->(:Card {last4:'4421'}) stores a person, a card and their link.
RDF and triples
W3C · Data model · Current · Intermediate · Graph
Resource Description Framework: facts as subject-predicate-object triples identified by URIs, used for linked data. <westmoor:alice> <foaf:knows> <westmoor:bob> states that Alice knows Bob in a shared vocabulary.
Neo4j
Neo4j Inc. · Database · Current · Beginner · Graph
The most widely used native property-graph database, with Cypher, graph data science algorithms and the AuraDB cloud. Westmoor's fraud team finds 23 linked accounts in Neo4j within 100 ms.
Cypher
Neo4j; openCypher project · Language · Current · Beginner · Graph
Declarative graph query language using ASCII-art patterns like (a)-[:KNOWS]->(b) to match graph shapes. MATCH (p:Person)-[:USES]->(d)<-[:USES]-(q) WHERE p.flagged RETURN q finds people sharing devices with flagged people.
openCypher
openCypher project (2015) · Standard · Current · Intermediate · Graph
Open specification of Cypher adopted by Neptune, Memgraph, Apache AGE and others. The same openCypher query runs on Memgraph in development and on Amazon Neptune in production.
GQL support in graph products
ISO/IEC 39075:2024; Neo4j, Google and others · Standard · Emerging · Intermediate · Graph
How vendors adopt the 2024 ISO graph language: Neo4j aligns Cypher 25 with GQL, and Spanner Graph accepts GQL queries. A team writes new graph queries in GQL-conformant syntax so they could later move between Neo4j and Spanner Graph.
Gremlin and Apache TinkerPop
Apache Software Foundation · Language · Current · Intermediate · Graph
Traversal-style graph language and framework where queries are chains of steps over vertices and edges. g.V().has('name','Alice').out('USES').in('USES').values('name') lists people sharing Alice's devices.
SPARQL
W3C (1.1 in 2013) · Language · Current · Intermediate · Graph
Query language for RDF that matches triple patterns and can federate queries across linked-data endpoints. A SPARQL query against Wikidata lists every Nobel laureate born in the Netherlands.
Amazon Neptune
Amazon Web Services · Service · Current · Intermediate · Graph
Managed graph database supporting property graphs (Gremlin, openCypher) and RDF (SPARQL). An insurer runs its claims-fraud graph on Neptune with read replicas across three Availability Zones.
Neptune Analytics
Amazon Web Services, 2023 · Service · Emerging · Advanced · Graph
In-memory analytics engine for running graph algorithms and vector similarity on large graphs in seconds. Analysts load 2 billion edges and run community detection to find fraud rings in minutes.
TigerGraph
TigerGraph Inc. · Database · Current · Advanced · Graph
Distributed, parallel graph database with the GSQL language, aimed at deep-link analytics on very large graphs. A payment network runs 10-hop money-laundering queries across billions of transfers in TigerGraph.
Memgraph
Memgraph Ltd. · Database · Current · Intermediate · Graph
In-memory graph database written in C++, compatible with Cypher and the Bolt protocol, aimed at streaming data. A logistics firm updates its route graph from Kafka in Memgraph and recomputes shortest paths live.
JanusGraph
Linux Foundation · Database · Legacy · Advanced · Graph
Open-source distributed graph database storing data in Cassandra, HBase, Bigtable or ScyllaDB and queried with Gremlin. A telecom keeps its network-inventory graph in JanusGraph on top of an existing Cassandra cluster.
Index-free adjacency
Native graph databases · Concept · Current · Advanced · Graph
Storage design where each node links directly to its neighbours, making each traversal hop cheap. Finding a person's friends costs the same whether the graph holds a thousand or a billion people.
Knowledge graph and GraphRAG
Industry practice; Microsoft Research GraphRAG (2024) · Pattern · Emerging · Intermediate · Graph
Graphs of entities and facts used to ground AI answers, retrieving connected context rather than isolated text chunks. A legal chatbot follows links between cases, judges and statutes before drafting an answer.
Ontotext GraphDB and Stardog
Graphwise; Stardog Union · Database · Current · Advanced · Graph
Enterprise RDF triple stores with SPARQL, reasoning over ontologies and knowledge-graph tooling. A pharmaceutical company links drug, gene and trial data in an RDF knowledge graph for researchers.

6.7 Time-Series Databases — InfluxDB 3, TimescaleDB, Prometheus, QuestDB

Time-series data is a stream of measurements stamped with a time: CPU usage every 10 seconds, a sensor reading every minute, a stock price every millisecond. Time-series databases are built to ingest these points very fast, compress them heavily, and answer questions like 'average per 5 minutes over the last day'. They also delete or summarise old data automatically.

Data gets cheaper to store, and coarser, as it ages.
The life of a time-series point. Data gets cheaper to store, and coarser, as it ages.

Think of it like…

A time-series database is like a ship's logbook: you only ever add new lines at the bottom, you read it by date ranges, and after a while old pages are summarised and archived.

Real-world example

The fictional wind-farm operator Breezeway has 300 turbines, each sending 50 sensor values every second — 15,000 points per second. InfluxDB 3 stores them as Parquet files on object storage, and engineers query 'average blade vibration per minute for turbine 112 this week' in SQL. Raw data is kept 30 days; per-minute averages are kept for ten years.

How it works

  1. Devices or agents send points: a measurement name, tags (such as turbine_id), field values and a timestamp.
  2. The database buffers recent points in memory and a write-ahead log, then writes them to time-partitioned chunks or files.
  3. Older chunks are compressed with delta-of-delta timestamps and XOR or columnar encodings, often 10x or more.
  4. Queries scan only the time ranges needed and aggregate into time buckets (per minute, per hour).
  5. Retention policies drop raw data after a set age, and downsampling jobs keep cheaper summaries.

Who uses it

SREs and DevOps engineers (monitoring), IoT and industrial engineers, energy and finance teams (tick data) and observability platforms all run time-series databases.

Key takeaway

Watch cardinality: the number of unique tag combinations (series), not the number of points, is what usually breaks a time-series database.

Going deeper

  • InfluxDB 3 Core (open source) and Enterprise reached general availability in April 2025; the engine is written in Rust on Apache Arrow, DataFusion and Parquet and supports SQL and InfluxQL but not InfluxDB 2's Flux language.
  • Facebook's Gorilla paper (VLDB 2015) compressed timestamps with delta-of-delta encoding and values with XOR, averaging about 1.37 bytes per point; Prometheus and many others use similar schemes.
  • Prometheus (CNCF graduated) pulls metrics over HTTP, keeps recent data in an in-memory head block with a WAL, writes two-hour blocks to disk and keeps 15 days by default; Prometheus 3.0 shipped in November 2024.
  • TimescaleDB hypertables split a table into time-based chunks and convert old chunks to compressed columnar form; the company behind it renamed itself TigerData in 2025.
  • QuestDB uses a column-oriented, time-partitioned storage model with SQL extensions such as SAMPLE BY and LATEST ON, and accepts InfluxDB line protocol for high-speed ingestion.
  • Long-term Prometheus storage at scale usually means Thanos, Grafana Mimir or VictoriaMetrics, which accept remote write and store blocks in object storage.

Common mistakes & misconceptions

  • Using unbounded values (user IDs, request IDs) as tags or labels: each creates a new series and can exhaust memory — keep them as fields or in logs.
  • Keeping raw high-frequency data forever: set retention and downsampling from day one so storage costs stay predictable.
  • Treating Prometheus as long-term storage: its local TSDB is meant for weeks, not years; use remote write to a long-term store.

Try it: Time-bucket queries in TimescaleDB and PromQL

-- TimescaleDB (PostgreSQL extension)
SELECT time_bucket('5 minutes', ts) AS bucket, avg(vibration)
FROM turbine_readings WHERE turbine_id = 112 AND ts > now() - interval '1 day'
GROUP BY bucket ORDER BY bucket;

# Prometheus PromQL: per-second request rate over 5 minutes
rate(http_requests_total{job="api"}[5m])

Key databases, concepts, languages, tools & standards

Name What it does Example
Time-series data
General · Concept · Current · Beginner · Time-series
Measurements recorded with timestamps, usually appended in time order and queried by time range. Breezeway's turbines report vibration, temperature and rotor speed every second.
InfluxDB 3
InfluxData (Core: MIT/Apache 2.0) · Database · Current · Intermediate · Time-series
Rust-based time-series database on Arrow, DataFusion and Parquet with SQL and InfluxQL; Core is open source, Enterprise is commercial. Breezeway writes 15,000 points per second to InfluxDB 3 and stores history as Parquet in S3.
InfluxDB line protocol
InfluxData · Format · Current · Intermediate · Time-series
Text format for points: measurement, tags, fields and timestamp, accepted by InfluxDB, QuestDB and others. turbine,id=112 vibration=0.83,temp=41.2 1791446400000000000 records one turbine reading.
Flux and InfluxQL
InfluxData · Language · Legacy · Intermediate · Time-series
InfluxDB's earlier query languages; InfluxQL lives on in InfluxDB 3, while Flux is not supported there. A team rewrites its Flux dashboards into SQL when migrating from InfluxDB 2 to InfluxDB 3.
Hypertables and continuous aggregates
TimescaleDB (TigerData) · Feature · Current · Intermediate · Time-series
Time-partitioned PostgreSQL tables plus automatically refreshed materialised rollups for fast dashboards. A continuous aggregate keeps hourly turbine averages up to date, so ten-year charts load instantly.
Prometheus
CNCF (graduated project) · Tool · Current · Beginner · Time-series
Open-source monitoring system and time-series database that scrapes metrics from services and evaluates alerting rules. Prometheus scrapes 400 Kubernetes pods every 15 seconds and alerts when error rates rise.
PromQL
Prometheus project · Language · Current · Intermediate · Time-series
Prometheus's functional query language for selecting series by labels and computing rates, sums and quantiles. histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) shows p99 latency.
Gorilla compression
Facebook, VLDB 2015 · Algorithm · Current · Advanced · Time-series
Delta-of-delta encoding for timestamps and XOR encoding for float values, shrinking points to under two bytes on average. Regular 10-second samples compress to a single bit per timestamp because the deltas never change.
QuestDB
QuestDB Ltd. (Apache 2.0) · Database · Current · Intermediate · Time-series
High-performance columnar time-series database with SQL extensions and fast ingestion, popular in finance and IoT. A trading desk ingests market ticks into QuestDB and runs SAMPLE BY 1s queries for charts.
VictoriaMetrics
VictoriaMetrics Inc. (Apache 2.0) · Database · Current · Intermediate · Time-series
Prometheus-compatible time-series database known for efficient storage and simple single-binary or cluster deployment. An observability team replaces three Prometheus servers with one VictoriaMetrics cluster for a year of metrics.
Grafana Mimir and Thanos
Grafana Labs; CNCF · Tool · Current · Advanced · Time-series
Horizontally scalable long-term storage for Prometheus metrics using object storage. Prometheus remote-writes to Mimir, which keeps 13 months of metrics in Google Cloud Storage.
kdb+ and q
KX Systems · Database · Current · Advanced · Time-series
Columnar in-memory time-series database and its terse q language, a long-time standard for financial tick data. A bank stores every equity trade and quote in kdb+ and runs microsecond-level analytics in q.
Amazon Timestream
Amazon Web Services · Service · Current · Intermediate · Time-series
AWS time-series services; AWS now steers new users towards Timestream for InfluxDB, a managed InfluxDB offering. A start-up provisions Timestream for InfluxDB to avoid running its own InfluxDB servers.
Downsampling and retention
Time-series practice · Pattern · Current · Intermediate · Time-series
Keeping raw data briefly and lower-resolution summaries for longer to control storage cost. Breezeway keeps 1-second data for 30 days, 1-minute averages for 2 years and hourly averages for 10.
Series cardinality
Time-series databases · Metric · Current · Intermediate · Time-series
Number of unique time series, defined by distinct tag or label combinations; the main scaling limit. Adding a user_id label to a metric for 2 million users creates 2 million series and crashes Prometheus.

6.8 Search Engines — Elasticsearch, OpenSearch, Solr, Meilisearch, Typesense

Search engines are databases specialised in finding the most relevant text, not just exact matches. They break text into words, normalise them (lower-case, remove endings), and build an inverted index that lists, for every word, the documents containing it. Results are ranked by relevance. Many are also used for logs and analytics, and most now combine keyword search with AI vector search.

Indexing and querying use the same analyzer so words match consistently.
From text to ranked results. Indexing and querying use the same analyzer so words match consistently.

Think of it like…

A search engine is like the index at the back of a textbook: instead of reading every page to find 'photosynthesis', you look it up once and jump straight to pages 12, 48 and 203 — with the most important page listed first.

Real-world example

The fictional online bookshop PageTurn has 3 million titles. A customer types 'harry poter spells'. OpenSearch's analyzer lower-cases the words, fuzzy matching fixes the typo, BM25 ranks books where 'spells' appears in the title above those mentioning it once in a review, and a vector query adds magic-themed books that never use the word. Results return in 40 ms.

How it works

  1. Documents are sent as JSON and passed through an analyzer: a tokenizer splits text, filters lower-case, remove stop words and stem.
  2. Each resulting term is added to an inverted index pointing to the documents (and positions) where it appears.
  3. The index is split into shards (each a Lucene index) replicated across nodes; new data becomes searchable after a refresh, about every second.
  4. A query is analysed the same way, matching documents are scored (BM25 by default) and the top results returned.
  5. Aggregations count and group results for filters and dashboards; vector fields enable semantic and hybrid search.

Who uses it

Search and relevance engineers, e-commerce and content teams, SREs and security analysts (log search and SIEM) and AI engineers building retrieval systems use search engines.

Key takeaway

Name the engine under most of them: Apache Lucene powers Elasticsearch, OpenSearch and Solr, so concepts like analyzers, segments and BM25 transfer between them.

Going deeper

  • Apache Lucene (Doug Cutting, 1999) stores immutable segments that are merged in the background; BM25 has been Lucene's default similarity since version 6 (2016) and Lucene 10 arrived in 2024.
  • Elastic changed Elasticsearch from Apache 2.0 to SSPL and the Elastic License in 2021, then added AGPLv3 as an option in 2024; Elasticsearch 9.0 (April 2025) moved to Lucene 10.
  • AWS forked Elasticsearch 7.10.2 as OpenSearch in 2021; in September 2024 the project moved to the OpenSearch Software Foundation under the Linux Foundation, and OpenSearch 3.0 shipped in 2025.
  • Near-real-time search: new documents become visible after a refresh (default about one second) creates a new searchable segment; the translog keeps them durable before that.
  • Hybrid search runs a keyword query and a vector query and merges the lists, often with Reciprocal Rank Fusion (score = sum of 1/(k + rank), commonly with k = 60).
  • Smaller engines trade features for simplicity: Meilisearch (Rust) and Typesense (C++) give typo-tolerant instant search with tiny setup, while Vespa targets large-scale ranking and recommendation.

Common mistakes & misconceptions

  • Using a search engine as the system of record: indexes can need rebuilding; keep the source of truth in a database and re-index from it.
  • Too many small shards: each has memory overhead; aim for shards of roughly 10-50 GB rather than thousands of tiny ones.
  • Ignoring analyzers per language: English stemming on German or Hindi text gives poor matches; choose language-specific analyzers.

Try it: Run OpenSearch locally and search with a typo

docker run -d -p 9200:9200 -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true opensearchproject/opensearch:latest
curl -XPUT localhost:9200/books/_doc/1 -H 'Content-Type: application/json' -d '{"title":"Harry Potter and the Book of Spells"}'
curl 'localhost:9200/books/_search?pretty' -H 'Content-Type: application/json' \
  -d '{"query":{"match":{"title":{"query":"harry poter","fuzziness":"AUTO"}}}}'

Key databases, concepts, languages, tools & standards

Name What it does Example
Apache Lucene
Apache Software Foundation · Library · Current · Intermediate · Search
Java search library providing inverted indexes, analyzers, scoring and vector search; the core of most open search engines. Elasticsearch, OpenSearch and Solr all store each shard as a Lucene index on disk.
Elasticsearch
Elastic N.V. · Database · Current · Beginner · Search
Distributed search and analytics engine on Lucene, central to the Elastic Stack for search, logs and security. PageTurn's first search system used Elasticsearch with three nodes and 12 shards.
Elastic licensing changes
Elastic N.V. (2021, 2024) · Concept · Current · Intermediate · Search
Elasticsearch moved from Apache 2.0 to SSPL/Elastic License in 2021 and added AGPLv3 as an open-source option in 2024. A SaaS company checks the licence terms before embedding Elasticsearch in its hosted product.
Elastic Stack (ELK)
Elastic N.V. · Framework · Current · Intermediate · Search
Elasticsearch, Logstash, Kibana and Beats/Elastic Agent working together for log collection, search and dashboards. SREs ship container logs with Elastic Agent and investigate errors in Kibana dashboards.
OpenSearch
OpenSearch Software Foundation (Linux Foundation) · Database · Current · Beginner · Search
Apache-2.0 fork of Elasticsearch 7.10 with dashboards, security, vector search and observability plugins. PageTurn moves to Amazon OpenSearch Service to keep an Apache-licensed engine.
Amazon OpenSearch Service
Amazon Web Services · Service · Current · Intermediate · Search
Managed OpenSearch clusters and a serverless option, used for search, log analytics and vector search on AWS. A security team ingests VPC flow logs into OpenSearch Serverless and queries them during incidents.
Apache Solr
Apache Software Foundation · Database · Current · Intermediate · Search
Mature Lucene-based search server with SolrCloud clustering, long used in enterprise, library and e-commerce search. A university library's catalogue search runs on Solr with facets for author, year and subject.
Meilisearch
Meili (open-source community edition) · Database · Current · Beginner · Search
Lightweight Rust search engine offering typo tolerance, instant search-as-you-type and simple setup. A small online store adds instant product search with Meilisearch in an afternoon.
Typesense
Typesense Inc. (GPLv3) · Database · Current · Beginner · Search
Open-source, in-memory, typo-tolerant search engine written in C++, positioned as a simpler Algolia alternative. A recipe site gets 5 ms search-as-you-type results from a single Typesense node.
Vespa
Vespa.ai (open source) · Database · Current · Advanced · Search
Engine for large-scale search, recommendation and ranking with machine-learned models, originally built at Yahoo. A news app ranks millions of articles per user request with a custom model running inside Vespa.
Algolia
Algolia · Service · Current · Beginner · Search
Hosted search-as-a-service with fast APIs, front-end widgets and AI-powered relevance features. A fashion retailer adds Algolia search to its storefront without running any search servers.
BM25
Robertson and Spärck Jones lineage · Algorithm · Current · Intermediate · Search
Relevance formula scoring documents by term frequency, rarity of terms (IDF) and document length. 'Dragon' in a short title scores higher than 'the' appearing 40 times in a long review.
Analyzer
Lucene-based engines · Feature · Current · Intermediate · Search
Pipeline of character filters, a tokenizer and token filters that turns text into searchable terms. An English analyzer turns 'Running Shoes' into the terms run and shoe.
Hybrid search and RRF
Search industry practice · Pattern · Emerging · Intermediate · Search
Combining keyword (BM25) and vector results, typically merged with Reciprocal Rank Fusion. PageTurn's hybrid query finds exact title matches and semantically similar books in one result list.
PostgreSQL full-text search
PostgreSQL Global Development Group · Feature · Current · Intermediate · Relational
Built-in tsvector/tsquery types with GIN indexes and ranking, often enough for small and medium search needs. WHERE to_tsvector('english', title) @@ plainto_tsquery('trail shoes') finds matching products without a separate engine.
ParadeDB
ParadeDB (AGPLv3) · Tool · Emerging · Advanced · Search
PostgreSQL extension (pg_search) adding Elasticsearch-style BM25 search using the Rust Tantivy library. A SaaS team replaces a separate Elasticsearch cluster with BM25 indexes inside its Postgres database.

6.9 Vector Databases — pgvector, Pinecone, Milvus, Weaviate, Qdrant, Chroma

AI models can turn text, images or audio into embeddings: lists of hundreds or thousands of numbers where similar meanings end up close together. A vector database stores these embeddings and quickly finds the nearest ones to a query, which powers semantic search, recommendations and retrieval-augmented generation (RAG) for chatbots. You can use a dedicated vector database or add vector search to a database you already run.

Dedicated vector DB or vectors in your existing DB?

Dedicated vector database Vector search in existing DB
Examples Pinecone, Milvus, Qdrant, Weaviate pgvector, OpenSearch, Atlas, Oracle
Extra system to run Yes No
Keeping data in sync Pipeline needed Same transaction
Very large scale Billions of vectors, tuned Millions to low billions
Joins & SQL filters Metadata filters only Full SQL or native queries
Best for Vector-heavy AI products Apps adding AI features

Most teams start on the right-hand side and move left only when needed.

Think of it like…

Embeddings place every item on a giant map of meaning; a vector database is the map's 'what's near me?' button, finding the closest cafés to where you are standing even though their names share no words with your search.

Real-world example

The fictional help-desk company SupportSphere has 200,000 support articles. Each is split into chunks and embedded into 1,024-number vectors stored in Qdrant with metadata such as product and language. When a customer asks 'my invoice shows the wrong VAT', the system embeds the question, finds the 8 nearest chunks filtered to the billing product, and an LLM writes an answer citing them — in under a second.

How it works

  1. An embedding model converts each item (text chunk, image) into a vector with a fixed number of dimensions, such as 768 or 1,536.
  2. Vectors are stored with an ID and metadata (source, date, permissions) in the vector store.
  3. An approximate nearest-neighbour (ANN) index such as HNSW or IVF is built so searches avoid comparing against every vector.
  4. A query is embedded with the same model, and the k closest vectors by cosine, dot-product or Euclidean distance are returned, often with metadata filters.
  5. Results feed search pages, recommendations or an LLM prompt (RAG); recall is checked against exact search to tune the index.

Who uses it

AI and ML engineers, search engineers, backend developers building chatbots and recommendation systems, and data platform teams evaluating AI infrastructure use vector databases.

Key takeaway

A strong interview answer: 'Start with vector search in the database you already have (pgvector, your search engine, your document DB); move to a dedicated vector database when scale, latency or filtering needs demand it.'

Going deeper

  • Raw float32 vectors are large: one million 1,536-dimension embeddings take about 6 GB before index overhead, so products offer scalar, binary or product quantization to shrink memory several-fold.
  • Approximate search trades accuracy for speed; recall@10 of 0.95 means 95 % of the true ten nearest neighbours were found, and parameters such as HNSW's ef_search tune that balance.
  • Filtering is hard: post-filtering after ANN can return too few results, so engines like Qdrant, Weaviate and pgvector 0.8+ (iterative scans) integrate filters into the search.
  • Mainstream databases added vectors: PostgreSQL (pgvector), MySQL HeatWave, Oracle 23ai/26ai, SQL Server 2025, MongoDB Atlas, Redis 8, Elasticsearch/OpenSearch and Cassandra 5.
  • Milvus graduated in the LF AI & Data Foundation and is offered as Zilliz Cloud; Qdrant is written in Rust; Weaviate in Go; Chroma (Apache 2.0) targets developer simplicity and offers Chroma Cloud.
  • Object-storage-based vector stores emerged for cheap, large collections: Amazon S3 Vectors (2025) stores vectors directly in S3 buckets, trading some latency for much lower cost.

Common mistakes & misconceptions

  • Mixing embeddings from different models or versions in one index: their spaces are incompatible; re-embed everything when you change models.
  • Ignoring access control: a RAG system must filter by the user's permissions, or it can leak documents they are not allowed to see.
  • Judging only by benchmark speed: real quality depends on chunking, the embedding model and filters; measure recall and answer quality on your own data.

Try it: Nearest-neighbour search with pgvector

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE docs (id bigserial PRIMARY KEY, body text, embedding vector(3));
INSERT INTO docs (body, embedding) VALUES ('invoice VAT', '[0.9,0.1,0.0]'), ('reset password', '[0.0,0.2,0.9]');
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);
SELECT body FROM docs ORDER BY embedding <=> '[0.8,0.2,0.1]' LIMIT 1;

Key databases, concepts, languages, tools & standards

Name What it does Example
Embedding
Machine learning · Concept · Current · Beginner · Vector
A fixed-length list of numbers produced by a model so that similar items have nearby vectors. The questions 'reset my password' and 'forgot login' produce vectors with cosine similarity of about 0.9.
Similarity metrics
Vector search · Concept · Current · Intermediate · Vector
Cosine similarity, dot product and Euclidean (L2) distance measure how close two vectors are. Normalised OpenAI-style embeddings give the same ranking with cosine or dot product.
Metadata filtering
Vector databases · Feature · Current · Intermediate · Vector
Restricting vector search by attributes such as tenant, language, date or permission while keeping results fast. SupportSphere searches only vectors where product = 'billing' and language = 'en'.
Postgres with pgvector
PostgreSQL + pgvector · Pattern · Current · Intermediate · Vector
Using PostgreSQL as the vector store so embeddings live beside relational data with joins, transactions and backups. A start-up keeps users, documents and embeddings in one Postgres database and avoids a sync pipeline.
Pinecone
Pinecone Systems · Service · Current · Intermediate · Vector
Fully managed, serverless vector database service separating storage and compute, with namespaces and hybrid search. A legal-tech company stores 500 million clause embeddings in Pinecone without running any servers.
Milvus and Zilliz Cloud
LF AI & Data Foundation; Zilliz · Database · Current · Intermediate · Vector
Open-source, distributed vector database built for billions of vectors, with a managed version from Zilliz. An image-search company indexes 2 billion photo embeddings in a Milvus cluster with GPU acceleration.
Weaviate
Weaviate B.V. (BSD 3-Clause) · Database · Current · Intermediate · Vector
Open-source vector database in Go with built-in hybrid search, multi-tenancy and model-integration modules. A SaaS gives each customer an isolated Weaviate tenant for their own documents.
Qdrant
Qdrant (Apache 2.0) · Database · Current · Intermediate · Vector
Rust vector database with strong payload filtering, quantization and a managed cloud service. SupportSphere runs Qdrant with scalar quantization, cutting memory use about four-fold.
Chroma
Chroma (Apache 2.0) · Database · Current · Beginner · Vector
Developer-friendly open-source embedding database that runs in a notebook or as a server, with a cloud service. A student builds a RAG demo over lecture notes with Chroma in 20 lines of Python.
LanceDB
LanceDB Inc. (Apache 2.0) · Database · Emerging · Intermediate · Vector
Embedded vector database built on the Lance columnar format, storing vectors and multimodal data on local disk or object storage. A robotics team keeps camera frames and their embeddings together in LanceDB on S3.
Amazon S3 Vectors
Amazon Web Services, 2025 · Service · Emerging · Intermediate · Vector
Vector storage and similarity search built into S3 buckets, aimed at large, cost-sensitive collections. An archive stores 1 billion rarely queried document vectors in S3 Vectors for a fraction of in-memory costs.
Built-in vector search
Oracle, SQL Server, MongoDB, Redis, Elastic, Cassandra and others · Feature · Current · Intermediate · Vector
Vector types and indexes added to mainstream databases so AI features need no separate system. A bank adds a vector index in Oracle 26ai rather than copying customer data to a new vector service.

6.10 In-Memory Databases and Data Grids

An in-memory database keeps its working data in RAM instead of reading it from disk, which makes reads and writes thousands of times faster. Because RAM is lost when power fails, these systems protect data with snapshots, logs and replicas. Some are full relational databases (SAP HANA, VoltDB), some are features inside bigger databases, and some are distributed 'data grids' that spread data across many servers' memory.

Log scale: each step down the storage hierarchy is roughly ten to a hundred times slower.
Why memory is fast: typical access latency. Log scale: each step down the storage hierarchy is roughly ten to a hundred times slower.

Think of it like…

Working from RAM versus disk is like cooking with ingredients already chopped on the counter instead of fetching each one from a basement storeroom — but you still keep a written recipe log in case someone knocks the counter over.

Real-world example

The fictional mobile operator TeleNova must check a caller's balance and authorise each call in under 5 milliseconds for 30 million subscribers. It runs Volt Active Data, which keeps all balances in memory across 12 nodes, logs every transaction to disk and replicates to a second site. Disk-based tests had peaked at 8,000 authorisations a second; the in-memory cluster handles over 150,000.

How it works

  1. All active data and indexes are loaded into RAM, often in structures designed for memory rather than disk pages.
  2. Transactions run against memory with lightweight concurrency control, sometimes on single-threaded partitions to avoid locks.
  3. For durability, changes go to a sequential command or redo log on fast disk, and periodic snapshots capture full state.
  4. Replicas on other nodes or sites take over if a node fails; after a restart, the snapshot plus log rebuild memory.
  5. Data that outgrows memory is spilled, tiered to SSD or offloaded to cheaper storage.

Who uses it

Telecom, trading, gaming, ad-tech and ERP teams (SAP HANA) use in-memory databases; Java architects use data grids such as Hazelcast and Apache Ignite.

Key takeaway

Remember 'in-memory does not mean volatile': good in-memory databases are durable through logs, snapshots and replication — the difference is where the working set lives.

Going deeper

  • SAP HANA (2010) is an in-memory, column-oriented relational database; SAP S/4HANA runs only on HANA, which makes HANA skills important for SAP consultants.
  • VoltDB grew from the H-Store research project (Michael Stonebraker and colleagues); it partitions data and runs each partition's transactions serially on one thread, avoiding locks entirely.
  • SQL Server In-Memory OLTP (project Hekaton, 2014) and the Oracle Database In-Memory option (dual row and column formats, 2014) bring in-memory techniques inside traditional databases.
  • Data grids such as Hazelcast and Apache Ignite partition key-value data across a cluster's RAM with backups on other nodes and can run computations next to the data.
  • Durability usually combines a write-ahead or command log fsynced in batches (group commit) with periodic snapshots; recovery time grows with memory size, so snapshots matter.
  • Intel's Optane persistent memory promised byte-addressable durable memory but was discontinued in 2022; CXL memory expansion is the newer route to larger memory pools.

Common mistakes & misconceptions

  • Underestimating RAM cost and growth: memory costs far more per gigabyte than SSD; plan sizing for indexes, replicas and growth headroom.
  • Turning off logging for speed in a system of record: a crash then loses data since the last snapshot.
  • Ignoring restart time: reloading hundreds of gigabytes into memory can take many minutes, so keep replicas ready to take over.

Try it: See the memory-vs-disk gap yourself (Linux)

# write 1 GB to RAM-backed tmpfs vs disk and compare speed
dd if=/dev/zero of=/dev/shm/test bs=1M count=1024 conv=fdatasync
dd if=/dev/zero of=./test bs=1M count=1024 conv=fdatasync
rm /dev/shm/test ./test

Key databases, concepts, languages, tools & standards

Name What it does Example
In-memory database
General · Concept · Current · Beginner · General
Database whose primary copy of working data lives in RAM, using logs, snapshots and replicas for durability. TeleNova authorises 150,000 calls a second because balances never wait for disk reads.
Snapshot plus log durability
In-memory database design · Pattern · Current · Intermediate · General
Periodic full snapshots combined with a continuous change log so memory can be rebuilt after a crash. After a restart, TeleNova loads the 02:00 snapshot and replays ten minutes of command log.
SAP HANA
SAP · Database · Current · Intermediate · Analytical
In-memory, columnar relational database powering SAP S/4HANA and analytics, on premises or as SAP HANA Cloud. A manufacturer's S/4HANA system runs month-end closing on a 6 TB HANA database in minutes.
Volt Active Data (VoltDB)
Volt Active Data Inc. · Database · Current · Advanced · Relational
In-memory, partitioned relational database running serial transactions per partition for very high throughput and low latency. A telecom charges prepaid calls in real time with VoltDB stored procedures.
Oracle TimesTen
Oracle Corporation · Database · Legacy · Advanced · Relational
In-memory relational database often used as a cache or for low-latency telecom and trading systems. A trading system keeps reference data in TimesTen for microsecond lookups.
Oracle Database In-Memory
Oracle Corporation · Feature · Current · Advanced · Analytical
Paid option keeping a columnar copy of chosen tables in memory alongside normal row storage for fast analytics. ALTER TABLE sales INMEMORY; makes ad-hoc sales reports run many times faster without changing the app.
SQL Server In-Memory OLTP
Microsoft (Hekaton) · Feature · Current · Advanced · Relational
Memory-optimised tables and natively compiled stored procedures with lock-free concurrency inside SQL Server. A betting site moves its hottest session table to a memory-optimised table and removes latch contention.
Hazelcast
Hazelcast Inc. · Database · Current · Intermediate · Key-value
Distributed in-memory data grid and stream-processing platform for Java and other languages. An airline caches seat availability in a Hazelcast cluster shared by 50 booking services.
Apache Ignite
Apache Software Foundation · Database · Current · Advanced · Key-value
Distributed in-memory database and compute grid with SQL, key-value APIs and optional native persistence. A bank runs risk calculations next to cached positions in Ignite instead of pulling data over the network.
Tarantool
Tarantool (VK) · Database · Current · Advanced · Key-value
In-memory database and Lua application server with write-ahead logging and replication. A social network keeps user sessions and counters in Tarantool with Lua stored procedures.
Data grid
Distributed systems · Concept · Current · Intermediate · Key-value
Cluster that pools many servers' RAM into one partitioned, replicated store and can run code near the data. Adding four nodes to the grid adds 1 TB of shared memory for the pricing service.
Intel Optane persistent memory
Intel (discontinued 2022) · Concept · Legacy · Advanced · General
Byte-addressable, durable memory modules once used by SAP HANA and others; production has ended. A 2020 HANA server used Optane to fit a 12 TB database in memory more cheaply than DRAM alone.
CXL memory expansion
CXL Consortium · Standard · Emerging · Advanced · General
Compute Express Link lets servers add or share large pools of memory over PCIe-based links. A database server attaches 2 TB of extra CXL memory to hold a larger in-memory working set.

6.11 Multi-Model Databases — Cosmos DB, ArangoDB, SurrealDB and More

A multi-model database supports several data models — for example documents, graphs, key-value and sometimes relational tables or vectors — in one engine with one query language or API. The aim is to avoid running and synchronising several separate databases. The trade-off is that one engine may not match the best specialist at every model.

Multi-model databases expose several data models over shared storage, indexes and replication.
One engine, many models. Multi-model databases expose several data models over shared storage, indexes and replication.

Think of it like…

A multi-model database is like a multi-tool: knife, screwdriver and scissors in one pocket. Handy and lighter than a toolbox, though a dedicated screwdriver may still be better for a big job.

Real-world example

The fictional travel start-up TripWeave stores user profiles as documents, a graph of cities and flight connections, and session data as key-value — all in ArangoDB. One AQL query finds users' saved destinations, then walks the flight graph for routes with at most one stop. Running one database instead of three cut their on-call load, though they later added a separate search engine for free-text hotel search.

How it works

  1. Data is stored once in a common internal format, often JSON documents in collections.
  2. Different models are views over that storage: edges are documents with _from and _to; key-value is lookup by key.
  3. A single query language or several APIs access the models, sometimes combining them in one query.
  4. Indexes, transactions and replication are shared across all the models.
  5. You decide which features to use and whether some workloads still need a specialist system.

Who uses it

Start-ups wanting fewer moving parts, enterprise architects consolidating systems, and Azure developers using Cosmos DB's multiple APIs encounter multi-model databases.

Key takeaway

For Cosmos DB, memorise the five consistency levels from strongest to weakest: Strong, Bounded Staleness, Session (the default), Consistent Prefix and Eventual.

Going deeper

  • Azure Cosmos DB (2017) offers several APIs — NoSQL, MongoDB, Apache Cassandra, Apache Gremlin and Table — plus a PostgreSQL option based on Citus, with turnkey multi-region replication.
  • Cosmos DB bills throughput in Request Units: a point read of a 1 KB item costs 1 RU, and writes and queries cost more; capacity can be provisioned, autoscaled or serverless.
  • ArangoDB (2011) combines documents, graphs and key-value with the AQL language; from version 3.12 (2024) it moved from Apache 2.0 to a source-available licence.
  • SurrealDB (Rust, Business Source License) combines document, graph, relational-style, time-series and vector features behind its SurrealQL language, and can run embedded, in the browser or as a cluster.
  • Fauna, a serverless multi-model document-relational database, shut down its service in May 2025 — a reminder to weigh vendor viability and keep export paths.
  • Mainstream engines have become multi-model too: PostgreSQL (JSONB, PostGIS, pgvector, Apache AGE) and Oracle's 'converged database' cover many models through extensions and built-in types.

Common mistakes & misconceptions

  • Expecting best-in-class performance for every model: benchmark your heaviest workload; a specialist may still be needed.
  • Ignoring Cosmos DB RU costs: unindexed queries or cross-partition scans can burn thousands of RUs; check the request charge of each query.
  • Overlooking licence changes: ArangoDB and SurrealDB use source-available terms — check them before embedding or reselling.

Try it: Combine documents and graph traversal in ArangoDB (AQL)

docker run -d -p 8529:8529 -e ARANGO_ROOT_PASSWORD=ChangeMe_123 arangodb:latest
// in the web UI at http://localhost:8529, AQL:
FOR city IN cities FILTER city.name == 'Lisbon'
  FOR v, e IN 1..2 OUTBOUND city flights
    RETURN DISTINCT v.name

Key databases, concepts, languages, tools & standards

Name What it does Example
Azure Cosmos DB
Microsoft Azure · Service · Current · Intermediate · Multi-model
Globally distributed, multi-API database service with single-digit-millisecond latency SLAs and five consistency levels. A retail app replicates its Cosmos DB account to four regions so shoppers read from the nearest one.
Cosmos DB APIs
Microsoft Azure · Feature · Current · Intermediate · Multi-model
Wire-compatible interfaces — NoSQL, MongoDB, Cassandra, Gremlin, Table and PostgreSQL — over Microsoft's managed service. A team migrates a MongoDB app to Cosmos DB for MongoDB, keeping its existing driver code.
Request Units (RU/s)
Microsoft Azure Cosmos DB · Metric · Current · Intermediate · Multi-model
Normalised currency for Cosmos DB throughput covering CPU, memory and I/O of each operation. A 1 KB point read costs 1 RU, while a cross-partition query costs 300 RUs — so the team redesigns it.
Cosmos DB consistency levels
Microsoft Azure Cosmos DB · Feature · Current · Advanced · Multi-model
Five options from Strong to Eventual, letting each account or request trade consistency for latency and availability. Session consistency guarantees a user always reads their own writes while staying fast globally.
ArangoDB
ArangoDB Inc. (source-available since 3.12) · Database · Current · Intermediate · Multi-model
Native multi-model database combining JSON documents, graphs and key-value with the AQL query language. TripWeave finds one-stop routes between saved cities with a single AQL graph traversal.
AQL
ArangoDB · Language · Current · Intermediate · Multi-model
ArangoDB Query Language: declarative queries mixing document filters, joins and graph traversals. FOR v IN 1..2 OUTBOUND 'cities/LIS' flights RETURN v.name lists cities within two flights of Lisbon.
SurrealDB
SurrealDB Ltd. (BSL 1.1) · Database · Emerging · Intermediate · Multi-model
Rust multi-model database with document, graph, relational-style and vector features, queried with SurrealQL. A hackathon team uses SurrealDB embedded in the browser and as a server with the same queries.
SurrealQL
SurrealDB · Language · Emerging · Intermediate · Multi-model
SQL-like query language with record links and graph arrows for traversing relationships. SELECT ->bought->product.name FROM user:ana; lists the products Ana has bought.
MarkLogic
Progress Software (acquired 2023) · Database · Current · Advanced · Multi-model
Enterprise multi-model database for documents, RDF triples and search, used in publishing, government and finance. A publisher stores articles as XML and JSON with semantic triples for topic linking in MarkLogic.
OrientDB and ArcadeDB
OrientDB (SAP); ArcadeDB (Apache 2.0) · Database · Legacy · Advanced · Multi-model
OrientDB was an early document-graph database now largely dormant; ArcadeDB is a newer fork-inspired successor by its founder. A team migrates an ageing OrientDB graph to ArcadeDB because OrientDB receives few updates.
Fauna (shut down 2025)
Fauna Inc. · Service · Legacy · Intermediate · Multi-model
Serverless document-relational database whose service ended in May 2025, forcing customers to migrate quickly. A start-up exports its Fauna data and rebuilds on PostgreSQL within the shutdown window.
Converged database
Oracle and others · Concept · Current · Intermediate · Multi-model
Vendor term for one database engine handling relational, JSON, graph, spatial, vector and other workloads. A bank uses Oracle's JSON, graph and vector features rather than adding three new database products.

6.12 Ledger, Immutable and Versioned Databases

Some data must never be secretly changed: financial records, audit trails, medical consents, supply-chain events. Ledger databases make tampering detectable by chaining every change with cryptographic hashes, so altering an old record breaks the chain. Related ideas include immutable databases that only ever add facts, temporal tables that keep every past version, and 'Git for data' systems that version whole databases.

Changing any old row changes its hash and breaks every later link.
How a ledger proves nothing changed. Changing any old row changes its hash and breaks every later link.

Think of it like…

A ledger database is like a bound, numbered notebook where each page carries a fingerprint of the page before it: tear out or rewrite a page and every later fingerprint stops matching.

Real-world example

The fictional pharmacy chain MediTrust must prove to auditors that no prescription record was edited after dispensing. It stores dispensing events in SQL Server ledger tables. Every night the database digest — a single hash summarising all history — is saved to immutable cloud storage. When an auditor checks a 2025 record, the verification procedure recomputes the hashes and confirms nothing has changed.

How it works

  1. Each committed transaction's rows are hashed, and transactions are grouped into blocks.
  2. Each block's hash includes the previous block's hash, forming a chain (often organised as a Merkle tree).
  3. A small digest of the latest block is published or stored somewhere the database administrators cannot alter.
  4. Updates and deletes keep the old versions in history tables instead of destroying them.
  5. Verification recomputes hashes from the data and compares them with the saved digests; any mismatch reveals tampering.

Who uses it

Auditors, compliance and security engineers, fintech, healthcare and supply-chain developers, and teams needing provable history use ledger and immutable databases.

Key takeaway

Distinguish ledger databases from blockchains: a ledger database is run by one trusted organisation and makes tampering detectable; a blockchain spreads trust across many parties with consensus.

Going deeper

  • Amazon QLDB, AWS's managed ledger database launched in 2019, reached end of support on 31 July 2025; AWS pointed customers to Aurora PostgreSQL with audit and history patterns instead.
  • SQL Server 2022 and Azure SQL offer updatable and append-only ledger tables; digests can be stored automatically in immutable Azure Blob Storage or Azure Confidential Ledger.
  • Oracle offers immutable tables (insert-only) and blockchain tables (insert-only with each row chained by hash and optionally signed) in 19c release updates and later versions.
  • SQL:2011 introduced system-versioned temporal tables that keep every row version with validity periods; SQL Server, MariaDB and Db2 support them, and PostgreSQL relies on extensions or triggers.
  • Datomic stores immutable facts (datoms) and lets you query the database 'as of' any past time; its binaries became free under Apache 2.0 in 2023 under Nubank's ownership.
  • Dolt is a MySQL-compatible database with Git-style branch, diff and merge for data; immudb is an open-source database that uses Merkle trees to provide cryptographic proofs of inclusion.

Common mistakes & misconceptions

  • Thinking a ledger stops tampering: it detects tampering; you still need access control and digests stored outside the administrators' reach.
  • Forgetting privacy laws: immutable records clash with the right to erasure, so keep personal data separable (for example encrypted with deletable keys).
  • Using a blockchain when one organisation controls all the data: a ledger table or audit log is simpler, faster and cheaper.

Try it: Create an append-only ledger table (SQL Server 2022+)

CREATE TABLE dbo.Dispensing (
  Id INT NOT NULL, Patient NVARCHAR(50), Drug NVARCHAR(50), At DATETIME2
) WITH (LEDGER = ON (APPEND_ONLY = ON));

INSERT INTO dbo.Dispensing VALUES (1, 'P-881', 'Amoxicillin', SYSUTCDATETIME());
EXECUTE sp_generate_database_ledger_digest;   -- save this JSON somewhere immutable

Key databases, concepts, languages, tools & standards

Name What it does Example
Ledger database
Industry category · Concept · Current · Beginner · General
Database that records an append-only, cryptographically verifiable history of changes so tampering is detectable. MediTrust proves to auditors that no dispensing record changed since 2024.
Hash chain and Merkle tree
Cryptography · Data structure · Current · Intermediate · General
Structures where each hash depends on earlier data, so any change to history alters the top-level hash. Changing one 2025 row changes its block hash and every block hash after it.
Database digest
Ledger databases · Concept · Current · Intermediate · General
Small hash summarising the entire ledger at a moment, stored externally as a trust anchor. MediTrust saves the nightly digest to write-once storage that even its DBAs cannot overwrite.
Amazon QLDB
Amazon Web Services (ended July 2025) · Service · Legacy · Intermediate · General
Former managed ledger database with an immutable journal and PartiQL queries; support ended on 31 July 2025. A logistics firm migrated its QLDB shipment history to Aurora PostgreSQL before the shutdown date.
SQL Server ledger tables
Microsoft (SQL Server 2022, Azure SQL) · Feature · Current · Intermediate · Relational
Updatable or append-only tables whose changes are hashed into a verifiable ledger with external digests. sp_verify_database_ledger confirms the Dispensing table matches the digest saved last night.
Oracle blockchain and immutable tables
Oracle Corporation · Feature · Current · Advanced · Relational
Insert-only tables; blockchain tables also chain row hashes and can require signatures to prove origin. A government agency stores signed land-registry changes in an Oracle blockchain table.
immudb
Codenotary (Apache 2.0) · Database · Current · Intermediate · Key-value
Open-source immutable database with SQL and key-value interfaces that provides cryptographic proofs of data integrity. A software supply-chain tool stores build signatures in immudb so clients can verify none were altered.
Datomic
Nubank (Cognitect) · Database · Current · Advanced · General
Database of immutable facts with time-travel queries, built around the idea that history is never overwritten. A bank asks Datomic what a customer's address was as of a past date during a dispute.
XTDB
JUXT (open source) · Database · Emerging · Advanced · Relational
Bitemporal SQL database tracking both when facts were true and when they were recorded. An insurer corrects a backdated policy change while keeping both timelines queryable.
Dolt
DoltHub (Apache 2.0) · Database · Current · Intermediate · Relational
MySQL-compatible SQL database with Git-style branches, commits, diffs and merges of data. A data team reviews a pull request that shows exactly which rows of a price table changed.
System-versioned temporal tables
ISO SQL:2011 · Feature · Current · Intermediate · Relational
Tables that automatically keep every previous row version with validity timestamps for 'as of' queries. SELECT * FROM Prices FOR SYSTEM_TIME AS OF '2026-01-01' shows prices on New Year's Day.
WORM storage
Storage industry (e.g. S3 Object Lock) · Concept · Current · Intermediate · General
Write-once-read-many storage where objects cannot be changed or deleted until a retention date passes. MediTrust's digests sit in an S3 bucket with Object Lock in compliance mode for seven years.
Blockchain vs ledger database
Architecture choice · Concept · Current · Beginner · General
Blockchains share trust among many parties via consensus; ledger databases serve one trusted operator more simply. Five competing shipping firms choose a shared blockchain; one hospital chooses ledger tables.
Hyperledger Fabric
Linux Foundation (LF Decentralized Trust) · Framework · Current · Advanced · General
Permissioned blockchain framework for consortia, with ledgers whose world state is held in LevelDB or CouchDB. A food-supply consortium records shipments on Hyperledger Fabric so each member can verify provenance.

6.13 Embedded, Mobile and In-Browser Databases

Many apps keep data on the device itself so they work offline and feel instant. Phones use SQLite underneath higher-level libraries such as Android's Room and Apple's Core Data or SwiftData. Web browsers offer IndexedDB, and can now even run SQLite or Postgres compiled to WebAssembly. Sync engines then copy changes between the device and a server database when a connection is available.

Apps read and write locally; sync moves changes to the server when a connection exists.
Offline-first data flow. Apps read and write locally; sync moves changes to the server when a connection exists.

Think of it like…

An embedded database is like a field notebook a park ranger carries: they write observations all day without signal, then copy them to the central office records when they get back to base.

Real-world example

The fictional vet-clinic app PawChart runs on tablets in rural farms with no signal. Visits are written to a local SQLite database through Room on Android and SwiftData on iPads. When a vet drives back into coverage, a sync engine uploads new visits to the clinic's PostgreSQL and downloads updated vaccine schedules, resolving conflicts by last edit per field.

How it works

  1. Pick a local store: SQLite (directly or via Room, Core Data, SwiftData), an object store (ObjectBox, Realm) or, on the web, IndexedDB or SQLite-Wasm on OPFS.
  2. Define models; the library maps objects to tables or documents and handles schema migrations between app versions.
  3. The UI reads from and writes to the local database, so it works offline and responds instantly.
  4. A sync layer records changes and exchanges them with a server when online, handling conflicts.
  5. Server-side, the canonical data lives in a cloud database, with per-user permissions on what syncs.

Who uses it

Android and iOS developers, front-end and full-stack web developers, desktop and embedded/IoT engineers, and teams building offline-first or 'local-first' apps use these databases.

Key takeaway

Remember that localStorage is not a database: it is a tiny (about 5 MB), synchronous string store — use IndexedDB or SQLite-Wasm for real data in the browser.

Going deeper

  • Android ships SQLite in the OS and Google recommends the Room library (Jetpack), which checks SQL at compile time and manages migrations; Apple's Core Data (2005) and SwiftData (2023) also store data in SQLite by default.
  • MongoDB deprecated Atlas Device Sync and the Realm SDKs in September 2024 and ended them on 30 September 2025; the Realm database remains open source but apps needed new sync solutions.
  • IndexedDB is the W3C browser API for transactional, indexed storage of structured objects; quotas are usually a share of free disk space, managed per origin by the browser.
  • The Origin Private File System (OPFS) gives web apps fast private file access, which lets the official SQLite WebAssembly build and DuckDB-Wasm persist real databases inside browsers.
  • PGlite (by ElectricSQL) compiles PostgreSQL to WebAssembly in a few megabytes, so a real Postgres can run in a browser tab or Node.js process for local-first apps and tests.
  • Local-first sync engines such as ElectricSQL, PowerSync and Zero replicate subsets of a server database (often Postgres) to device storage and stream changes both ways.

Common mistakes & misconceptions

  • Shipping an app update that changes the schema without a migration: users' existing local databases fail to open or crash on start.
  • Relying on browser storage as permanent: browsers can evict data under storage pressure; request persistent storage and keep a server copy.
  • Ignoring sync conflicts: two offline edits to the same record must be resolved by clear rules or CRDTs, not silently overwritten.

Try it: Store and read data with IndexedDB in a browser console

const req = indexedDB.open('pawchart', 1);
req.onupgradeneeded = () => req.result.createObjectStore('visits', { keyPath: 'id' });
req.onsuccess = () => {
  const tx = req.result.transaction('visits', 'readwrite');
  tx.objectStore('visits').put({ id: 1, pet: 'Bella', note: 'vaccinated' });
  tx.oncomplete = () => console.log('saved');
};

Key databases, concepts, languages, tools & standards

Name What it does Example
SQLite on mobile
Bundled in Android and iOS · Database · Current · Beginner · Relational
SQLite is built into Android and iOS and sits underneath most mobile persistence libraries. PawChart's visits table on a tablet is an ordinary SQLite file inside the app's private storage.
Room
Google (Android Jetpack) · Library · Current · Beginner · Relational
Android persistence library over SQLite with annotated entities, DAOs, compile-time SQL checks and migrations. @Query("SELECT * FROM visit WHERE petId = :id") returns a Flow that updates the UI automatically.
Core Data and SwiftData
Apple · Framework · Current · Intermediate · Relational
Apple's object-persistence frameworks; SwiftData (2023) is the modern Swift-native layer, both typically backed by SQLite. An @Model class Visit in SwiftData gives PawChart's iPad app local storage with a few lines of code.
Realm
MongoDB Inc. (open source) · Database · Legacy · Intermediate · Document
Object-oriented mobile database whose Atlas Device Sync and SDK support ended in September 2025. A field-sales app replaces Realm sync with a SQLite-based sync engine before the end-of-life date.
IndexedDB
W3C · Standard · Current · Intermediate · Key-value
Browser API for storing large amounts of structured data with indexes and transactions, per website origin. A web email client keeps 10,000 recent messages in IndexedDB for instant offline search.
Web Storage (localStorage)
WHATWG · Standard · Current · Beginner · Key-value
Simple synchronous key-value string storage in browsers, limited to a few megabytes per origin. A site saves the user's dark-mode preference in localStorage, not their documents.
SQLite Wasm with OPFS
SQLite team; browser vendors · Library · Emerging · Intermediate · Relational
Official WebAssembly build of SQLite that persists databases in the browser's Origin Private File System. A note-taking web app runs real SQL queries offline in the browser using SQLite Wasm.
PGlite
ElectricSQL (open source) · Library · Emerging · Intermediate · Relational
PostgreSQL compiled to WebAssembly, running in browsers, Node.js or Bun with optional persistence. Unit tests start PGlite in-process in milliseconds instead of a Postgres container.
PouchDB
Apache-licensed open source · Library · Current · Intermediate · Document
JavaScript database for browsers and Node.js that syncs with CouchDB using its replication protocol. Field inspectors' tablets sync PouchDB data to a central CouchDB at the end of each day.
Couchbase Lite
Couchbase Inc. · Database · Current · Intermediate · Document
Embedded document database for mobile and edge devices with sync to Couchbase servers. Airline cabin crew record orders in Couchbase Lite during flights and sync after landing.
ObjectBox
ObjectBox Ltd. · Database · Current · Intermediate · Vector
Fast embedded object database for mobile and IoT with built-in on-device vector search. A camera app finds similar photos on the phone using ObjectBox's on-device vector index.
LMDB
Symas Corporation (OpenLDAP project) · Library · Current · Advanced · Key-value
Lightning Memory-Mapped Database: compact, crash-proof embedded key-value store using memory-mapped B+trees. An LDAP directory server stores millions of entries in LMDB with very fast reads.
Berkeley DB
Oracle (originally Sleepycat) · Library · Legacy · Advanced · Key-value
Classic embedded key-value library from the 1990s, still inside older systems but rarely chosen today. A legacy mail server keeps its alias tables in Berkeley DB files.
Local-first sync engines
ElectricSQL, PowerSync, Zero and others · Tool · Emerging · Advanced · General
Tools replicating parts of a server database to devices and syncing changes both ways for offline-first apps. PowerSync streams each vet's assigned farms from Postgres into the tablet's SQLite database.
CRDTs
Distributed systems research · Data structure · Emerging · Advanced · General
Conflict-free replicated data types that merge concurrent edits automatically without coordination. Two vets edit a pet's notes offline, and a text CRDT merges both edits instead of losing one.

6.14 Choosing the Right Data Model — Polyglot Persistence

Polyglot persistence means using different databases for different jobs inside one system — for example PostgreSQL for orders, Redis for sessions, a search engine for product search and a graph for recommendations. Each piece fits its job well, but every extra database adds cost, operations work and the challenge of keeping data in sync. Good architects start simple and add specialist stores only when a clear need appears.

One system of record; every other store is derived from it and can be rebuilt.
A typical polyglot e-commerce system. One system of record; every other store is derived from it and can be rebuilt.

Think of it like…

It is like a kitchen: you could cook everything in one big pot, or use a wok, an oven and a kettle for different dishes. More tools give better results — but also more washing up and more things that can break.

Real-world example

The fictional marketplace CraftHub began with only PostgreSQL. At 2 million products, keyword search became slow, so they added OpenSearch fed by change data capture. Then a Valkey cache cut page loads from 300 ms to 40 ms. They considered a graph database for 'people who bought this also bought', but pgvector and a nightly job in Postgres were good enough — one fewer system to run.

How it works

  1. List the access patterns: lookups by key, rich ad-hoc queries, full-text search, relationship traversals, time ranges, similarity.
  2. Note the requirements: consistency, latency, write volume, data size, retention and compliance.
  3. Start with one strong general-purpose database (usually relational) and measure where it struggles.
  4. Add a specialist store only for a proven need, fed from the system of record by change data capture or events.
  5. Budget for the operations: monitoring, backups, upgrades, security and on-call skills for every database you add.

Who uses it

Solution and software architects, tech leads, platform teams and system-design interview candidates make these choices.

Key takeaway

In system-design interviews, name one system of record and explain how each secondary store (cache, search, analytics) is derived from it and what happens if it is stale.

Going deeper

  • Martin Fowler popularised the term polyglot persistence around 2011, by analogy with polyglot programming; the risk he highlighted was operational complexity, not technical capability.
  • Writing to two stores from application code (the dual-write problem) leaves them inconsistent when one write fails; deriving secondary stores from the primary's log via CDC or an outbox avoids this.
  • Model-to-workload rules of thumb: key-value for lookups by ID, document for self-contained aggregates, wide-column for massive write streams, graph for multi-hop relationships, search for relevance, time-series for metrics.
  • YCSB (Yahoo! Cloud Serving Benchmark, 2010) provides standard workloads A-F for comparing key-value and NoSQL stores, while TPC-C and TPC-H remain the classic relational benchmarks.
  • Jepsen, Kyle Kingsbury's testing project, has found consistency bugs in many distributed databases by injecting network partitions and clock faults — read its reports before trusting marketing claims.
  • Vendor viability and licence risk matter: Fauna's 2025 shutdown, QLDB's retirement and licence changes at Redis, Elastic, MongoDB and others all forced customers to react.

Common mistakes & misconceptions

  • Adding a new database for every feature: each one needs backups, monitoring, upgrades, security reviews and on-call knowledge.
  • Letting the application dual-write to several stores: failures leave silent inconsistencies; derive copies from one source via CDC or an outbox.
  • Choosing on hype or benchmarks alone: test your own access patterns and failure modes, and check the vendor's track record.

Try it: Benchmark a key-value workload with YCSB (Redis/Valkey example)

# Yahoo! Cloud Serving Benchmark against a local Redis-compatible server
curl -O -L https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-0.17.0.tar.gz
tar xfz ycsb-0.17.0.tar.gz && cd ycsb-0.17.0
bin/ycsb load redis -s -P workloads/workloada -p redis.host=127.0.0.1
bin/ycsb run  redis -s -P workloads/workloada -p redis.host=127.0.0.1

Key databases, concepts, languages, tools & standards

Name What it does Example
System of record
Architecture practice · Concept · Current · Beginner · General
The authoritative store for a piece of data; every other copy is derived from it and can be rebuilt. When OpenSearch's index is corrupted, CraftHub rebuilds it from PostgreSQL overnight.
Dual-write problem
Distributed systems · Concept · Current · Intermediate · General
Inconsistency that arises when an application writes the same change to two systems and one write fails. An order saved in Postgres never reaches the search index because the second write timed out.
Change data capture (as a feed)
Debezium and database logs · Pattern · Current · Intermediate · General
Reading the primary database's change log to update caches, search indexes and warehouses reliably. Every product update in Postgres flows through CDC to OpenSearch within two seconds.
Workload-to-model matching
Database selection · Concept · Current · Beginner · General
Matching each access pattern to the data model that serves it naturally, such as graph for multi-hop relationships. Fraud rings go to a graph store; invoice line items stay relational.
Just use Postgres
Common industry practice · Pattern · Current · Beginner · Relational
Approach of meeting most needs — JSON, search, vectors, queues, time-series — with PostgreSQL and its extensions first. A five-person start-up handles search with Postgres full-text and vectors with pgvector, delaying new systems.
Operational overhead
Platform engineering · Metric · Current · Intermediate · General
Ongoing human and infrastructure cost of running a database: upgrades, backups, monitoring, security and on-call. Adding a graph database would mean a new on-call rotation and backup process for a two-person team.
Managed vs self-hosted
Cloud architecture · Concept · Current · Beginner · General
Trade-off between paying a provider to operate a database and running it yourself for control or cost. CraftHub uses managed OpenSearch so its small team never patches search nodes.
YCSB
Yahoo! Research (2010), open source · Tool · Current · Intermediate · Key-value
Yahoo! Cloud Serving Benchmark: standard workloads for comparing key-value and NoSQL database performance. Workload A (50 % reads, 50 % updates) compares Valkey and DynamoDB latency for a session store.
Vendor viability
Procurement and architecture · Concept · Current · Intermediate · General
Risk that a database company or service changes terms, is acquired or shuts down, forcing migration. Fauna's 2025 shutdown taught CraftHub to prefer databases with open-source or multi-vendor options.
Licence risk
Legal and architecture · Concept · Current · Intermediate · General
Risk that a database's licence changes in ways that restrict your use, as happened with Redis and Elasticsearch. CraftHub picks Valkey and OpenSearch partly because foundations govern their licences.
Data gravity
Cloud architecture · Concept · Current · Intermediate · General
Tendency for apps and services to move toward where large data already lives, because moving data is slow and costly. With 400 TB already in one cloud region, new analytics jobs are built there rather than elsewhere.
Consistency requirements
System design · Concept · Current · Intermediate · General
Deciding which data must always be exact (money, stock) and which can briefly be stale (likes, recommendations). Stock levels are read from Postgres; the 'trending' list may be five minutes old from the cache.