Glossary
The vocabulary of big data, in plain words.
- Cluster
- A group of machines working as one system.
- Node
- One machine in a cluster.
- Scale out / scale up
- Add more machines / buy a bigger machine.
- An architecture in which nodes share neither memory nor disks and communicate only over the network.
- Partition (shard)
- One slice of a dataset, held by one node.
- Replication
- Keeping copies of the same data on several nodes.
- Fault tolerance
- Continuing to work correctly when parts fail.
- Data locality
- Running a computation on the node that already holds its data.
- MapReduce
- A programming model: a map step applied to every record, a reduce step that combines the results.
- Batch processing
- Computing over a complete, stored dataset.
- Stream processing
- Computing continuously over events as they arrive.
- Latency and throughput
- How long one answer takes / how much work is done per unit of time.
- CAP theorem
- Under a network partition a distributed store cannot be both consistent and available.
- Eventual consistency
- Replicas may briefly disagree but converge if updates stop.
- ACID
- Atomicity, consistency, isolation, durability: the guarantees of a traditional database transaction.
- Schema on read
- Storing raw data and imposing structure only when it is queried.
- Columnar format
- A file layout that stores values column by column.
- ETL
- Extract, transform, load: moving data from where it is produced to where it is analysed.
- Sketch
- A small data structure that answers a question about a large dataset approximately.
- Petabyte
- A million gigabytes (10¹⁵ bytes). An exabyte is a thousand petabytes.
- Sampling bias
- Error that comes from how the data were collected, and that does not shrink as the data grow.
- Differential privacy
- A guarantee that the output of an analysis changes very little whether or not any one person’s data is included.