Building blocks
A big-data system is assembled from a small number of kinds of component. The names change every few years; the kinds have been stable since the mid-2000s.
| Building block | What it does | First widely known form |
|---|---|---|
| Distributed file system | Stores very large files as blocks spread over many machines, each block in several copies | Google File System paper, 2003 |
| Object storage | Stores any blob under a key, with no folders and practically no size limit | Public cloud storage services, from 2006 |
| Batch engine | Runs one computation over an entire dataset, in parallel | MapReduce paper, 2004; Hadoop, 2006 |
| In-memory engine | The same, keeping intermediate results in memory | Spark, 2010 |
| Distributed log | An append-only sequence of events that many consumers read at their own pace | Kafka, 2011 |
| Stream processor | Maintains running results as events arrive | Open-source systems of the early 2010s |
| Key-value and wide-column stores | Look up and update single records at very large scale, relaxing some guarantees of a relational database | Bigtable paper, 2006; Dynamo paper, 2007 |
| Columnar storage | Stores each column together, so a query reads only the columns it needs | C-Store, 2005; Dremel paper, 2010 |
| Data warehouse | Cleaned, structured data organised for analysis | Described by Bill Inmon, 1992 |
| Data lake | Raw data kept as it arrived, given structure only when read | Term introduced by James Dixon, 2010 |
The three Vs
The usual definition comes from a 2001 research note by Doug Laney, which described data growing along three dimensions at once: volume, velocity and variety. Later writers added more Vs — veracity, value — with less agreement. The phrase “big data” itself was already in use in Silicon Valley in the 1990s.