Benchmarking big data systems is nontrivial and error prone. The authors, from Cloudera, catalogue five common pitfalls encountered in real customer benchmarking engagements - comparing apples to oranges, using unrealistic data, ignoring system architecture, poor experimental design and misreporting results - and give guidance for benchmarks that predict production behaviour.