Loading devreal.ai…

paper · community record

ADAM: Genomics Formats and Processing Patterns for Cloud Scale Computing

Current genomics data formats and processing pipelines are not designed to scale well to large datasets. The current Sequence/Binary Alignment/Map (SAM/BAM) formats were intended for single node processing. There have been attempts to adapt BAM to distributed computing environments, but they see limited scalability past eight nodes. Additionally, due to the lack of an explicit data schema, there are well known incompatibilities between libraries that implement SAM/BAM/Variant Call Format (VCF) data access. To address these problems, we introduce ADAM, a set of formats, APIs, and processing stage implementations for genomic data. ADAM is fully open source under the Apache 2 license, and is implemented on top of Avro and Parquet for data storage.

Read paper ↗Add or suggest a connection ↗

Connections

1 relationship

authored · incomingFrank Austin Nothaftperson ↗

Provenance

Type: paper
Status: Imported / unclaimed
year: 2013
venue: UC Berkeley EECS Technical Report UCB/EECS-2013-207
source: bythebay
pdf_url: https://www2.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-207.pdf
abstract: Current genomics data formats and processing pipelines are not designed to scale well to large datasets. The current Sequence/Binary Alignment/Map (SAM/BAM) formats were intended for single node processing. There have been attempts to adapt BAM to distributed computing environments, but they see limited scalability past eight nodes. Additionally, due to the lack of an explicit data schema, there are well known incompatibilities between libraries that implement SAM/BAM/Variant Call Format (VCF) data access. To address these problems, we introduce ADAM, a set of formats, APIs, and processing stage implementations for genomic data. ADAM is fully open source under the Apache 2 license, and is implemented on top of Avro and Parquet for data storage.
citation_count: 97