Flint
Large Scale Microbiome Profiling in the Cloud
People
Principal Investigator: Prof. Giri Narasimhan
Principal Architect: Camilo Valdes
Other Contributors: Vitalii Stebliankin
Abstract
Motivation: Bacterial metagenomics profiling for metagenomic whole sequencing (mWGS) usually starts by aligning sequencing reads to a collection of reference genomes. Current profiling tools are designed to work against a small representative collection of genomes, and do not scale very well to larger reference genome collections. However, large reference genome collections are capable of providing a more complete and accurate profile of the bacterial population in a metagenomics dataset. In this paper, we discuss a scalable, efficient, and affordable approach to this problem, bringing big data solutions within the reach of laboratories with modest resources.
Results: We developed Flint, a metagenomics profiling pipeline that is built on top of the Apache Spark framework, and is designed for fast real-time profiling of metagenomic samples against a large collection of reference genomes. Flint takes advantage of Spark's built-in parallelism and streaming engine architecture to quickly map reads against a large (170 GB) reference collection of 43,552 bacterial genomes from Ensembl. Flint runs on Amazon's Elastic MapReduce service, and is able to profile 1 million Illumina paired-end reads against over 40K genomes on 64 machines in 67 seconds — an order of magnitude faster than the state of the art, while using a much larger reference collection. Streaming the sequencing reads allows this approach to sustain mapping rates of 55 million reads per hour, at an hourly cluster cost of \$8.00 USD, while avoiding the necessity of storing large quantities of intermediate alignments.
Availability: Flint is open source software, available under the MIT License (MIT).
Download Github Site: Flint
Contact: Prof. Giri Narasimhan
