Next Generation Sequencing (NGS)/Big Data

Next Generation Sequencing (NGS) Big Data Introduction Bioinformatics from the outside Big_data Big Data Data Deluge The first problem you face is probably the large size of the NGS FASTQ files - the "data deluge" problem. You no longer only have to deal with microplate readings, or digitalized gel photos; the size of NGS data can be huge. For example, compressed FASTQ files from a 60x human whole genome sequencing can still require 200Gb. A small project with 10–20 whole genome sequencing (WGS) samples can generate ~4TB of raw data. Even these estimates do not include the disk space required for downstream analysis. Storing data Referenced from a post from BioStarshttps://www.biostars.org/p/6749/ Question: Huge Ngs Data Storage And Transferring Wo, H. Biostars Biostar Genomics, LLC 24 March 2011 28 April 2016: Very high end: enterprise cluster and SAN. High end: Two mirrored servers in separate buildings or Cloud. Typical: External hard drives and/or NAS with raid-5/6 Moving data Moving data between collaborators is also non-trivial. For RNA-Seq samples, FTP may suffice, but for WGS data, shipping hard drives may be the only solution. Externalizing compute requirements from the research group It is difficult for a single lab to maintain sufficient computing facilities. A single lab will probably own some basic computing hardware; however, many tasks will have huge computational demands (e.g. memory for de novo genome assembly) that require them to be performed elsewhere. An institution / core facility may host a centralized cluster. Alternatively, one might consider doing the task on the cloud. NIH maintains a centralized computing cluster called Biowulf. Bioinformatics cloud computing is suggested.https://www.youtube.com/watch?v=ZzBCvmV-6p4 Genomic Analysis in the Cloud Akhlaghpour, H. YouTube Google 3 July 2012 28 April 2016Computational solutions to large-scale data management and analysis Nature Reviews Genetics Schadt, E.E.; Linderman, M.D.; Sorenson, J.; Lee, L.; Nolan, G.P. 11 9 647-57 2010 10.1038/nrg2857 20717155 PMC3124937 EBI has adopted a cloud-based platform called Helix Nebula.http://www.helix-nebula.eu/sites/default/files/4.%20EMBL%20Flagship.pdf PDF Big data and HPC on-demand: Large-scale genome analysis on Helix Nebula – the Science Cloud Lueck, R. Trust-IT Services 16 January 2013 28 April 2016 References