Data storage and compression using CRAM

As our database expands in scale, managing and storing large genomic sequence data has become a challenge, particularly with the large BAM files. As we head towards cheaper sequencing costs we are anticipating a tsunami of data as researchers switch from exome to whole genome sequencing. In preparation, we've taken early steps of further compressing our BAMs using CRAM compression (lossless). Our database system manages its disk space autonomously such that if our allocated disk space reaches a threshold of 80%, it will automatically convert the oldest BAMs to CRAMs and archive them to tape storage, making way for newer datasets. Our testing has shown that the CRAM compression format saves roughly 30% and will provide significant costs savings. Users that wish to have access to the archived BAMs can click a button from our web interface and the system will automatically restore the CRAM from tape and convert them back to BAM.

Comments

Post a Comment

Popular posts from this blog

Measuring Variant Conservation with GERP Score and Siphy

The more annotations, the better! We've recently added 2 new annotations to assist with variant prioritisation as a measure of variant conservation and these are GERP scores and Siphy. GERP stands for Genomic Evolutionary Rate Profiling. Conceptually, GERP is a method for the identification of slowly evolving regions in a multiple sequence alignment, defined as ‘constrained elements’. "Constrained elements are identified by comparing the observed to the expected rates of evolution for each window, and defining all those regions whose collective observed rates of evolution are significantly lower than would be expected under a null model." More simply, it is a score used to calculate the conservation of each nucleotide in multi-species alignment with ranges from -12.3 to 6.17, with 6.17 being the most conserved. Positive scores (observed fewer than expected) indicate that a site is under evolutionary constraint. Negative scores may be weak evidence of accelerated rates of
Read more

Custom Exomiser reports

Users can now run Exomiser using parameters of their choosing. For example, users can set the allele frequency , variant type, inheritance mode, and model organisms. To use Exomiser, from the menu Reports -> Exomiser. Results will be emailed to you when completed.
Read more

Supplementary information available for download

As part of our datasets for download, we've added some supplementary information (generated as TXT files by the pipeline) to go along with the VCFs and BAMs files. The files contain information such as the cutoffs used to qualify variants as a PASS, which include things like read depth cutoffs, median quality score cutoffs and so on. Furthermore, the files also contain general statistics about total variants passed, the proportion of variants that are exonic, splice sites, number of distinct genes and averages of read depth and median quality. Another file called the 'readReport.summary' includes information about how many were paired, mispaired, aligned, and unaligned. The supplemantary files can be found under the 'Datasets' section.
Read more