TY - DATA T1 - Supporting data for "Field of Genes: Using Apache Kafka as a Bioinformatic Data Repository" AU - Brendan, Lawlor AU - Micheal, Mac Aogain AU - Paul, Walsh AU - Richard, Lynch DO - 10.5524/100430 UR - http://gigadb.org/dataset/100430 AB - Bioinformatic research is increasingly dependent on large-scale data sets, accessed either from private or public repositories. An example of a public repository is NCBI's RefSeq. These repositories must decide in what form to make their data available. Unstructured data can be put to almost any use, but are limited in how access to them can be scaled. Highly structured data offer improved performance for specific algorithms but limit the wider usefulness of the data. We present an alternative: lightly structured data stored in Apache Kafka in a way that is amenable to parallel access and streamed processing, including subsequent transformations into more highly structured representations. We contend that this approach could provide a flexible and powerful nexus of bioinformatic data, bridging the gap between low-structure on one hand, and high-performance and scale on the other. To demonstrate this, we present a proof of concept version of NCBI's RefSeq database using this technology. We measure the performance and scalability characteristics of this alternative with respect to flat files.
The proof of concept scales almost linearly as more compute nodes are added, outperforming the standard approach using files.
Apache Kafka merits consideration as a fast and more scalable but general-purpose way to store and retrieve bioinformatic data, for public, centralized reference datasets like RefSeq, and private clinical and experimental data. KW - Software KW - apache kafka KW - scalability KW - parallelization KW - big data PY - 2018 PB - GigaScience Database LA - en ER -