A community-developed open-source computational ecosystem for big neuro data
Recent technological developments, such as high-throughput imaging and sequencing, enable experimentalists to collect increasingly large, complex, and heterogeneous ‘big’ data. These studies result in terabytes of data per day, yielding petabytes across experiments and laboratories. These experimental capabilities exceed the scale or feature set of existing software. For example, such data cannot be stored, processed, and visualized on a laptop or workstation. Instead, big data must be stored on data centers and processed on high-performance compute clusters.In 2011, we launched Open Connectome Project1, an open-access data repository powered by open-source web-services software applications that store, analyze, and visualize large imaging datasets. However, as technology changed, features were added, and scale increased, our academic development team and resources became overwhelmed. We overhauled our custom stack into a community-built and -maintained software ecosystem deployed in the commercial cloud, integrating multiple open-source projects and extending them for our needs (https://neurodata.io). The ecosystem makes it possible to analyze disparate datasets by reusing components originally designed for other applications.
@articleVogelstein_2018 doi: 10.1038/s41592-018-0181-1 url: https://doi.org/10.1038/s41592-018-0181-1 year: 2018 month: oct publisher: Springer Science and Business Media LLC volume: 15 number: 11 pages: 846--847 author: Vogelstein Joshua T. and Perlman Eric and Falk Benjamin and Baden Alex and Roncal William Gray and Chandrashekhar Vikram and Collman Forrest and Seshamani Sharmishtaa and Patsolic Jesse L. and Lillaney Kunal and Kazhdan Michael and Hider Robert and Pryor Derek and Matelsky Jordan and Gion Timothy and Manavalan Priya and Wester Brock and Chevillet Mark and Trautman Eric T. and Khairy Khaled and Bridgeford Eric and Kleissas Dean M. and Tward Daniel J. and Crow Ailey K. and Hsueh Brian and Wright Matthew A. and Miller Michael I. and Smith Stephen J. and Vogelstein R. Jacob and Deisseroth Karl and Burns Randal title: A community-developed open-source computational ecosystem for big neuro data journal: Nature Methods