Building NDStore through Hierarchical Storage Management and Microservice Processing
We describe NDStore, a scalable multi-hierarchical data storage deployment for spatial analysis of neuroscience data on the AWS cloud. The system design is inspired by the requirement to maintain high I/O throughput for workloads that build neural connectivity maps of the brain from peta-scale imaging data using computer vision algorithms. We store all ourdata on the AWS object store S3 to limit our deployment costs S3 serves as our base-tier of storage. Redis, an in-memory key-value engine, is used as our caching tier. The data is dynamically moved between the different storage tiers based on user access.All programming interfaces to this system are RESTful web-services. We include a performance evaluation that shows thatour production system provides good performance for a variety of workloads by combining the assets of multiple cloud services.
@inproceedingsLillaney_2018 doi: 10.1109/escience.2018.00037 url: https://doi.org/10.1109/escience.2018.00037 year: 2018 month: oct publisher: IEEE author: Lillaney Kunal and Kleissas Dean and Eusman Alexander and Perlman Eric and Roncal William Gray and Vogelstein Joshua T. and Burns Randal title: Building NDStore Through Hierarchical Storage Management and Microservice Processing booktitle: 2018 IEEE 14th International Conference on e-Science (e-Science)