Building NDStore through Hierarchical Storage Management and Microservice Processing

2018 IEEE 14th International Conference on e-Science


Abstract

We describe NDStore, a scalable multi-hierarchical data storage deployment for spatial analysis of neuroscience data on the AWS cloud. The system design is inspired by the requirement to maintain high I/O throughput for workloads that build neural connectivity maps of the brain from peta-scale imaging data using computer vision algorithms. We store all ourdata on the AWS object store S3 to limit our deployment costs S3 serves as our base-tier of storage. Redis, an in-memory key-value engine, is used as our caching tier. The data is dynamically moved between the different storage tiers based on user access.All programming interfaces to this system are RESTful web-services. We include a performance evaluation that shows thatour production system provides good performance for a variety of workloads by combining the assets of multiple cloud services.

Citation

@inproceedingsLillaney_2018 doi: 10.1109/escience.2018.00037 url: https://doi.org/10.1109/escience.2018.00037 year: 2018 month: oct publisher: IEEE author: Lillaney Kunal and Kleissas Dean and Eusman Alexander and Perlman Eric and Roncal William Gray and Vogelstein Joshua T. and Burns Randal title: Building NDStore Through Hierarchical Storage Management and Microservice Processing booktitle: 2018 IEEE 14th International Conference on e-Science (e-Science)

Contact Us


Chief
Ashley Llorens
Ashley.Llorens@jhuapl.edu
240-228-0312

Physical Address
7701 Montpelier Road
Laurel, MD 20723


The Intelligent Systems Center is located at the Montpelier Campus of the Johns Hopkins Applied Physics Laboratory.
Click here for a map, directions and other visitor information.