Building NDStore through Hierarchical Storage Management and Microservice Processing

2018 IEEE 14th International Conference on e-Science


We describe NDStore, a scalable multi-hierarchical data storage deployment for spatial analysis of neuroscience data on the AWS cloud. The system design is inspired by the requirement to maintain high I/O throughput for workloads that build neural connectivity maps of the brain from peta-scale imaging data using computer vision algorithms. We store all ourdata on the AWS object store S3 to limit our deployment costs S3 serves as our base-tier of storage. Redis, an in-memory key-value engine, is used as our caching tier. The data is dynamically moved between the different storage tiers based on user access.All programming interfaces to this system are RESTful web-services. We include a performance evaluation that shows thatour production system provides good performance for a variety of workloads by combining the assets of multiple cloud services.


@inproceedingsLillaney_2018 doi: 10.1109/escience.2018.00037 url: year: 2018 month: oct publisher: IEEE author: Lillaney Kunal and Kleissas Dean and Eusman Alexander and Perlman Eric and Roncal William Gray and Vogelstein Joshua T. and Burns Randal title: Building NDStore Through Hierarchical Storage Management and Microservice Processing booktitle: 2018 IEEE 14th International Conference on e-Science (e-Science)

Contact Us

Ashley Llorens

Physical Address
7701 Montpelier Road
Laurel, MD 20723

The Intelligent Systems Center is located at the Montpelier Campus of the Johns Hopkins Applied Physics Laboratory.
Click here for a map, directions and other visitor information.