Comparison of data file and storage configurations for efficient temporal access of satellite image data

Abstract

Satellite data volumes have seen a steady increase in recent years due to improvements in sensor technology and increases in data acquisition frequency. The gridded MODIS data products, spanning a region of interest of approximately 10° by 10° for a single title, are stored as images containing almost six million pixels, with data in multiple spectral bands for each pixel. Time series analyses of a sequence of such images in order to perform automated change detection is a topic of growing importance. Traditional storage formats store such a series of images as a sequence of individual files, with each file internally storing the pixels in their spatial order. Consequently, the construction of a time series profile of a single pixel requires reading from several hundred large files, resulting in substantial performance overheads that severely constrain high-throughput analyses. We aim to minimize this performance limitation by restructuring the storage scheme for typical satellite imagery as temporal sequences in order to reduce overheads and improve throughput. Models are developed to compute the expected query time for both the time-sequential and the traditional image-based representations. These models are used to demonstrate the benefits of using a time-sequential representation. Four data structures (using the Hierarchical Data Format (HDF5), Network Common Data Format (netCDF) and a native file system approach) are implemented and compared in a series of experimental read tests to determine which format is most appropriate for implementation in the CSIR Cluster Computing Centre's facilities.