Jacob Hummel edited 2-Memory.tex  about 8 years ago

Commit id: 257a6c4214940186e927ff2d14ee4c495f452f8a

deletions | additions      

       

One of the primary goals of the \code{gadfly} project is to enable the analysis of large datasets on machines with limited memory.  Enabling this requires intelligent memory management, loading only the particle data of interest from the disk.  Fortunately the HDF5 protocol is well-suited to such non-contiguous file access, allowing not only individual data fields to be accessed independently, but also for the loading of select entries only from the field in question.  \code{Gadfly} allows two complementary approaches to minimizing the memory footprint.  The first method requires definition of a refinement criterion, such as particles above a given density threshold.  The resulting `refined' index can then be used to select only the corresponding values from subsequently loaded particle fields.  While this method is efficient, it is poorly suited to exploratory analysis where the proper refinement criterion may not be know \it{a priori}.