Data Management

This page describes how to organize production, publishable simulation results in a way that supports sponsor requirements, reproducibility, and good scientific practice.

Data Organization Approach

The group uses a paper-oriented data organization strategy. Simulation data should be associated with the publication in which it will eventually appear.

This keeps results close to the best available documentation and minimizes time lost when work passes between group members.

Paper Git Repository

Every paper should be under Git version control through Overleaf or GitHub. This applies even if the manuscript itself is written in Word for collaborator or journal reasons.

When you are ready to generate potentially publishable results, create the paper repository.

Before publication, use this naming convention:

PaperDescriptiveTitleMixCapsNoSpaces

Examples:

PaperElasticSolver
PaperMicrostructureEvolution

Counterexamples:

ElasticSolverPaper
Paper_ElasticSolver
paperelasticsolver
Paper Elastic Solver

After publication, rename the repository to match the Google Scholar BibTeX ID, such as authorlastnameYYYYfirstword.

Results Directory Structure

Create a results directory in the paper repository. Put all simulation data and postprocessing results there.

PaperDescriptiveName/
    main.tex
    main.bib
    figures/
        graphic.svg
        graphic.pdf
    results/
        TypeOne/
            README
            input
            hpc_batch.sh
            plotTypeOneStats.py
            TypeOneStats.pdf
            output_202310250824/
                metadata
                diff.patch
                plot_eta.pdf
                plot_temp.pdf
                00000cell/
                00001cell/
                outputdata.tar.gz
            output_202310250825/
            output_202310250826/
        TypeTwo/

Files in figures should be illustrations only, not simulation results. If a figure is editable, include both the editable source, such as an Inkscape SVG, and the rendered file used in the paper.

Files in results are simulation results. Any figure that contains simulation data must be stored in results, as close as possible to the data that generated it.

For example:

\includegraphics{results/TypeOne/output_202310250824/plot_eta.pdf}

This makes it clear how the visualization was generated.

Metadata

Each simulation directory must contain enough metadata to reproduce the result.

For Alamo, include the standard metadata outputs. For other codes, include whatever metadata is needed to recreate the calculation. Metadata files should be version controlled.

Raw simulation outputs may be too large for Git. They should still live in the same logical directory, but they should be excluded from version control and moved to archival storage as soon as practical.

Postprocessing Data

Most visualization and postprocessing is done in Python. All postprocessing scripts must be stored in version control. Raw Python scripts and Jupyter notebooks are both acceptable.

Postprocessing data should be stored as close as possible to the data being processed:

• If a visualization uses one simulation, store it in that simulation directory.
• If a visualization uses multiple simulations, store it in the lowest-level directory that contains all relevant simulations.

Name scripts so their outputs are obvious. For example, plot_eta.py should generate plot_eta.pdf.

Avoid generic script names such as:

analysis.py
get_x.py
randomfunctions.py

Avoid disconnected output names such as:

eta.pdf
myplot.pdf
output.pdf

If a project cannot follow the standard organization, add a README explaining the structure. Even then, the results directory must contain the information needed to generate the results and the outputs from those results.

Journal Submission

Some journals do not support complex LaTeX directory structures. In those cases, a submission-specific fork may be used to move or rename files for journal requirements.

The archival research structure should remain intact.

After acceptance, rename the repository according to the BibTeX tag.

Archiving

Most sponsored projects require data archiving. The group uses ISU Large Scale Storage for long-term archival storage.

Before archiving, compress large simulation data when possible. For example:

tar cvzf output.tar.gz *cell

Store the compressed output in the same simulation folder before transferring it to archival storage.

Example Workflow

Suppose you are working on PaperLargeDeformationElastic, which builds on Smith et al. 2015.

1. Find the BibTeX tag for the earlier paper, such as smith2015novel.
2. Check out the smith2015novel repository from Overleaf or GitHub.
3. In the LaTeX source, find where the relevant figure is generated:

\includegraphics{results/DamageModeling/output.20192930103/plot.pdf}

1. Go to results/DamageModeling/output.20192930103/ and inspect the metadata.
2. If rerunning the simulation is too expensive, locate the archived file:

smith2015novel/results/DamageModeling/output.20192930103/output.tar.gz

1. Copy the archive into the local clone, regenerate the plot using the local postprocessing script, and confirm the result.
2. Copy the needed input files and results into the corresponding folder in your new paper repository.
3. Add a note in the README explaining that the results derive from the previous work.

Good data organization prevents the original author from becoming a future bottleneck and makes prior work easier to understand, reproduce, and extend.