Reducing Tier 1 Footprint at MSI

This page gives a high-level overview of a common MSI workflow for moving inactive data off Tier 1 (primary storage) and into Tier 2 storage. The goal is to reduce Tier 1 footprint by identifying data that no longer needs high-performance storage, transferring it to the appropriate lower-cost storage tier, and then cleaning up the original Tier 1 copy once the transfer has been verified. Tier 2 is a good fit for data that does not need to stay on the high-performance filesystem but still needs to remain accessible.

1. Identify Inactive Data

The first stage is finding data on Tier 1 storage that would be a good candidate for transfer to Tier 2

  • Former user data

  • Project data that is no longer part of active shared workflows

  • Older outputs, archived inputs, or completed-project directories that need to be retained but not actively computed on If you do not currently have candidates for transfer out of Tier 1, we recommend to start by identifying former user data. You can do this with the command below, which will show storage usage for each group member, current or former, individually:

groupquota -a | grep -vi "current"
# replace "groupquota -a" with "groupquota -ag GROUP" for a specific group named "GROUP"

2. Request Data Release from the MSI Help Desk

If the data belongs to a former lab member or another owner, the next step is to request a data release through the MSI Help Desk, so that ownership of the data can be transferred. To request a data release, the PI or a group admin should email help@msi.umn.edu and include the following:

  • MSI group name

  • Directories (or former users) to release

  • New owner of the data (ideally the same group member making the transfer to Tier 2)

3. Create a Tier 2 Bucket

Once ownership and access are settled, you will need a destination in Tier 2. If the group does not already have a shared Tier 2 bucket set up, please follow the steps below to create it:

# 1. <Optional> Create a new bucket
s3cmd mb "s3://$(id -gn)-tier2"

# 2. Download a copy of our policy script.
s3cmd get s3://t2-tools/addpolicy.sh $HOME/

# 3. Call 'addpolicy.sh' with a list of users that need access
bash ~/addpolicy.sh --bucket-name "$(id -gn)-tier2" --userlist userA,userB,userC

If your group requires assistance creating the bucket, the group’s PI can email the MSI Help Desk at help@msi.umn.edu. At MSI, Tier 2 is available through Globus on the browser and S3 command line tools. For a simple transfer workflow, Globus is often the easiest interface.

4. Copy Data into Tier 2 Using Globus

After the Tier 2 destination is ready, transfer the data from Tier 1 to Tier 2 using Globus:

  • Open the “Files” tab in Globus

  • Set up the correct locations on the left and right:

    • UMN MSI Home for Tier 1 storage

    • UMN MSI Tier2 for a location on Tier 2

  • Explore the UMN MSI Home file tree to find and select data to transfer

  • Click “Start” at the top of that pane to start the transfer

Note: This step is only for copying. You are not deleting data from Tier 1 during this procedure. Be sure to verify that your data was correctly transferred to Tier 2 first, and then move on to the next step.

5. Remove the Data from Tier 1 After Verification

Only after the Tier 2 copy has been confirmed should the Tier 1 copy be removed. This final stage reduces the Tier 1 footprint by freeing space on MSI’s high-performance storage. Before deletion, make sure:

  • The transfer completed successfully

  • The destination bucket and path are correct

  • Collaborators know the new location Once those checks are done, the obsolete Tier 1 copy can be removed so the data no longer consumes premium primary storage.

Note: The output from groupquota may not be immediately up-to-date after the data is deleted.

End State

When this workflow is complete, inactive data has been moved off Tier 1 and placed into Tier 2, where it is better aligned with MSI’s storage model for less active datasets. The result is a smaller Tier 1 footprint, better use of high-performance storage, and a clearer separation between active research data and retained inactive data. Learn more about our data retention policies here.

Note: files in Tier 2 are not backed up by MSI and cannot be recovered if deleted or lost, so be sure to plan accordingly.