Managing Tier 1 User Data

Tier 1 user data at MSI commonly spans three storage locations:

private home directory: $HOME
project space: $MSIPROJECT, $SHARED, and $PUBLIC
global scratch: $SCRATCH or $SCRATCH_GLOBAL

These quotas are independent. Use userquota for your private home directory. Use groupquota for project space and global scratch.

Interactive Apps missing on Open OnDemand

Open OnDemand stores interactive session job scripts and logs in ~/.ondemand, so it depends on free space in your private home directory.

Each user is allocated a private home directory on /users/[0-9]/INTERNETID with:

200 GB storage limit
1 million file count limit

If either home directory limit is reached, the Interactive Apps interface may disappear. Reducing home directory usage below the limit restores the ability to create new sessions on Open OnDemand.

Checking current usage

Users can check quota from a remote shell or a login node.

Private home directory

Use userquota.

Example output from userquota:

11:59:23 [vega0051@ahl02 ~ ]$ userquota
Quota for user 'vega0051'
------------------------
BYTES        |
Usage        | 45.00 GB
Quota        | 200.00 GB
Percent used | 22.5 %
------------------------
FILES        |
Usage        | 176,452
Quota        | 1,000,000
Percent used | 17.6 %

Project space and global scratch

Use groupquota to review project space and global scratch for your primary group. If you belong to multiple groups, use groupquota -g GROUPNAME -r for a specific group.

Example output from groupquota may look similar to:

11:59:58 [vega0051@ahl02 ~ ]$ groupquota
Quota for group 'example-lab'
------------------------
PROJECT SPACE
BYTES        |
Usage        | 7.52 TB
Quota        | 10.00 TB
Percent used | 75.2 %
------------------------
FILES        |
Usage        | 3,841,219
Quota        | 5,000,000
Percent used | 76.8 %

------------------------
SCRATCH GLOBAL
BYTES        |
Usage        | 12.40 TB
Quota        | 40.00 TB
Percent used | 31.0 %
------------------------
FILES        |
Usage        | 4,112,804
Quota        | 13,200,000
Percent used | 31.2 %

If BYTES or FILES usage in userquota exceeds 100%, see below on how to identify and reduce home directory usage.

If project space or global scratch in groupquota is close to 100%, use groupquota -a to see approximate usage by user and groupquota -a -s files when file count is the main issue.

Reducing home directory BYTES usage

If you are not sure which directory is consuming most of your home directory quota, open a new remote shell and execute:

find $HOME -maxdepth 1 -exec du -sh '{}' \; | sort -sh

This measures the sizes of the top-level directories and files and sorts them from smallest to largest.

Once the larger directories have been identified, MSI recommends relocating them to your project’s /shared directory.

If you do not already have a user directory in /shared, create one with:

mkdir -p $SHARED/$USER

Note that $SHARED in this example points to your primary project’s shared directory. If you want to create a directory in a secondary project, you will need to enter the full path.

An example of moving a directory or file to /shared:

mv /path/to/file.txt $SHARED/$USER

Reducing home directory FILES usage

If you are not sure which directories are consuming most of the file count limit, open a new remote shell and execute the following commands.

Use these commands to create a directory in /scratch.global/, if it is not already present, make a local copy of the filecount.sh shell script, and execute it on the home directory:

mkdir -p $SCRATCH/$USER
s3cmd get s3://helpdesk/filecount.sh $SCRATCH/$USER
bash $SCRATCH/$USER/filecount.sh $HOME

If you have a significant number of files, this will take some time to count. The output on the screen will show the 10 directories with the largest number of files. Additionally, the output will be saved to your scratch global directory as a text file similar to INTERNETID_filecount_TIMESTAMP.txt.

For example, a directory with 170k files takes about 1 minute to traverse.

Managing project space and global scratch usage

Project space and global scratch quotas are attached to the MSI group, so coordinate with your collaborators before deleting shared data.

Use groupquota -a to identify which group members are consuming the most storage.
Use groupquota -a -s files when file count, rather than total bytes, is the main problem.
Move inactive project data off Tier 1 when appropriate. See Reducing Tier 1 Footprint at MSI.
Remove transient data from global scratch when it is no longer needed. Global scratch is temporary storage and should not be used for long-term retention.

Further Guidance

If you need help interpreting quota reports or deciding what to move, email MSI support and staff may be able to help.