Managing Tier 1 User Data

Tier 1 user data at MSI commonly spans three storage locations:

  • private home directory: $HOME

  • project space: $MSIPROJECT, $SHARED, and $PUBLIC

  • global scratch: $SCRATCH or $SCRATCH_GLOBAL

These quotas are independent. Use userquota for your private home directory. Use groupquota for project space and global scratch.

The Shared directory (/projects/standard/group/shared) is the primary location that project members should store their project related data that is actively being used. All members of a group have read and write access to their group’s Shared directory, but access permissions for directories and data within the Shared directory will vary, depending on how the data owner has set them up. By default, when a user creates a new folder in the shared directory, other project members do not have read or write access. You will need to add this access, see below How do I share files with project members. If you need access to data from a project member who is no longer a member of your project, please ask your PI or a group admin to contact the Help Desk.

Checking current usage

Users can check quota from a remote shell or a login node.

Private home directory

Use userquota.

Example output from userquota:

11:59:23 [vega0051@ahl02 ~ ]$ userquota
Quota for user 'vega0051'
------------------------
BYTES        |
Usage        | 45.00 GB
Quota        | 200.00 GB
Percent used | 22.5 %
------------------------
FILES        |
Usage        | 176,452
Quota        | 1,000,000
Percent used | 17.6 %

Project space and global scratch

Use groupquota to review project space and global scratch for your primary group. If you belong to multiple groups, use groupquota -g GROUPNAME -r for a specific group.

Example output from groupquota may look similar to:

11:59:58 [vega0051@ahl02 ~ ]$ groupquota
Quota for group 'example-lab'
------------------------
PROJECT SPACE
BYTES        |
Usage        | 7.52 TB
Quota        | 10.00 TB
Percent used | 75.2 %
------------------------
FILES        |
Usage        | 3,841,219
Quota        | 5,000,000
Percent used | 76.8 %

------------------------
SCRATCH GLOBAL
BYTES        |
Usage        | 12.40 TB
Quota        | 40.00 TB
Percent used | 31.0 %
------------------------
FILES        |
Usage        | 4,112,804
Quota        | 13,200,000
Percent used | 31.2 %

If BYTES or FILES usage in userquota exceeds 100%, see below on how to identify and reduce home directory usage.

If project space or global scratch in groupquota is close to 100%, use groupquota -a to see approximate usage by user and groupquota -a -s files when file count is the main issue.

Reducing home directory BYTES usage

If you are not sure which directory is consuming most of your home directory quota, open a new remote shell and execute:

find $HOME -maxdepth 1 -exec du -sh '{}' \; | sort -sh

This measures the sizes of the top-level directories and files and sorts them from smallest to largest.

Once the larger directories have been identified, MSI recommends relocating them to your project’s /shared directory.

If you do not already have a user directory in /shared, create one with:

mkdir -p $SHARED/$USER

Note that $SHARED in this example points to your primary project’s shared directory. If you want to create a directory in a secondary project, you will need to enter the full path.

An example of moving a directory or file to /shared:

mv /path/to/file.txt $SHARED/$USER

Reducing home directory FILES usage

If you are not sure which directories are consuming most of the file count limit, open a new remote shell and execute the following commands.

Use these commands to create a directory in /scratch.global/, if it is not already present, make a local copy of the filecount.sh shell script, and execute it on the home directory:

mkdir -p $SCRATCH/$USER
s3cmd get s3://helpdesk/filecount.sh $SCRATCH/$USER
bash $SCRATCH/$USER/filecount.sh $HOME

If you have a significant number of files, this will take some time to count. The output on the screen will show the 10 directories with the largest number of files. Additionally, the output will be saved to your scratch global directory as a text file similar to INTERNETID_filecount_TIMESTAMP.txt.

For example, a directory with 170k files takes about 1 minute to traverse.

Managing project space and global scratch usage

Project space and global scratch quotas are attached to the MSI group, so coordinate with your collaborators before deleting shared data.

  • Use groupquota -a to identify which group members are consuming the most storage.

  • Use groupquota -a -s files when file count, rather than total bytes, is the main problem.

  • Move inactive project data off Tier 1 when appropriate. See Reducing Tier 1 Footprint at MSI.

  • Remove transient data from global scratch when it is no longer needed. Global scratch is temporary storage and should not be used for long-term retention.

Sharing files with MSI members

How do I share files with project members?

MSI offers the directory /projects/standard/<project-name>/shared in the project space for collaboration between group members.

To allow all group members to read/write a specific directory whoever currently owns the files must copy the files to the shared space and run this command in the Linux prompt:

chmod -R g+rwx /projects/standard/projectname/shared/filestoshare

If you want all new files you create/copy over to be group read/writable, put this line in your .bashrc:

umask 007

How do I share files with members of other projects?

/scratch.global is a shared storage space accessible by all MSI users. Data stored in /scratch.global has a lifespan of 30 days, after which it is automatically removed. This makes the space ideal for temporary files, such as those you might wish to share with MSI members outside your project. To access /scratch.global with the cd command:

cd /scratch.global

or use the global identifier:

cd $SCRATCH

Please note that you may need to change the permissions on files and directories added to /scratch.global to allow others to access them.

Further Guidance

If you need help interpreting quota reports or deciding what to move, email MSI support and staff may be able to help.