Scratch
Scratch Space or Scratch Storage refers to any storage at MSI that is impermanent, meaning all data it contains is deleted periodically or after a given event.
Global Scratch
Global Scratch (/scratch.global) is a large storage space that is available to all MSI users collectively. It has a total storage capacity of 910 TB, and quotas are on a per-group basis at a limit of 40 TB or 10 million files.
In order to use Global Scratch, users must access /scratch.global and create a directory named after their user there. The following command is a good starting point, creating the user directory, changing into it, and printing the full path:
mkdir /scratch.global/$USER && cd /scratch.global/$USER && pwd
Data on Global scratch will become eligible for deletion 30 days after file creation. As such, it is not recommended that users store any important data in Global Scratch. Users are advised to keep data larger datasets in Tier 2 so they may be restored into Global scratch as needed.
Local Scratch
Local scratch (/scratch.local) is temporary storage space specifically allocated to a job’s compute node(s) while it is running. It must be specifically requested by the job slurm script by using the --tmp flag.
Any data in that allocated storage is immediately deleted after a job has finished running. It is crucial that users include commands to transfer any relevant job-generated data to Tier 1 storage before the job terminates.
Data Backups
No scratch space has any kind of backup system. Any data that is deleted from scratch space at any point is gone forever. This goes to reinforce our recommendation to never store important data in scratch storage.
Core Dumps
How do I enable core dumps?
A core dump is essentially a snapshot of what was in computer memory when a running program was halted by an interrupt. Core dumps are mostly used by people who are writing their own code, using these files as input to advanced tools like debuggers. Core dumps can be quite large in size, so if you do not have a need for them, we would recommend that you not enable their creation.
How to Enable
You can use this command to enable core dumps and have them generated no mattter how large they are
ulimit -c unlimited
If desired, you can also configure a size limit as a number of 512-byte blocks. For example, a 1 GiB limit would be
ulimit -c 2097152
With this, cores larger than 1 GiB will not be generated.
Where to Find
Core dumps are placed in the running directory of your executablem and have the name core.<PID> where
Quirks and Edge Cases
Some data transfer methods, such as
s3cmd syncwith default settings, will preserve timestamps on transferred data. Accurate timestamps are important for correctly determining whether items have exceeded a 30-day period in scratch space, so it is recommended that users favor methods that update timestamps after transfers. Withs3cmd sync, for example, the--no-preserveflag can be passed to enable this.