Rclone Backups to AWS

Rclone provides a simple way to backup data from the HPC cluster to Amazon S3.

AWS IAM Setup

1. Log into AWS Console → IAM

2. Create new user: rclone-backup-from-hpc

3. Do not enable AWS Management Console access

4. Attach policy: AmazonS3FullAccess

5. Create access key for “Applications running outside of AWS”

6. Save your Access Key ID and Secret Access Key

Rclone Configuration

Load the Module

module load rclone/1.62.2

Run Configuration

rclone config

Enter the following when prompted:

Prompt	Value
Name	my-aws-account (or your choice)
Storage type	5 (S3 compatible)
Provider	1 (AWS)
Credentials	1 (Enter manually)
Access Key ID	Your AWS access key
Secret Access Key	Your AWS secret key
Region	us-west-2
Location constraint	us-west-2
ACL	(press Enter for default)
Storage class	STANDARD_IA (recommended for backups)

Usage

Create Bucket

rclone mkdir my-aws-account:/my-hpc-backups/

List Contents

rclone ls my-aws-account:/my-hpc-backups/

Copy Data

rclone copy ~/my-data my-aws-account:/my-hpc-backups/my-data/

With progress:

rclone copy -P ~/my-data my-aws-account:/my-hpc-backups/my-data/

Sync (Mirror)

rclone sync ~/my-data my-aws-account:/my-hpc-backups/my-data/

Warning

sync will delete files in the destination that don’t exist in the source. Use copy if you want to preserve old versions.

Automated Backups

Edit your crontab:

crontab -e

Daily at 4 AM

0 4 * * * /resnick/software/rclone/1.62.2/rclone copy ~/important-data my-aws-account:/my-hpc-backups/

Every 5 Minutes (Testing)

*/5 * * * * /resnick/software/rclone/1.62.2/rclone copy ~/test-data my-aws-account:/my-hpc-backups/

Configuration Location

Your rclone config is stored at:

~/.config/rclone/rclone.conf

Important Notes

• Use : after remote name (e.g., my-aws-account:)

• Backups are user-managed; monitor manually

• US-WEST-2 (Oregon) recommended for lower costs