Transferring Files

This guide covers all methods for transferring files to and from the Resnick High Performance Computing Center.

Quick Reference 

Method	Best For	Requires
Open OnDemand	Small files, quick uploads	Web browser
Network Mount	Drag-and-drop convenience	Campus network or VPN
SCP/SFTP	Command-line workflows	SSH client
rsync	Syncing directories, resuming transfers	SSH client
SSHFS	Seamless local integration	macOS with FUSE
Globus	Large datasets, reliable transfers	Globus account
Cloud Storage	S3/GCS integration	Cloud credentials

Open OnDemand 

The easiest way to transfer files is through the Open OnDemand web interface.

Tip

You only need to authenticate with Duo once per session, allowing sustained access without repeated logins.

Navigate to the HPC Open OnDemand portal
Log in with your Caltech credentials
Use the Files app to upload, download, and manage files

Network Mounting 

Mount HPC storage directly to your local machine for drag-and-drop file transfers.

macOS

Open Finder
Press Cmd + K or select Go → Connect to Server
Enter one of the following addresses:

smb://fs.hpc.caltech.edu/groups/<groupname>
smb://fs.hpc.caltech.edu/home/<username>
smb://fs.hpc.caltech.edu/scratch

Authenticate with your Caltech credentials

Windows

Open File Explorer
Click on the address bar
Enter one of the following paths:

\\fs.hpc.caltech.edu\groups\<groupname>
\\fs.hpc.caltech.edu\home\<username>
\\fs.hpc.caltech.edu\scratch

Authenticate with your Caltech credentials

Note

Network mounting requires you to be on the campus network or connected via VPN.

SCP/SFTP 

Transfer files using secure command-line tools or graphical clients.

Command Line 

Upload a file to the cluster:

scp local_file.txt <username>@login.hpc.caltech.edu:~/destination/

Download a file from the cluster:

scp <username>@login.hpc.caltech.edu:~/remote_file.txt ./

Transfer entire directories (use -r flag):

scp -r local_directory/ <username>@login.hpc.caltech.edu:~/destination/

Graphical Clients 

For a visual interface, use one of these recommended SFTP clients:

Client	Platform	Download
WinSCP	Windows	winscp.net
FileZilla	Cross-platform	filezilla-project.org
Cyberduck	macOS, Windows	cyberduck.io

Connection settings:

Host: login.hpc.caltech.edu
Protocol: SFTP (SSH File Transfer)
Port: 22
Username: Your Caltech username

Tip

PuTTY users on Windows can use pscp and psftp for command-line transfers.

rsync is the best choice for syncing directories and for transfers you might need to resume — it copies only what’s changed, so re-running the same command after an interruption picks up where it left off.

Upload a directory to the cluster:

rsync -avzP local_directory/ <username>@login.hpc.caltech.edu:/resnick/groups/<groupname>/destination/

Download from the cluster:

rsync -avzP <username>@login.hpc.caltech.edu:/resnick/groups/<groupname>/results/ ./local_results/

The flags do the heavy lifting: -a preserves permissions and timestamps, -v is verbose, -z compresses data in transit, and -P shows a progress bar and keeps partially-transferred files so an interrupted run can resume.

Tip

The trailing slash on the source (local_directory/) matters — it copies the contents of the directory rather than the directory itself. Leave it off to copy the directory as a subdirectory of the destination.

SSHFS (macOS)

Mount remote HPC directories as local drives using SSHFS.

Installation 

Install macFUSE
Install SSHFS via Homebrew:
```
brew install sshfs
```

Usage 

Mount a remote directory:

sshfs -o allow_other,defer_permissions,auto_cache \
    <username>@login.hpc.caltech.edu:/home/<username> \
    ~/Desktop/HPC-Mount

Unmount when finished:

umount ~/Desktop/HPC-Mount

Warning

Create the mount point directory before running the sshfs command: mkdir ~/Desktop/HPC-Mount

Globus 

Globus provides reliable, high-performance transfers for large datasets.

Recommended for Large Transfers

Globus automatically handles retries, checksums, and can resume interrupted transfers. Use it for datasets larger than a few GB.

HPC Endpoint Details 

Property	Value
Endpoint Name	Resnick-HPC-Cluster
Endpoint UUID	`5be0c110-3df6-4c24-ae62-7fd88da26e3b`

Setup 

Log in to Globus Web App with your Caltech credentials
Search for Resnick-HPC-Cluster in the endpoint search
Authenticate when prompted
For transfers from your local machine, install Globus Connect Personal

Important

Transfer files to /groups/<groupname>/ rather than /home/. Group storage has significantly more capacity.

Cloud Storage 

Access cloud storage providers directly from the cluster.

Amazon S3 

# Load the AWS CLI module
module load awscli

# Configure credentials (first time only)
aws configure

# Download from S3
aws s3 cp s3://bucket-name/path/to/file.txt ./

# Upload to S3
aws s3 cp local_file.txt s3://bucket-name/path/

# Sync entire directories
aws s3 sync s3://bucket-name/data/ ~/local_data/ --recursive

Google Cloud Storage 

# Load the Google Cloud SDK module
module load gsutil

# Authenticate (first time only)
gcloud auth login

# Download from GCS
gsutil cp gs://bucket-name/path/to/file.txt ./

# Upload to GCS
gsutil cp local_file.txt gs://bucket-name/path/

# Sync directories
gsutil -m rsync -r gs://bucket-name/data/ ~/local_data/

Best Practices 

Performance Tips

Use /groups/ for large files — Home directories have limited quota
Compress before transferring — Use tar -czf archive.tar.gz directory/ for many small files
Use Globus for large datasets — Built-in reliability and parallelism
Avoid transferring to /scratch/ — It’s for computation, not storage

Getting Help 

If you encounter issues with file transfers:

Check your network connection and VPN status
Verify your credentials are correct
For Globus issues, see Globus Support
Contact HPC support for persistent problems