Data Transfer - FSU Research Computing Center Documentation
Skip to content
Data Transfer
This page describes how to transfer data to and from the HPC filesystem (GPFS) and Research Archival filesystems using
SFTP, Globus, or Rsync.
Choosing a Transfer Option
There are three options for moving data to and from our storage systems:
Secure File Transfer Protocol (SFTP)
(on-campus or VPN only) - We recommend this method when your data transfer is
relatively small and will take only a few minutes, or when you are connected on-campus via a reliable high-speed network.
This method is quick and convenient, but less reliable for long-running transfers.
Globus
- We recommend this method when your data transfer will take a long time (at least several hours) and/or when the
connection may be periodically interrupted during transfer. This method requires a bit of setup ahead of time, but it is robust and
appropriate for any data transfer task. Globus is also the
fastest
method out of these three options.
Rsync/SCP
- We recommend this method when you are connected on-campus with a reliable high-speed network connection and your data transfer
is part of a scripted process. Globus is still a more reliable option in this case, but Rsync/SCP doesn't require as much setup
on the client-side.
Tip
If you are using SFTP, Rsync, or SCP, be sure to connect to
export.rcc.fsu.edu
and not a HPC login node.
The export cluster has a
40Gb/s
uplink to our parallel storage system (GPFS), and a
20Gb/s
uplink to our
Archival Storage system.
This is more than double the capacity of the HPC login node uplinks, and it will ensure that your transfers run
faster.
Using MobaXTerm (Windows)
MobaXTerm
is our recommended SSH client for Windows users. It also has out-of-the-box SFTP support.
Click the "Session" icon in the upper left-hand corner of the MobaXTerm window:
In the dialog that appears, click the "SFTP" icon:
In the
Remote Host
input, enter:
export.rcc.fsu.edu
. In the
Username
input, enter your RCC user account.
In the
Port
input, enter
22
. Click the
OK
button.
Enter your password when prompted. After that, you will be returned to the main SFTP interface:
Using SFTP with Cyberduck (Mac/Windows)
Cyberduck
is our recommended SFTP client option for Mac. It is also available for Windows.
Click the "Open Connection" button in the upper left-hand corner of the CyberDuck Window:
In the "Open Connection" dialog, select SFTP
(SSH File Transfer Protocol)
Enter
export.rcc.fsu.edu
in the Server textfield and your RCC User account credentials:
Using SFTP from Nautilus File Manager (Linux)
Most popular linux distributions come with the
Nautilus file manager
, which is part of the Gnome Desktop Environment.
If your distribution uses KDE, Cinnamon, or another desktop environment, you will need to look up the procedure using your file
manager
du jour
Open the
Files
app, and locate the
Other Locations
option on the left side of the file browser.
On the bottom of the window, enter
sftp://export.rcc.fsu.edu
in the
Connect to Server
dialog.
Enter your RCC user credentials in the dialog that appears.
Once logged in, the file system will appear as a location on the left-hand side of the Nautilus file manager. You can now
drag/drop, copy and paste, and interact with the files on the server the same way that you do with files on your computer. Press
the eject button to disconnect.
Using terminal commands
There are many command-line utilities to transfer data, and automate transferring data to and from RCC storage systems. Below
are some examples of commands supported on the RCC systems.
Warning
You need to be on-campus or connected to the
FSU VPN
to use the following commands.
The
sftp
command
Use SFTP to interact with files on the storage systems
10
11
12
13
14
15
16
17
18
19
20
$ sftp USERNAME@export.rcc.fsu.edu
USERNAME@export.rcc.fsu.edu's password:
Connected to export.rcc.fsu.edu.
sftp>
# List files on the server
sftp> ls
# Copy local file (/tmp/test.txt) to the server
sftp> put /tmp/test.txt
/tmp/test.txt 100% 88KB 88.5KB/s 00:00
# Copy remote file on server to computer (downloads to current working directory)
sftp> get test.txt
# Access archival storage directory
sftp> cd /mnt/archival/GROUP_PATH
# Disconnect
sftp> exit
Read more information about using SFTP at
this DigitalOcean tutorial page
The
scp
command
Use SCP to perform scripted, automated file transfers.
10
11
12
13
14
15
16
17
# Usage
$ scp [source-file(s)] [destination]
# Ex: use scp a file to your GPFS home directory
$ scp /tmp/test.txt USERNAME@export.rcc.fsu.edu:~
# Ex: use the `-r` (recursive) option to copy an entire directory to your GPFS home directory
$ scp -r ~/Documents/my_research_data USERNAME@export.rcc.fsu.edu:~
# Ex: copy data to a shared research volume on GPFS
$ scp /tmp/test.txt USERNAME@export.rcc.fsu.edu:/gpfs/research/GROUP_PATH/SUB_PATH
# Ex: copy data to a shared archival volume
$ scp /tmp/test.txt USERNAME@export.rcc.fsu.edu:/mnt/archival/GROUP_PATH/SUB_PATH
# Ex: copy data from the storage system to your local disk (. = current working path)
$ scp USERNAME@export.rcc.fsu.edu:~/test.txt .
Read more information about using SCP from
this Linuxize tutorial
The
rsync
command
Use RSYNC as an alternative to
scp
when you want to keep files synchronized between two systems:
# Use Rsync to copy a directory to your GPFS home directory
$ rsync -v -a -e ssh /tmp/test.txt USERNAME@export.rcc.fsu.edu:~
Read more about rsync at
this DigitalOcean tutorial page
The
rclone
command
Use Rclone to move data between cloud storage providers (AWS, OneDrive, etc.) and your storage space on our storage systems.
Warning
FSU has blocked inbound/outbound FTP connections throughout all of campus, so FTP remotes will not work
in rclone.
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
$ ssh USERNAME@export.rcc.fsu.edu
# Use the 'config' subcommand of rclone to enter an interactive setup utility.
[USERNAME@o22-export-31 ~]$ rclone config
Current remotes:
Name Type
==== ====
onedrive onedrive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> my-cloud-storage
Type of storage to configure.
Choose a number from below, or type in your own value
1 / Alias for a existing remote
\ "alias"
2 / Amazon Drive
\ "amazon cloud drive"
3 / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio)
\ "s3"
4 / Backblaze B2
\ "b2"
5 / Box
\ "box"
# etc...
For further information about rclone, refer to the
official documentation page
Data transfer between GPFS and Archival systems
To transfer data between GPFS and Archival storage systems, use the
export.rcc.fsu.edu
server. Both filesystems are
mounted on that server:
System
Mount path on export.rcc.fsu.edu
HPC filesystems (GPFS)
/gpfs/home
and
/gpfs/research
Archival
/mnt/arch
Note
As of November 1, 2024, we have transitioned all Archival volumes to a new storage backend. The new path for
all Archival volumes is
/mnt/arch
November 7, 2024