Downloading Your Data

_________________________________________________________

JUMP TO TOPIC:

__________________________________________________________

Downloads from gc3fstorage.uoregon.edu

We've long provided downloads via our webserver located at gc3fstorage.uoregon.edu. There are two methods for downloading from our webserver: "wget" and "rclone"

wget

"wget" is a download tool found on many many systems. Traditionally, we've constructed a "wget" command (at the bottom of your download page) that you can copy and paste into a terminal, which will then download your data for you. It might look a little something like this:

wget -r -c -nH --cut-dirs=1 --no-parent --reject "index.html*" https://gc3fstorage.uoregon.edu/HXY5HDRX9/7777/

Running that command will result in all all the files and folders found at gc3fstorage.uoregon.edu/HXY5HDRX9/7777/ to be downloaded to a local folder named "7777".

While wget has traditionally worked fine, we have noticed that wget has recently started to fail to download files that are over 500GB. If you are encountering this error, you may wish to try rclone instead.

Please note also that the "-c" flag enables the "continue" functionality of wget. If you have a spotty internet connection, and your initial wget attempt fails halfway through your download, you can re-run the same "wget" command, and it will resume from where it left off.

rclone

"rclone" is a newer tool that supports large files (such as subreads.bam), and is available on many platforms. The easiest way to get it installed is to ask your friendly system adminstrator (i.e., the people administrering your cluster or compute system) to install it for you. If you want to install it yourself on your laptop or other system that you have administrator privileges on, see the instructions here.

We now construct a "rclone" command (at the bottom of your download page) that you can copy and paste into a terminal, which will then download your data for you. It might look a little something like this:

rclone copy --progress --http-url https://gc3fstorage.uoregon.edu/HXY5HDRX9/7777/ :http: ./7777

Running that command will result in all all the files and folders found at gc3fstorage.uoregon.edu/HXY5HDRX9/7777/ to be downloaded to a local folder named "7777".

Please note that rclone does not have any "resume a previously incomplete download" functionality like wget does. See the documentation here.

Downloads using Globus

We'll soon be adding documentation about getting Globus installed and running on your laptop or lab computer.

Data Retention Policy

After 30 days, data is written to a cold storage system consisting of single, non-redundant drives. We maintain this system as a last hope to retrieve your data after a loss event, but it should not be relied upon as a secure, long-term storage location.

If you have experienced a data loss event, please submit a Data Retrieval request through iLab. GC3F does not provide any warranty for data published more than 30 days prior, nor can we guarantee the timeliness or success of retrieval efforts. Each lane or cell of data incurs a $100 fee upon successful retrieval.