Wasabi is a popular cloud storage provider known for its affordability, speed, and compatibility with Amazon S3 APIs. Developers often choose Wasabi for its predictable pricing model and high-performance storage capabilities.

In this DevTip, we'll explore how to efficiently export files to Wasabi using the powerful command-line tool, rclone.

What is wasabi and why use it for cloud storage?

Wasabi provides cloud storage solutions that are fully compatible with Amazon S3 APIs, making it easy to integrate into existing workflows. It offers:

  • Predictable and affordable pricing
  • High-speed data transfers
  • Strong security and compliance
  • No egress or API request charges

How does rclone simplify file management in the cloud?

rclone is an open-source command-line tool designed to manage files across various cloud storage providers. It simplifies file transfers, synchronization, and backups with a straightforward CLI interface.

Installing and configuring rclone for wasabi

Installation

You can install rclone using one of these methods:

# Script installation (recommended)
sudo -v ; curl https://rclone.org/install.sh | sudo bash

# Or package manager installation
brew install rclone    # macOS with Homebrew
sudo apt install rclone # Debian/Ubuntu

Verify the installation:

rclone --version

Configuration

Configure rclone to connect to Wasabi:

rclone config

Follow the interactive prompts:

  1. Choose n for a new remote.
  2. Enter a name for the remote (e.g., wasabi).
  3. Choose Wasabi Object Storage from the storage options.
  4. Enter your Wasabi access key and secret key.
  5. Choose the correct endpoint for your region (e.g., s3.wasabisys.com).
  6. Leave the location constraint empty.
  7. Choose private or public-read for the ACL.
  8. Confirm the configuration.

Security considerations

When working with cloud storage credentials, security is paramount:

  • Store your configuration file securely - it contains sensitive credentials.
  • Default config location: ~/.config/rclone/rclone.conf (Linux/macOS) or %USERPROFILE%\.config\rclone\rclone.conf (Windows).
  • Use the --config flag to specify a different config location.
  • Consider using environment variables for credentials in automated scripts.

Basic rclone commands for exporting files

Copy files

To copy files from your local machine to Wasabi:

rclone copy /local/path wasabi:bucket-name/path

Sync files

To synchronize a local directory with Wasabi:

rclone sync /local/path wasabi:bucket-name/path

Advanced rclone features for efficient file transfer

Parallel transfers

Increase transfer speed by adjusting parallel transfers:

rclone copy /local/path wasabi:bucket-name/path --transfers=8

Filtering files

Use filters to include or exclude specific files:

rclone sync /local/path wasabi:bucket-name/path --exclude "*.tmp"

Bandwidth control

Limit bandwidth usage to avoid network congestion:

rclone copy /local/path wasabi:bucket-name/path --bwlimit=10M

Automating file exports to wasabi with rclone scripts

Automate regular backups or exports by creating a simple bash script:

#!/bin/bash

# Backup script to wasabi
rclone sync /local/path wasabi:bucket-name/path --transfers=8 --exclude "*.tmp"

Make the script executable and schedule it with cron:

chmod +x backup-to-wasabi.sh
crontab -e

Add a cron job to run daily at midnight:

0 0 * * * /path/to/backup-to-wasabi.sh

Troubleshooting common rclone issues with wasabi

Common issues

  1. Error 403 - Check your access credentials and bucket permissions.
  2. Connection reset - Verify your endpoint matches your bucket region.
  3. Config file not found - Use the absolute path with the --config flag.

Verification commands

Use these commands to test your configuration and diagnose issues:

# Test configuration
rclone lsd wasabi:

# Check bucket access
rclone ls wasabi:your-bucket

# Test upload speed
rclone copy --progress test.file wasabi:your-bucket/

Debugging

For more detailed error information, use the -v flag:

rclone -v copy /local/path wasabi:bucket-name/path

For even more verbose output, use -vv or -vvv.

Conclusion

Using rclone to export files to Wasabi simplifies cloud storage management, offering powerful features like parallel transfers, filtering, and automation. It's an excellent choice for developers looking for efficient and reliable file handling.

Transloadit offers native Wasabi integration via the 🤖 /wasabi/store Robot as part of our File Exporting service.