In many development workflows, exporting files directly to cloud storage is a common requirement. Automating this process reduces errors and accelerates your workflow. In this guide, we explain how to batch export files to Amazon S3 by generating pre-signed URLs using AWS CLI (version 2 or later is recommended) and uploading via cURL in a shell script.

Prerequisites

Before you begin, ensure you have:

  • AWS CLI version 2 (or later) installed and configured with the appropriate credentials
  • cURL installed on your system
  • A Bash shell environment
  • An existing S3 bucket with the necessary permissions
  • An IAM user or role with permission to perform S3 operations

To install AWS CLI, run:

sudo apt-get update
sudo apt-get install -y python3-pip
pip3 install awscli
aws --version  # Verify installation (should display AWS CLI/2.x.x)

Configure AWS CLI with your credentials:

aws configure
# Follow the prompts to enter:
# - AWS access key id
# - AWS secret access key
# - default region
# - default output format

Understanding the components

Amazon S3 is a scalable storage solution that many developers rely on. By using pre-signed URLs, you can temporarily grant permission to upload a file without exposing your credentials. This method combines the automation capabilities of AWS CLI with the flexibility of cURL, enabling efficient file transfers.

Generating pre-signed URLs with AWS CLI

To securely upload a file, generate a pre-signed URL that permits the PUT operation on a specific S3 object. For example, the command below creates a URL valid for one hour:

aws s3 presign s3://my-bucket/my-object \
  --region your-region \
  --http-method PUT \
  --expires-in 3600 \
  --profile your-profile  # Omit or modify if not using multiple profiles

This command returns a URL that you can use with cURL to upload your file. Adjust the bucket name, object key, region, and expiration time as required.

Uploading files with cURL

After obtaining a pre-signed URL, you can upload a file with cURL. The -T flag indicates the file to transfer, and -fsSL ensures a silent operation with proper error management. The following command automatically detects the file's MIME type and uploads it:

curl -fsSL --retry 3 --retry-delay 2 \
     -T localfile.txt \
     -H "Content-Type: $(file -b --mime-type localfile.txt)" \
     "https://example-presigned-url"

Automating batch file exports

Handling multiple files manually can be tedious. The enhanced shell script below automates the process, incorporating improved error handling, directory checks, and file size validation (note the 5 GiB limit for single PUT uploads):

#!/bin/bash

set -euo pipefail  # Enable strict error handling

BUCKET="my-bucket"
EXPIRE=3600
FILES_DIR="/path/to/files"
REGION="your-region"

# Ensure the files directory exists
if [ ! -d "$FILES_DIR" ]; then
    echo "Directory $FILES_DIR does not exist." >&2
    exit 1
fi

# Check for AWS CLI installation
if ! command -v aws &> /dev/null; then
    echo "AWS CLI is not installed." >&2
    exit 1
fi

# Verify AWS credentials
if ! aws sts get-caller-identity &> /dev/null; then
    echo "AWS credentials are not configured properly." >&2
    exit 1
fi

for file in "$FILES_DIR"/*; do
    [[ -f "$file" ]] || continue  # Skip non-files

    filename=$(basename "$file")
    echo "Processing $filename..."

    # Check file size (limit: 5 GiB for single PUT)
    file_size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file")
    if [ "$file_size" -gt 5368709120 ]; then
        echo "File $filename is too large for a direct upload." >&2
        continue
    fi

    # Generate a pre-signed URL with error handling
    url=$(aws s3 presign "s3://$BUCKET/$filename" \
        --region "$REGION" \
        --http-method PUT \
        --expires-in "$EXPIRE" 2>/dev/null) || {
            echo "Failed to generate pre-signed URL for $filename." >&2
            continue
        }

    # Upload file using cURL
    echo "Uploading $filename..."
    curl -fsSL --retry 3 --retry-delay 2 \
         -T "$file" \
         -H "Content-Type: $(file -b --mime-type "$file")" \
         "$url" || {
             echo "Error occurred while uploading $filename." >&2
             continue
         }

    echo "Successfully uploaded $filename."

done

echo "Batch export completed."

Security considerations

When implementing this solution, keep these best practices in mind:

  • Use IAM roles rather than long-term access keys whenever possible
  • Set restrictive bucket policies to limit access
  • Grant only the minimum permissions necessary for your IAM users or roles
  • Monitor AWS CloudTrail for S3 access events
  • Consider using AWS KMS for encrypting sensitive data
  • Rotate credentials regularly and audit access patterns

Troubleshooting

If you encounter issues during the process, consider the following troubleshooting tips:

  • Verify AWS CLI credentials using aws configure list or aws sts get-caller-identity
  • Double-check the bucket permissions to ensure PUT operations are allowed
  • Adjust the cURL retry settings if you face intermittent network timeouts
  • Ensure the file size does not exceed the 5 GiB limit for single PUT uploads
  • Confirm that you are using the correct AWS region corresponding to your bucket
  • Validate that the content type is correctly detected by the file command

Conclusion

Combining AWS CLI for generating pre-signed URLs with cURL for file uploads offers a robust, open-source solution for batch exporting files to Amazon S3. This approach minimizes credential exposure while streamlining the export process. At Transloadit, our /s3/store robot employs similar techniques to facilitate smooth file exports. Adapt this script to meet your needs and enhance your workflow.

Happy coding!