Real-time data can be a game changer, especially when it comes to weather visualization. In this post, we explore how to create a live weather map composition pipeline using a shell script that fetches remote images via cURL and seamlessly processes them with ImageMagick. This approach not only saves disk I/O but also provides a flexible method for on-the-fly image transformation.

Version compatibility

This guide works with:

  • ImageMagick 6.9.x and newer
  • ImageMagick 7.x
  • cURL 7.x and newer

Overview

In many scenarios, you might need to integrate external imagery without the overhead of temporary storage. Tools like cURL and ImageMagick have long been staples in the developer toolkit. cURL handles transferring data from URLs in a lightweight manner, while ImageMagick is a powerful suite for image manipulation. Combining these tools via command-line pipelines allows you to automate image transformation tasks efficiently.

Setting up the pipeline

To demonstrate our image processing pipeline, we'll use NASA's public domain images to create a composite. This example shows how to overlay the NASA logo on a moon landing photograph, streaming both images directly into ImageMagick without saving intermediary files.

Process substitution requirements

Note: Process substitution (<(...)) requires a POSIX-compliant shell like Bash. It won't work in basic /bin/sh or Windows CMD.

Fetching and compositing images on-the-fly

#!/bin/bash
set -e

# Define remote image URLs (nasa public domain images)
background_url="https://images.nasa.gov/images/as11-40-5874~orig.jpg"
overlay_url="https://images.nasa.gov/images/nasa-logo-web-rgb.png"

# Error handling for image fetching
if ! curl -sf "$background_url" > /dev/null; then
    echo "Failed to fetch background image"
    exit 1
fi

if ! curl -sf "$overlay_url" > /dev/null; then
    echo "Failed to fetch overlay image"
    exit 1
fi

# ImageMagick 7.x syntax
magick composite \
    -gravity center \
    <(curl -s --max-time 30 "$overlay_url") \
    <(curl -s --max-time 30 "$background_url") \
    output.jpg

# For ImageMagick 6.x, use:
# Composite \
#     -gravity center \
#     <(curl -s --max-time 30 "$overlay_url") \
#     <(curl -s --max-time 30 "$background_url") \
#     Output.jpg

echo "Composited image saved as output.jpg"

Memory considerations

When processing large images directly from URLs, ensure your system has enough memory as both cURL and ImageMagick will need to buffer the data. For large images, consider this alternative approach:

#!/bin/bash
set -e

# Download images first for large files
curl -sLo background.jpg "$background_url"
curl -sLo overlay.jpg "$overlay_url"

# Process images
magick composite overlay.jpg background.jpg -gravity center output.jpg

# Clean up temporary files
rm background.jpg overlay.jpg

Security considerations

When working with remote images, implement these security practices:

  • Validate URLs before processing
  • Set appropriate timeouts for cURL requests
  • Check ImageMagick's security policy limits
  • Verify SSL certificates when using HTTPS
# Example with security measures
curl -sS \
    --max-time 30 \
    --ssl-no-revoke \
    --cert-status \
    --fail \
    "$image_url"

Real-world applications

This approach is particularly useful in scenarios such as:

  • Dynamic Dashboards: Updating weather or traffic maps in real time on web dashboards
  • Live Event Overlays: Compositing live data streams with real-time annotations
  • Automated Reporting: Generating periodic reports that combine various data sources
  • Content Delivery: Creating on-the-fly image variations for different devices

Error handling and optimization

Implement robust error handling and optimize performance with these techniques:

#!/bin/bash
set -e

# Function for safe image fetching
fetch_image() {
    local url="$1"
    local max_retries=3
    local retry_count=0

    while [ $retry_count -lt $max_retries ]; do
        if curl -sf --max-time 30 "$url" > /dev/null; then
            return 0
        fi
        retry_count=$((retry_count + 1))
        sleep 1
    done

    echo "Failed to fetch image after $max_retries attempts: $url" >&2
    return 1
}

# Main processing with error handling
if ! fetch_image "$background_url"; then
    exit 1
fi

Conclusion

By combining cURL and ImageMagick, you can build robust, real-time image processing pipelines that minimize disk usage and boost automation. This method is both streamlined and flexible, making it ideal for applications that demand up-to-date visuals without the latency of file I/O operations.

At Transloadit, we employ similar techniques within our image-manipulation-service to ensure efficient, real-time processing at scale.