Large File Transfer with rsync: Complete Guide for Efficient Server Migration

Rsync is the industry-standard tool for efficient file synchronization and transfer between servers. Whether you're migrating terabytes of data, maintaining backup systems, or synchronizing production environments, rsync's delta-transfer algorithm, compression capabilities, and resume functionality make it the optimal choice for large file transfers. This comprehensive guide covers everything from basic usage to advanced optimization techniques for moving massive datasets between servers.

Understanding rsync and Its Advantages

Rsync (remote sync) is a fast, versatile file copying tool that can copy files locally, to/from remote hosts, or between remote hosts using an rsync daemon. Its key advantages include:

Core Benefits

1. Delta Transfer Algorithm: Only transfers differences between source and destination
2. Bandwidth Efficiency: Built-in compression reduces network usage
3. Resume Capability: Interrupted transfers can be resumed without starting over
4. Preservation: Maintains permissions, timestamps, ownership, and attributes
5. Incremental Transfers: Subsequent syncs only transfer changed files
6. Reliability: Checksums verify data integrity
7. Flexibility: Extensive filtering and exclusion options
8. Performance: Parallel file processing and efficient I/O

When to Use rsync

• Migrating large directories between servers (terabytes of data)
• Continuous synchronization during migration periods
• Backup and disaster recovery operations
• Website content synchronization
• Media library transfers
• Database file-level backups
• Virtual machine image migration
• Home directory migrations

rsync Fundamentals

Basic Syntax and Common Options

# Basic syntax
rsync [OPTIONS] SOURCE DESTINATION

# Local to local
rsync -avz /source/directory/ /destination/directory/

# Local to remote
rsync -avz /source/directory/ user@remote-host:/destination/directory/

# Remote to local
rsync -avz user@remote-host:/source/directory/ /destination/directory/

# Between remote hosts (through local host)
rsync -avz user@host1:/source/ user@host2:/destination/

Essential Options Explained

# -a, --archive: Archive mode (preserves almost everything)
# Equivalent to -rlptgoD:
#   -r: recursive
#   -l: copy symlinks as symlinks
#   -p: preserve permissions
#   -t: preserve modification times
#   -g: preserve group
#   -o: preserve owner
#   -D: preserve device files and special files

# -v, --verbose: Verbose output
# -z, --compress: Compress file data during transfer
# -h, --human-readable: Output numbers in human-readable format
# -P: Equivalent to --partial --progress
#   --partial: keep partially transferred files
#   --progress: show progress during transfer

# --delete: Delete files in destination that don't exist in source
# --delete-after: Delete after transfer, not before
# --delete-excluded: Also delete excluded files from destination

# -n, --dry-run: Perform a trial run with no changes made
# -i, --itemize-changes: Output a change-summary for all updates
# -u, --update: Skip files that are newer on the receiver

# --stats: Give some file-transfer stats
# --bwlimit=RATE: Limit bandwidth usage (KB/s)
# --exclude: Exclude files matching pattern
# --include: Include files matching pattern

# -e: Specify remote shell to use
# --rsync-path: Specify path to rsync on remote host
# --chmod: Override destination permissions

Pre-Transfer Planning and Preparation

Assess the Transfer Requirements

# Calculate total size of source data
du -sh /source/directory
du -h --max-depth=1 /source/directory | sort -h

# Count total files and directories
find /source/directory -type f | wc -l
find /source/directory -type d | wc -l

# Identify largest files
find /source/directory -type f -exec ls -lh {} \; | \
  awk '{print $5, $9}' | sort -h | tail -20

# Check available disk space on destination
ssh user@remote-host "df -h /destination"

# Estimate transfer time
# Formula: (Total Size / Network Speed) * 1.2 (overhead)
# Example: 1TB / 100Mbps = 1000GB / 12.5MB/s = 80,000 seconds ≈ 22 hours

Network and System Preparation

# Test network connectivity and bandwidth
ping -c 5 remote-host
mtr remote-host

# Test SSH connection
ssh user@remote-host "echo 'Connection successful'"

# Measure actual bandwidth
# Install iperf3 on both servers
sudo apt install iperf3 -y

# On destination server
iperf3 -s

# On source server
iperf3 -c remote-host

# Check SSH configuration for optimal performance
# On both servers
sudo nano /etc/ssh/sshd_config

# Optimize these settings:
# Compression yes
# TCPKeepAlive yes
# ClientAliveInterval 60
# ClientAliveCountMax 3

sudo systemctl restart sshd

# Increase SSH connection limits if needed
# MaxStartups 100:10:200
# MaxSessions 100

Pre-Transfer Checklist

• Calculate exact size of data to transfer
• Verify sufficient disk space on destination (1.2x source size)
• Test SSH connectivity and authentication
• Measure available bandwidth
• Check I/O performance on both servers
• Configure firewall rules if necessary
• Set up SSH keys for passwordless authentication
• Create destination directory structure
• Test with small dataset first
• Prepare monitoring and logging
• Schedule transfer during off-peak hours
• Notify stakeholders of planned transfer

Setting Up SSH Keys for Passwordless Authentication

# On source server: Generate SSH key
ssh-keygen -t ed25519 -C "migration-key"
# Or for older systems:
ssh-keygen -t rsa -b 4096 -C "migration-key"

# Copy public key to destination
ssh-copy-id user@remote-host

# Or manually:
cat ~/.ssh/id_ed25519.pub | ssh user@remote-host \
  "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

# Test passwordless login
ssh user@remote-host "echo 'Success'"

# Set proper permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Basic Transfer Examples

Simple Directory Transfer

# Basic transfer with common options
rsync -avzP /source/directory/ user@remote-host:/destination/directory/

# Note the trailing slash on source:
# /source/directory/  copies contents of directory
# /source/directory   copies directory itself

# Transfer with bandwidth limit (10MB/s = 10000KB/s)
rsync -avzP --bwlimit=10000 \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Dry run to test before actual transfer
rsync -avzPn /source/directory/ user@remote-host:/destination/directory/

Transfer with Progress and Statistics

# Detailed progress with file-by-file stats
rsync -avzP --stats \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Show only summary statistics
rsync -avz --stats --quiet \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Itemized changes for each file
rsync -avzi /source/directory/ user@remote-host:/destination/directory/

# Output explanation for -i:
# >f+++++++++ means new file
# .f....p.... means permissions changed
# .f...t..... means timestamp changed

Advanced Transfer Techniques

Incremental Transfers and Synchronization

# Synchronize directories (add --delete to remove files on destination)
rsync -avzP --delete \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Update only files that are newer on source
rsync -avzPu /source/directory/ user@remote-host:/destination/directory/

# Backup before deleting (move deleted files to backup dir)
rsync -avzP --delete --backup --backup-dir=/backup/deleted-$(date +%F) \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Compare directories without transferring
rsync -avzn --delete /source/ user@remote-host:/destination/ | \
  grep "^deleting\|>f"

Exclusion and Inclusion Patterns

# Exclude specific patterns
rsync -avzP \
  --exclude '*.log' \
  --exclude '*.tmp' \
  --exclude 'cache/' \
  --exclude '.git/' \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Exclude from file
cat > /tmp/rsync-exclude.txt << 'EOF'
*.log
*.tmp
cache/
temp/
.git/
node_modules/
*.pyc
__pycache__/
.DS_Store
Thumbs.db
EOF

rsync -avzP --exclude-from=/tmp/rsync-exclude.txt \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Include specific patterns, exclude rest
rsync -avzP \
  --include '*.jpg' \
  --include '*.png' \
  --include '*/' \
  --exclude '*' \
  /source/images/ \
  user@remote-host:/destination/images/

# Complex filtering
rsync -avzP \
  --include '**.php' \
  --include '**.html' \
  --exclude 'vendor/' \
  --exclude 'node_modules/' \
  /source/website/ \
  user@remote-host:/destination/website/

Handling Large Files Efficiently

# Transfer large files with checksum verification
rsync -avzP --checksum \
  /source/large-files/ \
  user@remote-host:/destination/large-files/

# Use less compression for already compressed files
rsync -avP --skip-compress=gz/zip/bz2/jpg/jpeg/png/mp3/mp4/avi/mov \
  /source/media/ \
  user@remote-host:/destination/media/

# Transfer without compression (faster for high-bandwidth, low-CPU scenarios)
rsync -avP --no-compress \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Partial directory transfers (transfer in chunks)
# Transfer first 1000 files
rsync -avzP --max-size=1G \
  /source/directory/ \
  user@remote-host:/destination/directory/ | head -1000

# Then transfer larger files
rsync -avzP --min-size=1G \
  /source/directory/ \
  user@remote-host:/destination/directory/

Performance Optimization

Network Optimization

# Use custom SSH options for better performance
rsync -avzP \
  -e "ssh -o Compression=no -o [email protected]" \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Multiple SSH connections (parallel transfers)
# Using GNU parallel (install: apt install parallel)
find /source -mindepth 1 -maxdepth 1 -type d | \
  parallel -j 4 rsync -avzP {} user@remote-host:/destination/

# Adjust TCP window size
rsync -avzP \
  -e "ssh -o 'TCPKeepAlive=yes' -o 'ServerAliveInterval=60'" \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Use rsync daemon for better performance (no SSH overhead)
# On destination server, configure /etc/rsyncd.conf
sudo tee /etc/rsyncd.conf << 'EOF'
uid = nobody
gid = nogroup
use chroot = no
max connections = 10
pid file = /var/run/rsyncd.pid
log file = /var/log/rsyncd.log

[backup]
path = /destination/directory
comment = Backup destination
read only = no
list = yes
auth users = rsyncuser
secrets file = /etc/rsyncd.secrets
hosts allow = source-server-ip
EOF

# Create secrets file
echo "rsyncuser:strongpassword" | sudo tee /etc/rsyncd.secrets
sudo chmod 600 /etc/rsyncd.secrets

# Start rsync daemon
sudo systemctl enable rsync
sudo systemctl start rsync

# Transfer using rsync daemon
rsync -avzP /source/directory/ rsyncuser@remote-host::backup

System-Level Optimization

# Increase system file limits
ulimit -n 65536

# Or permanently in /etc/security/limits.conf
echo "* soft nofile 65536" | sudo tee -a /etc/security/limits.conf
echo "* hard nofile 65536" | sudo tee -a /etc/security/limits.conf

# Tune kernel network parameters
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem='4096 87380 67108864'
sudo sysctl -w net.ipv4.tcp_wmem='4096 65536 67108864'
sudo sysctl -w net.ipv4.tcp_window_scaling=1

# Disable unnecessary SSH features for transfer
rsync -avzP \
  -e "ssh -c [email protected] -o StrictHostKeyChecking=no" \
  /source/directory/ \
  user@remote-host:/destination/directory/

I/O Optimization

# Reduce I/O priority to avoid system impact
ionice -c 3 rsync -avzP \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Limit CPU usage
nice -n 19 rsync -avzP \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Combine both
nice -n 19 ionice -c 3 rsync -avzP \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Use custom block size for large files
rsync -avzP --block-size=131072 \
  /source/large-files/ \
  user@remote-host:/destination/large-files/

Resumable Transfers and Error Handling

Handling Interrupted Transfers

# Partial transfer with resume capability (default with -P)
rsync -avzP /source/directory/ user@remote-host:/destination/directory/

# If interrupted, run same command again - it will resume

# Keep partial files explicitly
rsync -avz --partial --progress \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Store partial files in temporary directory
rsync -avzP --partial-dir=/tmp/rsync-partial \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Append data to shorter files (useful for log files)
rsync -avz --append \
  /source/logs/ \
  user@remote-host:/destination/logs/

# Use append with verification
rsync -avz --append-verify \
  /source/logs/ \
  user@remote-host:/destination/logs/

Robust Transfer Script with Retry Logic

# Create robust transfer script
cat > /root/rsync-with-retry.sh << 'EOF'
#!/bin/bash

SOURCE="/source/directory/"
DEST="user@remote-host:/destination/directory/"
LOGFILE="/var/log/rsync-transfer.log"
MAX_RETRIES=5
RETRY_DELAY=300  # 5 minutes

for i in $(seq 1 $MAX_RETRIES); do
  echo "=== Transfer attempt $i at $(date) ===" | tee -a $LOGFILE

  rsync -avzP \
    --timeout=300 \
    --contimeout=60 \
    --log-file=$LOGFILE \
    --exclude-from=/root/rsync-exclude.txt \
    $SOURCE $DEST

  EXIT_CODE=$?

  if [ $EXIT_CODE -eq 0 ]; then
    echo "=== Transfer completed successfully at $(date) ===" | tee -a $LOGFILE
    exit 0
  else
    echo "=== Transfer failed with exit code $EXIT_CODE ===" | tee -a $LOGFILE

    if [ $i -lt $MAX_RETRIES ]; then
      echo "Retrying in $RETRY_DELAY seconds..." | tee -a $LOGFILE
      sleep $RETRY_DELAY
    fi
  fi
done

echo "=== Transfer failed after $MAX_RETRIES attempts ===" | tee -a $LOGFILE
exit 1
EOF

chmod +x /root/rsync-with-retry.sh

Understanding Exit Codes

# Common rsync exit codes:
# 0  = Success
# 1  = Syntax or usage error
# 2  = Protocol incompatibility
# 3  = Errors selecting input/output files
# 4  = Unsupported action
# 5  = Error starting client-server protocol
# 10 = Error in socket I/O
# 11 = Error in file I/O
# 12 = Error in rsync protocol data stream
# 13 = Errors with program diagnostics
# 20 = Received SIGUSR1 or SIGINT
# 23 = Partial transfer due to error
# 24 = Partial transfer due to vanished source files
# 25 = Max --delete limit exceeded
# 30 = Timeout in data send/receive
# 35 = Timeout waiting for daemon connection

# Script to interpret exit codes
rsync -avzP /source/ user@remote-host:/destination/
EXIT_CODE=$?

case $EXIT_CODE in
  0)
    echo "Success"
    ;;
  23|24)
    echo "Partial transfer - resume recommended"
    ;;
  30)
    echo "Timeout - check network connectivity"
    ;;
  *)
    echo "Error occurred: $EXIT_CODE"
    ;;
esac

Monitoring and Logging

Real-Time Monitoring

# Detailed progress with estimated time
rsync -avzP --stats --human-readable \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Monitor in another terminal
watch -n 1 'du -sh /destination/directory'

# Monitor bandwidth usage
sudo iftop -i eth0

# Monitor with custom output format
rsync -avz --progress --out-format="%t %f %b" \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Create monitoring script
cat > /root/monitor-rsync.sh << 'EOF'
#!/bin/bash

SOURCE_SIZE=$(du -sb /source/directory | awk '{print $1}')
DEST_PATH="user@remote-host:/destination/directory"

while true; do
  DEST_SIZE=$(ssh user@remote-host "du -sb /destination/directory | awk '{print \$1}'")
  PERCENT=$(echo "scale=2; $DEST_SIZE * 100 / $SOURCE_SIZE" | bc)

  echo "$(date): $PERCENT% complete ($DEST_SIZE / $SOURCE_SIZE bytes)"

  if [ "$DEST_SIZE" -ge "$SOURCE_SIZE" ]; then
    echo "Transfer appears complete"
    break
  fi

  sleep 60
done
EOF

chmod +x /root/monitor-rsync.sh

Comprehensive Logging

# Transfer with detailed logging
rsync -avzP \
  --log-file=/var/log/rsync-transfer.log \
  --log-file-format="%t %o %f %b" \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Log only errors
rsync -avz --quiet \
  --log-file=/var/log/rsync-errors.log \
  /source/directory/ \
  user@remote-host:/destination/directory/ 2>&1

# Create detailed transfer report
cat > /root/rsync-with-report.sh << 'EOF'
#!/bin/bash

START_TIME=$(date +%s)
REPORT="/var/log/rsync-report-$(date +%F-%H%M%S).txt"

echo "=== Rsync Transfer Report ===" > $REPORT
echo "Start time: $(date)" >> $REPORT
echo "" >> $REPORT

rsync -avzP --stats \
  /source/directory/ \
  user@remote-host:/destination/directory/ 2>&1 | tee -a $REPORT

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

echo "" >> $REPORT
echo "=== Summary ===" >> $REPORT
echo "End time: $(date)" >> $REPORT
echo "Duration: $DURATION seconds" >> $REPORT

# Email report
mail -s "Rsync Transfer Report" [email protected] < $REPORT
EOF

chmod +x /root/rsync-with-report.sh

Verification and Integrity Checking

Post-Transfer Verification

# Compare file counts
echo "Source files:"
find /source/directory -type f | wc -l

echo "Destination files:"
ssh user@remote-host "find /destination/directory -type f | wc -l"

# Compare total sizes
du -sb /source/directory
ssh user@remote-host "du -sb /destination/directory"

# Generate checksums for verification
find /source/directory -type f -exec md5sum {} \; | \
  sort -k 2 > /tmp/source-checksums.txt

ssh user@remote-host \
  "find /destination/directory -type f -exec md5sum {} \; | sort -k 2" \
  > /tmp/dest-checksums.txt

diff /tmp/source-checksums.txt /tmp/dest-checksums.txt

# Use rsync's checksum option for verification
rsync -avzc --dry-run --itemize-changes \
  /source/directory/ \
  user@remote-host:/destination/directory/

# Any differences will be shown; if none, output is minimal

Continuous Verification Script

cat > /root/verify-transfer.sh << 'EOF'
#!/bin/bash

SOURCE="/source/directory"
DEST="user@remote-host:/destination/directory"

echo "Verifying transfer integrity..."

# Check if all source files exist on destination
find $SOURCE -type f | while read file; do
  rel_path="${file#$SOURCE}"

  if ! ssh user@remote-host "[ -f /destination/directory$rel_path ]"; then
    echo "MISSING: $rel_path"
  fi
done

echo "Verification complete"
EOF

chmod +x /root/verify-transfer.sh

Production-Ready Transfer Scripts

Complete Migration Script

cat > /root/production-rsync-migration.sh << 'EOF'
#!/bin/bash

set -euo pipefail

# Configuration
SOURCE_DIR="/source/directory/"
DEST_HOST="user@remote-host"
DEST_DIR="/destination/directory/"
LOG_DIR="/var/log/migration"
EXCLUDE_FILE="/root/rsync-exclude.txt"
BANDWIDTH_LIMIT=50000  # KB/s
MAX_RETRIES=3

# Create log directory
mkdir -p $LOG_DIR
LOGFILE="$LOG_DIR/rsync-$(date +%F-%H%M%S).log"

# Logging function
log() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a $LOGFILE
}

# Error handler
error_exit() {
  log "ERROR: $1"
  exit 1
}

# Validate source
[ -d "$SOURCE_DIR" ] || error_exit "Source directory not found"

# Test destination connectivity
ssh $DEST_HOST "mkdir -p $DEST_DIR" || error_exit "Cannot create destination directory"

log "Starting migration from $SOURCE_DIR to $DEST_HOST:$DEST_DIR"

# Calculate source size
SOURCE_SIZE=$(du -sb $SOURCE_DIR | awk '{print $1}')
log "Source size: $(numfmt --to=iec-i --suffix=B $SOURCE_SIZE)"

# Initial sync
log "Phase 1: Initial synchronization"
rsync -avzP \
  --bwlimit=$BANDWIDTH_LIMIT \
  --timeout=300 \
  --exclude-from=$EXCLUDE_FILE \
  --log-file=$LOGFILE \
  $SOURCE_DIR \
  $DEST_HOST:$DEST_DIR || log "Initial sync had errors, continuing..."

# Incremental syncs
for i in $(seq 1 $MAX_RETRIES); do
  log "Phase 2: Incremental sync #$i"

  rsync -avzP \
    --bwlimit=$BANDWIDTH_LIMIT \
    --timeout=300 \
    --exclude-from=$EXCLUDE_FILE \
    --log-file=$LOGFILE \
    $SOURCE_DIR \
    $DEST_HOST:$DEST_DIR

  if [ $? -eq 0 ]; then
    break
  fi

  sleep 60
done

# Final verification sync
log "Phase 3: Final verification sync"
rsync -avzP \
  --checksum \
  --dry-run \
  --itemize-changes \
  $SOURCE_DIR \
  $DEST_HOST:$DEST_DIR | tee -a $LOGFILE

# Verify destination
DEST_SIZE=$(ssh $DEST_HOST "du -sb $DEST_DIR | awk '{print \$1}'")
log "Destination size: $(numfmt --to=iec-i --suffix=B $DEST_SIZE)"

if [ "$SOURCE_SIZE" -eq "$DEST_SIZE" ]; then
  log "SUCCESS: Migration completed successfully"
  exit 0
else
  log "WARNING: Size mismatch detected"
  exit 1
fi
EOF

chmod +x /root/production-rsync-migration.sh

Best Practices and Recommendations

Security Best Practices

# Always use SSH for transport
# Never use rsh or unencrypted protocols

# Restrict rsync daemon access
# In /etc/rsyncd.conf:
# hosts allow = specific-ip-address
# use chroot = yes
# read only = yes  # unless writing is required

# Use SSH keys with passphrases
ssh-keygen -t ed25519 -f ~/.ssh/migration_key
ssh-add ~/.ssh/migration_key

# Limit rsync permissions
# Run as non-root user when possible
# Use sudo only for specific operations

Performance Best Practices

# For many small files: reduce compression, increase block size
rsync -av --no-compress --block-size=131072

# For few large files: use compression, enable partial transfers
rsync -avzP --partial

# For fast networks: disable compression
rsync -av --no-compress

# For slow networks: enable maximum compression
rsync -avzP --compress-level=9

# For continuous sync: use --delete-after and --delay-updates
rsync -avzP --delete-after --delay-updates

Troubleshooting Common Issues

# Issue: "connection timed out"
# Solution: Increase timeout values
rsync -avzP --timeout=600 --contimeout=120

# Issue: "permission denied"
# Solution: Check SSH keys and destination permissions
ssh user@remote-host "ls -la /destination/"

# Issue: "no space left on device"
# Solution: Check disk space
ssh user@remote-host "df -h /destination"

# Issue: "protocol version mismatch"
# Solution: Specify rsync path on remote
rsync -avzP --rsync-path=/usr/local/bin/rsync

# Issue: Too slow on many small files
# Solution: Use tar over SSH
tar czf - /source/directory | ssh user@remote-host "tar xzf - -C /destination"

# Issue: Network interruptions
# Solution: Use screen or tmux with retry script
screen -S rsync-transfer
# Run rsync inside screen
# Detach with Ctrl+A, D

Conclusion

Rsync is an indispensable tool for large file transfers and server migrations. Key takeaways:

1. Plan thoroughly: Assess size, bandwidth, and requirements before starting
2. Test first: Always do a dry run with -n flag
3. Use appropriate options: -avzP covers most use cases
4. Implement retry logic: Networks fail; be prepared to resume
5. Monitor actively: Watch progress and system resources
6. Verify integrity: Always check that transfers completed successfully
7. Optimize wisely: Tune based on your specific network and file characteristics
8. Log everything: Maintain detailed logs for troubleshooting
9. Security first: Use SSH keys and encrypted connections
10. Automate carefully: Scripts should handle errors gracefully

Whether transferring gigabytes or terabytes, rsync's efficiency, reliability, and flexibility make it the optimal choice for production server migrations. Master these techniques, and you'll be equipped to handle any large-scale data transfer scenario with confidence.