AWS S3 Sync for Backup and Storage

AWS S3 combined with the CLI's sync command provides a reliable, affordable solution for automating server backups from Linux VPS and baremetal servers. This guide covers S3 bucket creation, IAM policy setup for least-privilege access, encryption, lifecycle rules, and scheduling automated backup scripts with cron.

Prerequisites

  • AWS CLI v2 installed and configured (see the AWS CLI installation guide)
  • An AWS account with permissions to manage S3 and IAM
  • The server data you want to back up

Creating an S3 Bucket

# Create a bucket (name must be globally unique)
aws s3 mb s3://my-server-backups-2024 --region us-east-1

# Enable versioning to protect against accidental deletion
aws s3api put-bucket-versioning \
  --bucket my-server-backups-2024 \
  --versioning-configuration Status=Enabled

# Block all public access (critical for backup buckets)
aws s3api put-public-access-block \
  --bucket my-server-backups-2024 \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Confirm public access is blocked
aws s3api get-public-access-block --bucket my-server-backups-2024

IAM Policy for Backup Access

Create a minimal IAM policy that grants only the permissions needed for backup operations.

# Create the IAM policy document
cat > backup-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowBucketList",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::my-server-backups-2024"
    },
    {
      "Sid": "AllowObjectOperations",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::my-server-backups-2024/*"
    }
  ]
}
EOF

# Create the policy in AWS
aws iam create-policy \
  --policy-name ServerBackupPolicy \
  --policy-document file://backup-policy.json

# Create a dedicated IAM user for backups
aws iam create-user --user-name backup-agent

# Attach the policy to the user
aws iam attach-user-policy \
  --user-name backup-agent \
  --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/ServerBackupPolicy

# Create access keys for the backup user
aws iam create-access-key --user-name backup-agent

Configure the backup credentials as a named profile:

aws configure --profile backup
# Enter the access key and secret for the backup-agent user

Basic S3 Sync Usage

# Sync a directory to S3 (uploads new and changed files)
aws s3 sync /var/www/html s3://my-server-backups-2024/www/ --profile backup

# Sync with deletion (mirror local directory exactly)
aws s3 sync /var/www/html s3://my-server-backups-2024/www/ \
  --delete \
  --profile backup

# Exclude specific files or directories
aws s3 sync /home s3://my-server-backups-2024/home/ \
  --exclude "*.tmp" \
  --exclude "*/.cache/*" \
  --exclude "*/node_modules/*" \
  --profile backup

# Preview what would be synced (dry run)
aws s3 sync /var/backups s3://my-server-backups-2024/db/ \
  --dryrun \
  --profile backup

# Restore from S3 back to server
aws s3 sync s3://my-server-backups-2024/www/ /var/www/html/ --profile backup

Encryption and Security

Server-Side Encryption

# Sync with AES-256 encryption (managed by AWS)
aws s3 sync /var/backups s3://my-server-backups-2024/db/ \
  --sse AES256 \
  --profile backup

# Enforce encryption via bucket policy
cat > enforce-encryption.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnencryptedObjectUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-server-backups-2024/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      }
    }
  ]
}
EOF

aws s3api put-bucket-policy \
  --bucket my-server-backups-2024 \
  --policy file://enforce-encryption.json

Client-Side Encryption with GPG

# Encrypt a database dump before uploading
mysqldump mydb | gzip | gpg --encrypt --recipient [email protected] > db.sql.gz.gpg

# Upload the encrypted file
aws s3 cp db.sql.gz.gpg s3://my-server-backups-2024/db/ --profile backup

Lifecycle Rules for Cost Control

Move old backups to cheaper storage classes automatically.

cat > lifecycle.json << 'EOF'
{
  "Rules": [
    {
      "ID": "MoveToIA",
      "Status": "Enabled",
      "Filter": {"Prefix": ""},
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "NoncurrentVersionTransitions": [
        {
          "NoncurrentDays": 7,
          "StorageClass": "STANDARD_IA"
        }
      ],
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 90
      }
    }
  ]
}
EOF

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-server-backups-2024 \
  --lifecycle-configuration file://lifecycle.json

Automated Backup Script

cat > /usr/local/bin/s3-backup.sh << 'EOF'
#!/bin/bash
# S3 backup script — runs via cron

BUCKET="s3://my-server-backups-2024"
PROFILE="backup"
DATE=$(date +%Y-%m-%d)
LOG="/var/log/s3-backup.log"

log() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG"
}

log "=== Starting backup ==="

# Backup web files
log "Syncing /var/www..."
aws s3 sync /var/www "$BUCKET/www/" \
  --sse AES256 \
  --exclude "*.tmp" \
  --exclude "*/.git/*" \
  --profile "$PROFILE" >> "$LOG" 2>&1

# Backup MySQL databases
log "Dumping databases..."
mkdir -p /tmp/db-backups
for DB in $(mysql -e "SHOW DATABASES;" 2>/dev/null | grep -Ev "(Database|information_schema|performance_schema|sys)"); do
  DUMPFILE="/tmp/db-backups/${DB}-${DATE}.sql.gz"
  mysqldump "$DB" 2>/dev/null | gzip > "$DUMPFILE"
  aws s3 cp "$DUMPFILE" "$BUCKET/databases/" \
    --sse AES256 \
    --profile "$PROFILE" >> "$LOG" 2>&1
  rm -f "$DUMPFILE"
done

# Backup /etc configuration
log "Syncing /etc..."
aws s3 sync /etc "$BUCKET/etc/" \
  --sse AES256 \
  --exclude "*.swp" \
  --profile "$PROFILE" >> "$LOG" 2>&1

log "=== Backup complete ==="
EOF

chmod +x /usr/local/bin/s3-backup.sh

Scheduling with Cron

# Edit crontab for root
crontab -e

# Add these lines:
# Daily full backup at 2:00 AM
0 2 * * * /usr/local/bin/s3-backup.sh

# Hourly sync of web files
0 * * * * aws s3 sync /var/www s3://my-server-backups-2024/www/ --sse AES256 --profile backup >> /var/log/s3-hourly.log 2>&1

Verify cron is working:

# Check last run log
tail -50 /var/log/s3-backup.log

# Manually test the script
/usr/local/bin/s3-backup.sh

# List recent uploads
aws s3 ls s3://my-server-backups-2024/ --recursive \
  --human-readable \
  --profile backup | sort | tail -20

Troubleshooting

Upload failures / partial syncs

# Re-run sync — it only transfers missing or changed files
aws s3 sync /var/www s3://my-server-backups-2024/www/ --profile backup

# Check for permission errors in the log
grep "ERROR" /var/log/s3-backup.log

"Access Denied" errors

# Verify credentials
aws sts get-caller-identity --profile backup

# Check the bucket policy isn't blocking uploads
aws s3api get-bucket-policy --bucket my-server-backups-2024

Slow transfer speeds

# Increase multipart threshold and concurrency
aws configure set default.s3.multipart_threshold 64MB --profile backup
aws configure set default.s3.max_concurrent_requests 20 --profile backup
aws configure set default.s3.multipart_chunksize 16MB --profile backup

Conclusion

AWS S3 sync provides a robust, low-cost backup solution for Linux servers when combined with IAM least-privilege policies, server-side encryption, and lifecycle rules. Automating backups with a cron-scheduled shell script ensures your data is consistently protected without manual intervention.