Cloud-init Configuration for Server Provisioning

Cloud-init is a powerful tool for automating initial server configuration across major cloud providers. It provides a standard way to configure instances on first boot using cloud-config YAML syntax, enabling infrastructure-as-code provisioning without external tools. This guide covers cloud-config fundamentals, user data specification, package installation, user management, custom scripts, file provisioning, and advanced features.

Table of Contents

1. Cloud-init Overview
2. Cloud-Config Syntax
3. Package Installation
4. User and Group Management
5. Running Commands
6. File Management
7. Network Configuration
8. Cloud-init Callbacks
9. Debugging Cloud-init
10. Conclusion

Cloud-init Overview

Cloud-init is the industry standard for cloud instance initialization. It runs on first boot with root privileges, making it perfect for initial system configuration, package installation, user creation, and custom setup tasks.

Key benefits:

• Standard Across Cloud Providers: Works on AWS, Azure, GCP, DigitalOcean, Linode
• Simple YAML Syntax: Cloud-config provides readable, declarative configuration
• Fast Boot: Configured on first boot, doesn't require additional software
• Repeatable: Same configuration produces consistent results every time
• Extensible: Support for custom scripts and modules
• Integration Ready: Works with Terraform, CloudFormation, Ansible

Cloud-init workflow:

Instance Launch
    ↓
Cloud-init Agent Starts (root)
    ↓
Fetch User Data
    ↓
Parse Cloud-Config
    ↓
Execute Initialization Steps
    │
    ├── Module: Install Packages
    ├── Module: Create Users
    ├── Module: Write Files
    ├── Module: Run Commands
    └── Module: Configure Services
    ↓
Initialization Complete

Cloud-init stages:

• Generator: Generates additional services if needed
• Local: Early networking setup, mount filesystems
• Network: Full networking available, run network-dependent tasks
• Config: General configuration
• Final: Last stage, run final customization

Cloud-Config Syntax

Cloud-config uses YAML format starting with #cloud-config.

Basic structure:

#cloud-config

# Comments start with #
package_update: true
package_upgrade: true

packages:
  - nginx
  - curl
  - wget

runcmd:
  - systemctl start nginx
  - systemctl enable nginx

write_files:
  - path: /etc/nginx/sites-available/default
    content: |
      server {
        listen 80 default_server;
        server_name _;
        location / {
          return 200 "Hello from $(hostname)";
        }
      }

Data type conversions:

#cloud-config

# String
region: us-east-1

# Number
port: 8080

# Boolean
enabled: true
disabled: false

# List
packages:
  - nginx
  - curl

# Dictionary/Map
environment:
  APP_ENV: production
  LOG_LEVEL: debug

# Multiline string
config: |
  This is a multiline
  configuration string
  that preserves formatting

Package Installation

Manage packages easily with cloud-init.

Basic package management:

#cloud-config

# Update package index
package_update: true

# Upgrade existing packages
package_upgrade: true

# Upgrade distribution packages
distro_upgrade: true

# Install packages
packages:
  - nginx
  - curl
  - wget
  - htop
  - git
  - python3-pip

# Package reboot behavior
package_reboot_if_required: true

# Configure package manager
apt:
  conf: |
    APT::Acquire::http::Timeout "60";
    APT::Acquire::ftp::Timeout "60";

PPA and repository configuration:

#cloud-config

# Add custom repository
apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

# Or add PPA
apt_sources:
  - source: ppa:nginx/stable
  - source: ppa:deadsnakes/ppa

# After adding repos, install packages
packages:
  - docker-ce
  - docker-ce-cli
  - containerd.io
  - python3.9

Package configuration:

#cloud-config

packages:
  - nginx
  - mysql-server

# Configure packages
package_update: true
package_upgrade: true

# Prevent interactive prompts
debconf_selections: |
  mysql-server mysql-server/root_password password root
  mysql-server mysql-server/root_password_again password root
  mysql-server mysql-server/remove_databases_and_users boolean false
  mysql-server mysql-server/skip-install boolean false

User and Group Management

Create and configure user accounts.

Create users:

#cloud-config

users:
  - name: ubuntu
    groups: sudo
    shell: /bin/bash
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1... [email protected]

  - name: app
    groups: appgroup
    shell: /usr/sbin/nologin
    home: /opt/app
    inactive: true

  - name: devops
    groups: [sudo, docker, wheel]
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1... [email protected]
      - ssh-rsa AAAAB3NzaC1... [email protected]

# Create groups
groups:
  - docker
  - appgroup: [app, ubuntu]

# Ensure default user disabled
disable_root: true

User configuration options:

#cloud-config

users:
  - name: appuser
    # Set user ID
    uid: 1001
    
    # Set group ID
    gid: 1001
    
    # Home directory
    home: /opt/appuser
    
    # Shell
    shell: /bin/bash
    
    # Group membership
    groups: [sudo, adm]
    
    # Add to sudoers
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    
    # Don't create home directory
    no_create_home: false
    
    # Create home directory with specific permissions
    homedir: /home/appuser
    
    # SSH keys for login
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1... [email protected]
      - ssh-rsa AAAAB3NzaC1... [email protected]
    
    # Disable password login
    ssh_import_id: github:username
    
    # Lock account
    # lock_passwd: true
    
    # Set expiration
    expire: false
    
    # Create with password (usually avoided for security)
    # passwd: $1$salt$hashed_password

Group management:

#cloud-config

groups:
  - docker
  - postgres
  - www-data: [ubuntu]
  - developers: [app, deploy, ubuntu]

users:
  - name: deploy
    groups: [developers, sudo]
    
  - name: app
    groups: developers

Running Commands

Execute scripts and commands during initialization.

Basic command execution:

#cloud-config

# Run commands at final stage
runcmd:
  - /opt/bin/setup.sh
  - systemctl start nginx
  - systemctl enable nginx
  - echo "Server configured" | mail -s "Server Ready" [email protected]

# Run commands conditionally by list index
# First element runs before second, etc.
runcmd:
  - ['sh', '-c', 'echo "$HOSTNAME is up" > /tmp/status.txt']
  - systemctl restart nginx

Command output redirection:

#cloud-config

runcmd:
  # Redirect to file
  - 'echo "Configuration log" > /var/log/init-log.txt'
  
  # Append to file
  - 'echo "Additional info" >> /var/log/init-log.txt'
  
  # Suppress output
  - 'systemctl start nginx > /dev/null 2>&1'
  
  # Log with timestamp
  - 'echo "$(date): Server configured" >> /var/log/cloud-init-custom.log'

Bootcmd for early execution:

#cloud-config

# Run commands at boot (before packages installed)
bootcmd:
  - 'modprobe zfs'
  - 'mkfs -t ext4 /dev/sdb1'
  - 'mkdir -p /mnt/data'
  - 'mount /dev/sdb1 /mnt/data'

# Then install packages
packages:
  - nginx
  - curl

# Then run runcmd
runcmd:
  - systemctl start nginx

Custom scripts:

#cloud-config

packages:
  - python3

# Write custom script
write_files:
  - path: /opt/setup.py
    permissions: '0755'
    owner: root:root
    content: |
      #!/usr/bin/env python3
      import subprocess
      import logging
      
      logging.basicConfig(level=logging.INFO)
      logger = logging.getLogger(__name__)
      
      logger.info("Running custom setup")
      subprocess.run(['apt-get', 'install', '-y', 'nginx'])
      subprocess.run(['systemctl', 'start', 'nginx'])

runcmd:
  - /opt/setup.py

File Management

Create and manage files during initialization.

Write files:

#cloud-config

write_files:
  - path: /etc/app/config.yml
    owner: root:root
    permissions: '0644'
    content: |
      app:
        name: myapp
        port: 8080
        environment: production
        logging:
          level: info
          file: /var/log/app.log

  - path: /opt/app/run.sh
    owner: app:app
    permissions: '0755'
    content: |
      #!/bin/bash
      cd /opt/app
      ./myapp --config /etc/app/config.yml

  - path: /etc/systemd/system/app.service
    owner: root:root
    permissions: '0644'
    content: |
      [Unit]
      Description=My Application
      After=network.target
      
      [Service]
      Type=simple
      User=app
      WorkingDirectory=/opt/app
      ExecStart=/opt/app/run.sh
      Restart=on-failure
      RestartSec=10
      
      [Install]
      WantedBy=multi-user.target

  - path: /etc/environment
    append: true
    content: |
      APP_ENV=production
      LOG_LEVEL=info

File encoding and compression:

#cloud-config

write_files:
  - path: /tmp/data.txt
    encoding: base64
    content: |
      VGhpcyBpcyBhIGJhc2U2NCBlbmNvZGVkIGZpbGUK

  - path: /opt/archive.tar.gz
    encoding: gz+base64
    content: H4sIAIWLpGUC/wvJyCzRUMgsS8wpTVWoVkgsS8wpURIqS8wpTlWqAOJcJaX5JUWlJRnpxUX5JUWlJRmpxUWlJRmlmSWpyZkpGalFqWWpqUVQUqsUQGpqamlyUWlpRmlyUmlySmpRVGpJZWlJZllKUWlpZkpSZWlJUXlKZklqMxM9RWZMoRhiEqUSR+LKnFmZlZoKkSzhTLBqJTK0UBMsI0vMBb6yVnFJEkRhWhFgBTr3nMH0QzUSMqGCYHbPTi5JLEktLkksTs0rBsi55hUlJhWlKuSmFqSWlqSWlqQWFpUkKoEUlpQUl6ZmluQVpyZnlmSklmTkFpalJqakFhWnFpQWFqYWJBUVF6VkJUpgSmVOQWpJZUpiRWlyQWpEUsVKKJCOJCniBLSYC1EFWQ1sOQEhyAKG0o9KaLIFaMsRgO6y2oBl8yLmEzaLBqLqWvt9M27L+zJdmRpQMREONuA=

Copy files from source:

#cloud-config

# Note: write_files copies content, not external files
# For copying from source during build, use provisioners in Packer
# or download during execution

write_files:
  - path: /opt/setup.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      # Download configuration from S3
      aws s3 cp s3://config-bucket/app-config.yml /etc/app/config.yml
      
      # Clone application from GitHub
      git clone https://github.com/myorg/myapp /opt/app

Network Configuration

Configure networking during initialization.

Network configuration:

#cloud-config

# Configure hostname
hostname: webserver-01
fqdn: webserver-01.example.com
prefer_fqdn_over_hostname: true

# Update /etc/hosts
manage_etc_hosts: true

# NTP configuration
ntp:
  enabled: true
  ntp_client: systemd-timesyncd
  servers:
    - 0.ubuntu.pool.ntp.org
    - 1.ubuntu.pool.ntp.org

# DNS configuration
resolv_conf:
  nameservers:
    - 8.8.8.8
    - 8.8.4.4
  searchdomains:
    - example.com

Custom network setup:

#cloud-config

# Wait for network to be ready
wait_for_network: true
wait_for_network_timeout: 60

# Disable IPv6
bootcmd:
  - 'echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf'
  - 'sysctl -p'

# Configure static IP (if not using DHCP)
write_files:
  - path: /etc/netplan/99-custom-ip.yaml
    content: |
      network:
        version: 2
        ethernets:
          eth0:
            dhcp4: false
            addresses:
              - 192.168.1.100/24
            gateway4: 192.168.1.1
            nameservers:
              addresses: [8.8.8.8, 8.8.4.4]

runcmd:
  - netplan apply

Cloud-init Callbacks

Send notifications after configuration completion.

Phone home to notify completion:

#cloud-config

phone_home:
  url: http://example.com/callback?instance_id=$INSTANCE_ID&hostname=$HOSTNAME
  post: all
  tries: 10

# Or callback with webhook
phone_home:
  url: https://webhook.example.com/cloud-init
  post: all
  tries: 3
  retries: 5

Send notification to monitoring:

#cloud-config

packages:
  - curl

write_files:
  - path: /opt/notify.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      INSTANCE_ID=$(ec2-metadata --instance-id | cut -d' ' -f2)
      HOSTNAME=$(hostname)
      
      # Send to monitoring system
      curl -X POST https://monitoring.example.com/api/events \
        -H "Content-Type: application/json" \
        -d "{
          \"event_type\": \"instance_ready\",
          \"instance_id\": \"$INSTANCE_ID\",
          \"hostname\": \"$HOSTNAME\",
          \"timestamp\": \"$(date -Iseconds)\"
        }"

runcmd:
  - /opt/notify.sh

Debugging Cloud-init

Troubleshoot cloud-init execution.

View cloud-init logs:

# Main cloud-init log
cat /var/log/cloud-init-output.log

# Detailed log
cat /var/log/cloud-init.log

# System log
journalctl -u cloud-init -n 50

# Watch logs in real-time
tail -f /var/log/cloud-init.log

Check cloud-init status:

# Current status
cloud-init status

# Show all status
cloud-init status --long

# Check if running
cloud-init status | grep -q "running"

# Show available modules
cloud-init modules --list-sources

Enable debug logging:

#cloud-config

# Enable debug output
debug: true

# Set log level
output: {all: '>> /var/log/cloud-init-custom.log'}

Manual execution for testing:

# Clean up cloud-init state
sudo cloud-init clean --logs --seed

# Re-run cloud-init
sudo cloud-init init
sudo cloud-init modules --mode=config
sudo cloud-init modules --mode=final

# Or single run
sudo cloud-init -d single --name=runcmd

Validate cloud-config syntax:

# Check cloud-config validity
cloud-init schema --config-file user-data.txt --annotate

# Validate specific module
cloud-init schema --config-file user-data.txt --annotate | grep runcmd

Conclusion

Cloud-init provides a standard, simple way to automate server provisioning across cloud providers. By mastering cloud-config syntax, package management, user creation, script execution, and file provisioning, you can eliminate manual server setup and create reproducible infrastructure. Combined with infrastructure-as-code tools like Terraform, cloud-init forms a powerful foundation for automated, consistent infrastructure deployment at scale.