--- title: "How to Install Apache Spark on Ubuntu 24.04 (Standalone)" description: "Install Apache Spark 4 on Ubuntu 24.04, run the standalone master and worker as systemd services, submit PySpark jobs and review them in the History Server." category: "Data Engineering" language: en source: https://cubepath.com/docs/data-engineering/apache-spark-installation-on-linux updated: 2026-09-25 --- # How to Install Apache Spark on Ubuntu 24.04 (Standalone) Apache Spark is an open source engine for large-scale data processing. It splits work into tasks that run in parallel across the CPU cores of one or many machines, and offers APIs for Python (PySpark), SQL, Scala and Java. In this tutorial you will install Spark 4 on Ubuntu 24.04, run its built-in standalone cluster manager (a master and a worker) as systemd services, submit PySpark jobs, and keep a record of finished jobs with the History Server. The last section shows how to add more worker nodes. ## Prerequisites To follow this guide you need: - A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with at least 4 GB of RAM and 2 vCPUs. Spark keeps data in memory, so more RAM translates directly into larger jobs. - A non-root user with `sudo` privileges. - UFW enabled with only SSH allowed. The Spark web UIs have no authentication, so this guide keeps them closed and reaches them through an SSH tunnel. ## Step 1 - Installing Java Spark 4 runs on Java 17 or 21. Install the OpenJDK 17 runtime from the Ubuntu repositories, together with Python 3 for PySpark: ```bash sudo apt update sudo apt install openjdk-17-jre-headless python3 ``` Verify the Java version: ```bash java -version ``` ```output openjdk version "17.0.16" 2025-07-15 OpenJDK Runtime Environment (build 17.0.16+8-Ubuntu-0ubuntu124.04.1) OpenJDK 64-Bit Server VM (build 17.0.16+8-Ubuntu-0ubuntu124.04.1, mixed mode, sharing) ``` The exact build number will differ. ## Step 2 - Downloading and installing Spark Spark is distributed as a prebuilt archive. Check the latest release on the Apache Spark downloads page and set it in a variable; this guide uses 4.0.1 with the package prebuilt for Hadoop 3: ```bash SPARK_VERSION=4.0.1 cd /tmp curl -fLO "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz" curl -fLO "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz.sha512" ``` Compare the checksum of the archive with the published one. Both hashes must be identical: ```bash sha512sum "spark-${SPARK_VERSION}-bin-hadoop3.tgz" cat "spark-${SPARK_VERSION}-bin-hadoop3.tgz.sha512" ``` Extract the archive into `/opt` and create a version-independent symlink, which makes future upgrades a matter of pointing the link at a new directory: ```bash sudo tar -xzf "spark-${SPARK_VERSION}-bin-hadoop3.tgz" -C /opt sudo ln -sfn "/opt/spark-${SPARK_VERSION}-bin-hadoop3" /opt/spark ``` Create a system user to run the Spark daemons, and a directory for the worker's scratch space and the event logs. The event log directory is group writable so that jobs submitted by members of the `spark` group can write to it: ```bash sudo useradd --system --home-dir /var/lib/spark --shell /usr/sbin/nologin spark sudo mkdir -p /var/lib/spark/work /var/lib/spark/events sudo chown -R spark:spark /var/lib/spark sudo chmod 2775 /var/lib/spark/events sudo usermod -aG spark "$USER" ``` Log out and back in so the new group membership applies to your session. Add Spark to the `PATH` for all users: ```bash sudo nano /etc/profile.d/spark.sh ``` ```bash export SPARK_HOME=/opt/spark export PATH="$PATH:$SPARK_HOME/bin" ``` Load it in the current shell and check the installation: ```bash source /etc/profile.d/spark.sh spark-submit --version ``` ```output Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.1 /_/ ``` ## Step 3 - Configuring the standalone cluster The standalone cluster has two roles. The **master** accepts applications and assigns resources; each **worker** offers CPU cores and memory and launches executors that run the tasks. On a single server both run side by side. Create `spark-env.sh`, which the daemon scripts read at startup: ```bash sudo nano /opt/spark/conf/spark-env.sh ``` ```bash JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 SPARK_MASTER_HOST=127.0.0.1 SPARK_WORKER_CORES=2 SPARK_WORKER_MEMORY=3g SPARK_WORKER_DIR=/var/lib/spark/work ``` - `SPARK_MASTER_HOST` is the address the master listens on. `127.0.0.1` is enough for a single server; the last section changes it for multi-node setups. - `SPARK_WORKER_CORES` and `SPARK_WORKER_MEMORY` are what the worker offers to applications. Leave about 1 GB of RAM for the operating system and the daemons themselves. - On an arm64 server the Java path is `/usr/lib/jvm/java-17-openjdk-arm64`. Next, create `spark-defaults.conf`, which sets defaults for every application you submit: ```bash sudo nano /opt/spark/conf/spark-defaults.conf ``` ```ini spark.master spark://127.0.0.1:7077 spark.eventLog.enabled true spark.eventLog.dir file:///var/lib/spark/events spark.history.fs.logDirectory file:///var/lib/spark/events spark.executor.memory 2g spark.driver.memory 1g ``` With `spark.master` set here, you no longer need `--master` on every `spark-submit` call. The event log settings make each application write a record that the History Server reads in Step 6. ## Step 4 - Running the master and worker with systemd The scripts in `/opt/spark/sbin` start the daemons in the background by default. Setting `SPARK_NO_DAEMONIZE=true` keeps them in the foreground, which is what systemd expects, and sends their logs to the journal. Create the master unit: ```bash sudo nano /etc/systemd/system/spark-master.service ``` ```ini [Unit] Description=Apache Spark standalone master After=network-online.target Wants=network-online.target [Service] User=spark Group=spark Environment=SPARK_NO_DAEMONIZE=true ExecStart=/opt/spark/sbin/start-master.sh Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ``` Create the worker unit. It connects to the master URL passed as an argument: ```bash sudo nano /etc/systemd/system/spark-worker.service ``` ```ini [Unit] Description=Apache Spark standalone worker After=spark-master.service Wants=spark-master.service [Service] User=spark Group=spark Environment=SPARK_NO_DAEMONIZE=true ExecStart=/opt/spark/sbin/start-worker.sh spark://127.0.0.1:7077 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ``` Reload systemd and start both services: ```bash sudo systemctl daemon-reload sudo systemctl enable --now spark-master spark-worker ``` Check that the worker registered with the master: ```bash sudo journalctl -u spark-master --no-pager | grep -i "registering worker" ``` ```output INFO Master: Registering worker 127.0.0.1:41235 with 2 cores, 3.0 GiB RAM ``` The master also exposes its state as JSON on port 8080, which you can query locally: ```bash curl -s http://127.0.0.1:8080/json/ | python3 -m json.tool | grep -E '"(status|aliveworkers|cores|memory)"' ``` ```output "aliveworkers": 1, "cores": 2, "memory": 3072, "status": "ALIVE" ``` ## Step 5 - Submitting PySpark jobs Start with the Pi estimation example that ships with Spark. The argument `100` is the number of partitions, which means 100 parallel tasks: ```bash spark-submit /opt/spark/examples/src/main/python/pi.py 100 2>/dev/null ``` ```output Pi is roughly 3.141640 ``` Redirecting standard error hides Spark's own log lines; remove `2>/dev/null` when you need to debug a job. Now write a job of your own that reads a CSV file and aggregates it with the DataFrame API and with SQL. The executors run as the `spark` user and read input files themselves, so the data must live in a directory that user can access. Home directories on Ubuntu 24.04 are private (mode 750), so create a shared directory owned by you and the `spark` group: ```bash sudo install -d -o "$USER" -g spark -m 2775 /srv/spark-demo ``` Create a sample dataset: ```bash cat > /srv/spark-demo/sales.csv <<'EOF' product,region,amount vps,eu,20 vps,us,25 baremetal,eu,180 vps,eu,20 baremetal,us,200 kubernetes,eu,60 EOF ``` Create the job: ```bash nano /srv/spark-demo/sales_report.py ``` ```python from pyspark.sql import SparkSession from pyspark.sql import functions as F spark = SparkSession.builder.appName("SalesReport").getOrCreate() spark.sparkContext.setLogLevel("WARN") df = ( spark.read.option("header", True) .option("inferSchema", True) .csv("file:///srv/spark-demo/sales.csv") ) # DataFrame API by_product = ( df.groupBy("product") .agg(F.count("*").alias("orders"), F.sum("amount").alias("revenue")) .orderBy(F.desc("revenue")) ) by_product.show() # The same data through SQL df.createOrReplaceTempView("sales") spark.sql( "SELECT region, SUM(amount) AS revenue FROM sales GROUP BY region ORDER BY region" ).show() spark.stop() ``` Submit it: ```bash spark-submit /srv/spark-demo/sales_report.py 2>/dev/null ``` ```output +----------+------+-------+ | product|orders|revenue| +----------+------+-------+ | baremetal| 2| 380| | vps| 3| 65| |kubernetes| 1| 60| +----------+------+-------+ +------+-------+ |region|revenue| +------+-------+ | eu| 280| | us| 225| +------+-------+ ``` A `file://` path works here because the driver and the only worker share one disk. On a multi-node cluster every executor needs to reach the same data, so jobs read from and write to shared storage (HDFS, S3-compatible object storage or NFS), for example with `df.write.parquet("s3a://bucket/path")`. For interactive exploration, `pyspark` opens a Python shell with a ready `spark` session connected to the cluster: ```bash pyspark ``` ## Step 6 - Running the History Server The application UI on port 4040 only exists while a job is running. The History Server rebuilds the UI of finished applications from the event logs configured in Step 3. Create a unit for it: ```bash sudo nano /etc/systemd/system/spark-history.service ``` ```ini [Unit] Description=Apache Spark History Server After=network-online.target [Service] User=spark Group=spark Environment=SPARK_NO_DAEMONIZE=true ExecStart=/opt/spark/sbin/start-history-server.sh Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl daemon-reload sudo systemctl enable --now spark-history ``` Confirm that it found the jobs from Step 5: ```bash curl -s http://127.0.0.1:18080/api/v1/applications | python3 -m json.tool | grep '"name"' ``` ```output "name": "SalesReport", "name": "PythonPi", ``` To browse the web UIs from your workstation without opening any port, create an SSH tunnel: ```bash ssh -L 8080:127.0.0.1:8080 -L 18080:127.0.0.1:18080 your_user@your_server_ip ``` Then open `http://localhost:8080` for the master UI (workers, running and completed applications) and `http://localhost:18080` for the History Server (stages, tasks, executors and SQL plans of each finished job). ## Step 7 - Adding more worker nodes To grow the cluster, prepare each additional server with Steps 1 and 2 and connect them over a private network, which keeps Spark's unauthenticated ports off the internet. On the master, set `SPARK_MASTER_HOST` in `/opt/spark/conf/spark-env.sh` to the master's **`private_ip`**, update `spark.master` in `spark-defaults.conf` and the URL in `spark-worker.service` to `spark://private_ip:7077`, then restart: ```bash sudo systemctl daemon-reload sudo systemctl restart spark-master spark-worker ``` Spark nodes talk to each other on several ports, some of them random, so allow all traffic from the private subnet on every node. Replace `10.0.0.0/24` with your private range: ```bash sudo ufw allow from 10.0.0.0/24 ``` On each new worker, copy `spark-env.sh` without the `SPARK_MASTER_HOST` line (adjust cores and memory to that machine) and create `spark-worker.service` pointing to `spark://private_ip:7077`. After enabling it, the master's JSON endpoint shows the new total of `aliveworkers`. ## Troubleshooting **The worker does not register with the master.** Read `sudo journalctl -u spark-worker -n 50`. Connection refused means the master URL in the unit does not match `SPARK_MASTER_HOST`; the host in both must be identical, including `127.0.0.1` versus a hostname. **The job waits with "Initial job has not accepted any resources".** The application asks for more than a worker can offer. Make sure `spark.executor.memory` is lower than `SPARK_WORKER_MEMORY`, or pass a smaller value with `spark-submit --executor-memory 1g`. **Executors fail with `java.lang.OutOfMemoryError`.** Give executors more memory, or split the data into more partitions with `spark.sql.shuffle.partitions` so each task handles less data at a time. ## Conclusion You now have Apache Spark 4 running as a standalone cluster on Ubuntu 24.04, with systemd managing the master, worker and History Server, and PySpark jobs submitted with `spark-submit`. Next, connect Spark to your real data by adding the S3A or JDBC connectors with `--packages`, schedule recurring jobs with a workflow tool such as Apache Airflow, and enable Spark's authentication and encryption options (`spark.authenticate`, `spark.network.crypto.enabled`) before extending the cluster beyond a trusted private network.