---
title: "Spark & Kafka docker-compose"
url: "https://vanducng.dev/2020/09/20/Spark-Kafka-docker-compose/"
date: "2020-09-20"
categories: ["Data"]
tags: ["docker","spark","kafka"]
---

# Spark & Kafka docker-compose

![spark-kafka-docker.png](https://vanducng.dev/2020/09/20/Spark-Kafka-docker-compose/spark-kafka-docker.png)

To quickly launch spark and kafka for local development, docker would be the top choice as of its flexibily and isolated environment. This save lot of time for manually installing bunch of packages as well as conflicting issues.

## Prequesite
It requires your local computer installed the following:
* [Docker](https://docs.docker.com/engine/install/)
* [Docker Compose](https://docs.docker.com/compose/install/)
* Clone below repo and start docker-compose
  ```bash
  clone git@github.com:vanducng/spark-kafka-docker-compose.git
  cd spark-kafka-docker-compose
  docker-compose up
  ```

  [vanducng/spark-kafka-docker-compose](https://github.com/vanducng/spark-kafka-docker-compose)

## Docker compose
The docker-compose includes two main services Kafka and Spark simply described as below:

![spark-kafka-diagram.png](https://vanducng.dev/2020/09/20/Spark-Kafka-docker-compose/spark-kafka-diagram.png)

Here is the list of container after starting docker-compose
![docker-list.png](https://vanducng.dev/2020/09/20/Spark-Kafka-docker-compose/docker-list.png)

Access to spark master via http://localhost:8080
![spark-master.png](https://vanducng.dev/2020/09/20/Spark-Kafka-docker-compose/spark-master.png)