Running Elasticsearch & Kibana using Docker

I’m going to show how to use Docker to quickly get started with a development environment for Elasticsearch & Kibana.

Why Docker?

Docker is a very useful tool to package software builds and distribute them onwards. It allows you to define a universal configuration file and run lightweight virtual machines, called containers.

First, install the version of Docker for your operating system.

Getting Started

After you have downloaded and installed Docker, you can run a container process from the command line, however docker-compose offers a better workflow; see its documentation for details.

The simplest docker-compose.yaml file looks as follows:

image — There are number of Docker images with Elasticsearch, but the one maintained by elastic.co are the best.

ports —For Elasticsearch, the setting will map port 9200 of your container to your host port 9200. For Kibana, the setting will map port 5601 of your container to your host port 5601.

environment — There are three environment variables. The first two are mandatory, while the third is optional.

• cluster.name — The name for our Elasticsearch cluster.
• bootstrap.memory_lock — We need to lock the process address space into RAM, preventing any Elasticsearch memory from being swapped out, read more here.
• ES_JAVA_OPTS — We set the minimum and maximum heap size so Elasticsearch has enough memory to run, read more here.

ulimits — This is a Linux command to set user limits on how many processes can be run or how memory can be used, see this article.

Run this command:

>> docker-compose up -d

You should be able to run docker ps and see the 2 containers:

You can use the elasticsearch Python package to create a test index and query it:

You can view the output on your console to confirm its working.

You can also go to localhost:5601 and play around with the Kibana dashboard:

Become an Elasticsearch Master

Elasticsearch is built on top of the Apache Lucene project, which is written in Java. As such, most of the new features are only accessible through that language.

• You can still get a decent amount of functionality with Python, use the official package documentation for more details.
• SQL interfaces are getting more and more common, Elastic just released Elasticsearch SQL, so you may want to try that out if you believe SQL will continue to unify everything.

If you are interested in learning data engineering, check out the course below.