---
title: "Quick tour with Elasticsearch 6.x"
url: "https://vanducng.dev/2019/11/26/Quick-tour-with-Elasticsearch-6-x/"
date: "2019-11-26"
categories: ["Data"]
tags: ["tutorial","elasticsearch"]
---

# Quick tour with Elasticsearch 6.x

When do researching to choose a good data storage technique for log collection, searching and analytic; I found elasticsearch is a ideal choice because of following reasons:

* **Performance**: fast query with million records within miliseconds, it is thanks to indexing document technique with Lucene engine running under-the-hood.
* **Scalability**: elasticsearch can be expanded by simply configuring new nodes when resource increase needed.
* **Integration** : it is compatible with elastic stacks (beats: metric, file, heart, etc. ) and others (Fluentd, grafana, etc.) which support many purposes to monitor multiple system and services.

## Installation
In the meanwhile, the latest ElasticSearch is in version 7.x with numerous breaking change and not 100% nicely work with our current services, then ES 6.8.5 is selected. The installation is documented in detailed from [Official Website](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/install-elasticsearch.html). Since our server is running CentOS 7, the RPM [installation method](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/rpm.html) is used.  

```bash
yum install -y java-1.8.0-openjdk-devel

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.5.rpm
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.5.rpm.sha512
shasum -a 512 -c elasticsearch-6.8.5.rpm.sha512 
sudo rpm --install elasticsearch-6.8.5.rpm
```

Following the recommend steps to get the ES started or stopped.
```bash
sudo chkconfig --add elasticsearch
sudo -i service elasticsearch start
sudo -i service elasticsearch stop
```

## Configuration
### System scope
This is where you can configure location of elasticsearch or java installation, keep it as default. In version 7.x, saperation installation of Java is not required.
```bash
nano /etc/sysconfig/elasticsearch
```

### Application scope
#### elasticsearch.yml
Within elasticsearch.yml file, generic information for targeted cluster, node, data storage path, etc. can be customized and override the default settings. In this example, i did change the cluster, node name and elastic data location. 
```bash
nano etc/elasticsearch/elasticsearch.yml
```

<p align="center">
		<image src="/media/images/elasticsearch-yml.png" width="70%" />
</p>

#### jvm.options
This is where the JVM resource configuration dedicated for your elasticsearch application. Initial and maximum heap are what should be noticed.  
  
Best practice is to allocate haft of your available system memory to achieve the optimal performance with Elasticsearch. Our CentOS has 3GB RAM free when running along with other applications, then the heap size is recommended with 1.5GB. However, for experiment purpose with few small logs, I reserved only 1GB as below. Check your RAM available with command `free -m`

``` bash
nano /etc/elasticsearch/jvm.options
```

<p align="center">
		<image src="/media/images/jvm-options.png" width="100%" />
</p>

#### Change elastic data location
In some cases, there is a need of store elastic data in separate drive for better management, below steps will help. For instance, I want to store data in `/elasticdata/`.

``` bash
mkdir /elasticdata/
```

Check user/group running elasticsearch application:
```bash
cat /etc/passwd
```

<p align="left">
	<image src="/media/images/elastic-user.png" width="70%" />
</p>

Change owner of elasticdata location to elasticsearch application user
```bash
chown elasticsearch.elasticsearch /elasticdata/
```

Restart elasticsearch service, you can check the RAM status and running services after that.
```bash line-numbers
service elasticsearch restart
free -m
ps -eaf|grep elastic
```

#### Log exploration
The log will be generated in `/var/log/elasticsearch/`
```bash
ls -latr /var/log/elasticsearch/
```

<p align="center">
		<image src="/media/images/elastic-log-1.png" width="100%" />
</p>

Explore the log:
```bash
cat tail /var/log/elasticsearch/elastic01.log
```

<p align="center">
		<image src="/media/images/elastic-log-2.png" width="100%" />
</p>

## Quick tour with Elasticsearch
### Physical data
You can check the data in our node which is currently node 0. It is mostly nothing as there are no indices avaliable.
```bash
ls /elasticdata/nodes/0/
```

<p align="left">
		<image src="/media/images/elastic-data-1.png" width="70%" data-/>
</p>

### Curl command
Now you can verify if the ElasticSearch is working by simple queries as below to check the cluster health and statistic:
```bash
curl -XGET 'localhost:9200/_cluster/health?pretty'
```

<p align="left">
		<image src="/media/images/elastic-query-1.png" width="70%" data-/>
</p>

```bash
curl -XGET 'localhost:9200/_cluster/stats?human&pretty&pretty'
```

<p align="left">
		<image src="/media/images/elastic-query-2.png" width="70%" data-/>
</p>

### Python ingestion with requests library
At first, you can dump some faking data with simple python2 script as below.
```python
from datetime import timedelta, date, datetime
import random
import string
import requests
import json

f = requests.Session()
headers = {'content-type':'application/json'}
start_date = datetime(2019, 11, 25)
for x in range(60*48):
print x, '', '',
single_date = start_date + timedelta(minutes=x)
dt = single_date.strftime("%Y-%m-%d")
url = "http://localhost:9200/{}-fake/fake".format(dt)
doc = { "user" : "fauie.com",
		"post_date" : single_date.isoformat(),
		"message" : ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(12)) ,
		"x": x,
		"cpu": random.randint(0,100),
		"io": random.randint(0,100)
}
resp = f.post(url, data=json.dumps(doc),headers=headers,verify=False)
print resp.text
```

### Python API
Once get familiar, take the advantage of available python api client (see below link) to make your job done in a timely manner.  
[https://elasticsearch-py.readthedocs.io/en/master/](https://elasticsearch-py.readthedocs.io/en/master/)

<a name="References"/>
## References
* [Mastering ElasticSearch 6.x and the Elastic Stack](https://www.packtpub.com/web-development/mastering-elasticsearch-6x-and-elastic-stack-video)