Skip to main content

What is the Elastic Stack and how to post data to an ElasticSearch DB in an Amazon ES Service

Amazon ES Service is a fully managed system that makes it easy to deploy Elastic Stack to AWS servers in an integrated way. Some features like installing Kibana plugins are not yet available.

ElasticSearch is part of the Elastic Stack, a group of tools/services from the Elastic Company (elastic.co)

Elastic Stack:
* Kibana
* ElasticSearch
* Beats
* Logstash

ElasticSearch is a NoSQL document database and is most common used with Kibana, a UI tool to visualize data from the ES database. ElasticSearch is used for high speed text search.

It was previously known as the ELK Stack because of the tools/services from Elastic Company that are used togheter:
* ElasticSearch
* Logstash
* Kibana

In elasticsearch db you post data to an index in the same way you insert data into tables in a RDBMS (Relational Database Management Systems). We use index to separate and group different types of information (data) in the same way we use tables in a database.

SO INDEXES ARE FOR ES DB´S WHAT TABLES ARE FOR A RDBMS.

To POST data to a ES DB we need to construct our url in the following format:
url = host/index/type

We also need to set aws credentials with the aws service, region and our access keys and secret.

To save data in AWS ES Service you need to send a post request to your ES endpoint domain. One great thing of a NoSQL database is the ability to send JSON objects to the engine while making the properties of our object searchable by the database.

Example of a POST request made in node-fetch:
fetch('https://endpoint.region.es.amazonaws.com:443/nameof-your-index/doc-type', { method: 'POST', body: product_obj, headers: { 'Content-Type': 'application/json' } })
     .then(res => {
         return res.json();
     }).catch( err => {
         console.log(err);
     });
You can downlaod the complete source code, from the Github.

Labels

Labels:

Comments

Post a Comment

Popular posts from this blog

How to run OPA in Docker

From the introduction of the openpolicyagent.org site: OPA generates policy decisions by evaluating the query input against policies and data. In this post i am going to show you an easy and fast way to test your policies by running OPA in Docker. First, make sure you have already installed Docker and have it running: docker ps Inside your choosen directory, create two files. One called input.json file for your system representation and one file called example.rego for your rego policy rules. Add the following content to your json file: Add the following content for the example.rego: Each violation block represents the rule that you want to validate your system against. The first violation block checks if any of the system servers have the http protocol in it. If that is the case, the server id is added to the array. In the same way, the second violation block checks for the servers that have the telnet protocol in it and if it finds a match the server id is also...
Post a Comment
Read more

How to use Splunk SPL commands to write better queries - Part I

Introduction As a software engineer, we are quite used to deal with logs in our daily lives, but in addition to ensuring that the necessary logs are being sent by the application itself or through a service mesh, we often have to go a little further and interact with some log tool to extract more meaningful data. This post is inspired by a problem I had to solve for a client who uses Splunk as their main data analysis tool and this is the first in a series of articles where we will delve deeper and learn how to use different Splunk commands. Running Splunk with Docker To run Splunk with docker, just run the following command: docker run -d —rm -p 8000:8000 -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_PASSWORD=SOME_PASSWORD --name splunk splunk/splunk:latest Sample Data We are going to use the sample data provided by Splunk. You can find more information and download the zip file from their web site . How does it work? In order to be able to interact with Splunk t...
Post a Comment
Read more

Log Aggregation with ELK stack and Spring Boot

Introduction In order to be able to search our logs based on a key/value pattern, we need to prepare our application to log and send information in a structured way to our log aggregation tool. In this article I am going to show you how to send structured log to ElasticSearch using Logstash as a data pipeline tool and how to visualize and filter log information using Kibana. According to a definition from the Wikipedia website: Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. According to Elasticsearch platform website , Elasticsearch is the heart of the Elastic stack, which centrally stores your data for lightning fast search. The use of Elasticsearch, Kibana, Beats and Logstash as a search platform is commonly known as the ELK stack. Next we are going to start up Elasticsearch, Kibana and Logstash using docker so we can better underst...
Post a Comment
Read more