Today we are going to have a closer look at Elastic, a company founded in 2012 and located in Amsterdam. They are best known for their ELK stack or Elastic stack (used by eBay, Wikipedia, Uber, Netflix….). ELK stands for the open source trio Elasticsearch, Logistash and Kibana. 

Elasticsearch is a document-oriented database. Using the API, users can send raw data (log files, document files, …) to elasticsearch that will convert and store the data in JSON objects. These document-oriented databases are, compared to relational databases, much faster when it comes to searching large amounts of data.

Logstash is a processing pipeline that fetches data from different sources simultaneously, converts the data and then sends it again to other sources. So logstash will (1) fetch data from data files, log files, github, from databases, pipes, ….,  (2) convert the data and (3) sends it (simultaneously) to sources like elasticsearch, files, csv, mongo, rabbitmq, http, ….

And then we also have Kibana, a visualization plugin for elasticsearch. The tool allows the use of graphs and reports to investigate and monitor the elasticsearch data. It already contains a lot of built-in filters, geospatial capabilities, , … but users can also build their own customized dashboards

As you can imagine, these three tools together are a winning combination in all kind of companies and for all kind of purposes. Logstash to collect, synchronize and convert data from your system to other sources, elasticsearch to store the data and as a superfast text search / analytics engine and Kibana to visualize and monitor the data.

Today we no longer speak about the ELK stack. The reason for this is the introduction of Beats, light weight single-purpose applications used to ship data to the ELK stack. There are several beats that you can download from the elastic website like: filebeat (to read log files), metricsbeat (to track CPU usage, memory, disk IO, …), heartbeat (to monitor the uptime and response time of your systems),… They are a great addition to the elk stack, but of course, where does that leave the ELK acronym? Would it become BELK, BLEK, ELKB …. no sir, they just called it the ‘Elastic stack’.

Now with elasticsearch, logstash, kibana and beats we covered the base of the elastic stack. But there are so many other solutions available. Elastic has software for security, alerting and monitoring (X-pack), graph analysis, machine learning, performance monitoring (APM), …. On-premise or in the cloud.

To conclude, the biggest advantage of the elastic stack is that it is so multipurpose. Do you want to; monitor your system, sync two or more systems, have an overview of the data in your SQL db or a super-fast search database, have a dashboard to follow up on the status of your different projects,… everything is possible and almost all is open source. If you are thinking about using one of these products, let us know, we are more than happy to assist you.