Elasticsearch Alerts: One day “hack” to create a “watcher” like alerting open-source tool in Node.js

The ELK stack (Elasticsearch, Logstash , Kibana) are great tool to collect and analyze data from various sources. Making use of this data requires often alerting mechanisms to notify users about critical business or operational issues discovered by Elasticsearch queries. Unfortunately the open-source ELK stack does not include any tool to schedule queries or notify users. Elastic, the company behind ELK offers commercial software extensions called X-Pack, including X-Pack Alerting (formally called “watcher”). The pricing is not listed on the web-page, however reading this reddit thread it was clear to me that pricing is over the top for many projects and open-source alternatives are rare (e.g. Elastalert by Yelp) — so why not build another Elasticsearch alerting tool and share it on github. I hope other people will contribute and help to make it even better e.g. by contributing more notification functions e.g. for e-mail or Twitter.

What is X-Pack Alerting?

1. Surprisingly for a commercial product “watcher” has no UI — just a large JSON configuration, stored in Elasticsearch
2. Watcher schedules Elasticsearch queries
3. Transforms results
4. Sends notification to E-Mail or Slack

Sounds easy: schedule queries, do a bit of ETL and format the output for Slack notifications.

As main contributor to Logagent open-source tool — a kind of Logstash made with Node.js (yeah, why use Logstash made in Ruby when you could save 1,5 GB main memory on each server and get the power of async I/O in Node.js …), it was clear to me that I could use Logagent framework to implement “watcher” like functionality with a few simple steps, and so I did:

1. Create an input plugin for Elasticsearch queries — similar to the command scheduling plugin, running a task with a configurable interval. Elasticsearch query language is a bit ugly with its long JSON queries, so I decided to use YAML format for better maintainability. The resulting plugin has only 85 lines of Node.js code!
2. Use existing Logagent filter and transform functions
3. Implement the Slack output plugin. 75 lines of Node.js code!

Done! Final step, create a Logagent configuration to use the new Logagent plugins for Slack Alerting based on any scheduled Elasticsearch query: 67 lines of YAML code!

Let me explain the configuration file, because this is the only thing Logagent users need to customize to create Elasticsearch alerts:

1. Define a search query for the alert, in this case looking for HTTP error codes from HTTP Logs stored in Elasticsearch, or in this specific case in Sematext Cloud (query & indexing API is compatible to Elasticsearch). We use 1 minute interval for scheduling and a date range query for the last minute. Elasticsearch supports Date math like “now-1m/m” in range queries. Here is my config for the new Logagent input plugin:

Input plugin configuration to schedule Elasticsearch queries

1. Write a transform function e.g. to rename field names or group results. We will use here Logagents SQL output filter to rename fields or aggregate Elasticsearch results with in-memory SQL. In this case we count errors per HTTP status code (Elasticsearch DSL aggregation queries are sometimes hard to write … and most people know SQL)

Output filter using SQL output filter plugin

1. Get a URL for Slack API integration here: https://api.slack.com/incoming-webhooks and use it for the Slack Webhook output plugin. The output section includes a JavaScript filter function to implement the alert trigger logic — hey full control to make the trigger decision in a well-known programming language and the option to make complex calculations! Sorry Logagent scripts are not “painless” — just JavaScript ;)

Slack output plugin configuration with trigger logic as filter function in JavaScript

Note the Slack text template it could include any field from the resulting query in the Slack message to create readable messages for the end users.

If you don’t have Logagent installed get Node.js and install it:

> npm install -g @sematext/logagent

Run Logagent with the new configuration:

> logagent — config watchlogs.yml

And wait for the Alert in Slack:

Please note Logagent comes with setup scripts to run as systemd, launchd (Mac OS X), upstart or Windows service or Docker container — so the alert engine could run permanently on one of your servers. Once you get started with it, you might quickly discover more options for the new Logagent plugins e.g. store the Alerts in Elasticsearch in parallel to Slack, re-index aggregated and transformed data or watch simply log files or unix pipes to create real time Slack notifications without having Elasticsearch involved at all.

If you have suggestions for improvements or like to contribute to Logagent meet us on Github.