Quick links:
- Intro to Watcher alerting
- Intro to ElastAlert
- Reasons to convert from one to the other
- Examples
- Conclusion
Introduction
In this guide, we will explain what Watcher and ElastAlert are and give the pros and cons of each. We will then show what it looks like to convert a watch to an ElastAlert. This guide gives a very basic example to illustrate that process; follow-up pieces will explore more complicated transitions.
Intro to Watcher alerting
In its simplest form, Watcher is able to search Elasticsearch indices to check for certain conditions and alert us through a variety of mediums if those conditions have been met. Let’s look at the basic anatomy of a watch before turning to the pros and cons of Watcher.
The basic anatomy of a watch
- Trigger: when the watch should run
- Input: which data to look at, including, but not limited to, Webhooks and Elasticsearch indices
- Condition: should the watch send an alert
- Transform: changes the data to a more readable/understandable format
- Actions: what should Watcher do with the alert (e.g., it can be written to the logs or sent to a number of systems, including, but not limited to, Webhooks, email, and Slack).
The complexity of a watch varies from checking if a field is above a certain value to being as complicated as you can imagine. However, Watcher is usually used for checking static values, meaning that users will find predictive models very difficult to implement.
Pros
- Built into Elasticsearch
- UI component in Kibana
- You will most likely design in Kibana, then automate through the APIs
Cons
- If you are using just this Platinum/Enterprise feature, then the license can become restrictively expensive when sending external alerts for large clusters
- For more complicated conditions and transforms, you will need knowledge of the Painless scripting language
Intro to ElastAlert
ElastAlert is one of the many alternatives to Watcher. Just like Watcher, in its simplest form, it can search Elasticsearch indices to check for certain conditions and alert us through a variety of mediums if those conditions have been met. Now, let’s look at the basic anatomy of an ElastAlert before turning to its pros and cons.
The basic anatomy of an ElastAlert
- config.yml: some of the things in the config are the Elasticsearch URL, the rules folder, how often to run the rules, and the writeback_index, which has the status of ElastAlert
- Rules folder: contains one or more rules to run
- Rules: set out what to check for and, if the condition is met, where to send the event
- Optional files: such as auth to send data.
There are a number of prebuilt rule types (`any`, `change`, `frequency`, `flatline`, etc.) that limit the flexibility of ElastAlert, but you do have the ability to create custom rules.
Pros
- Free
- Default rules are shipped with ElastAlert and these are simple to configure
Cons
- Knowledge of Python is needed for the more complicated custom rules
- ElastAlert must be run separately since it is not built into Elasticsearch or Kibana.
Reasons you may want to convert a Watch to an ElastAlert
There are several reasons why you might want to convert from Watcher to ElastAlert. The four main ones are listed below:
- Cost reduction because ElastAlert is open source and free
- Painless is a restricted set of Java APIs (with an annoying learning curve), meaning you cannot do anything you want with it
- With Python you can do anything you want in ElastAlert, with no limitations; the rules can become very complicated, and you have access to any packages and methods you may want to use to make your life easier
- Finally, you might want to convert to ElastAlert as a preparation step before migrating to OpenSearch.
Example
There is a lot of documentation about setting up watches and ElastAlerts. So, here, we will focus on a specific example of converting a watch to an ElastAlert.
Sending threshold alerts to Gmail
Watcher
PUT _watcher/watch/logs-threshold
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"bool": {
"filter": {
"range": {
"@timestamp": {
"gte": "{{ctx.trigger.scheduled_time}}||-5m",
"lte": "{{ctx.trigger.scheduled_time}}",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
}
}
}
},
"condition": {
"script": {
"source": "if (ctx.payload.hits.total > params.threshold) { return true; } return false;",
"lang": "painless",
"params": {
"threshold": 1000
}
}
},
"actions": {
"email_1": {
"email": {
"profile": "standard",
"to": [
"person@example.com"
],
"subject": "Watch logs-threshold has exceeded the threshold",
"body": {
"text": "logs index has received more than 1000 documents over the last 5 minutes"
}
}
}
}
}To send the email, you will also need to set up an email account.
Add the following to your elasticsearch.yml file on every node:
xpack.notification.email.account:
gmail_account:
profile: gmail
smtp:
auth: true
starttls.enable: true
host: smtp.gmail.com
port: 587
user: person@example.comAnd your password will be stored in the Elasticsearch keystore on every node.
bin/elasticsearch-keystore add xpack.notification.email.account.gmail_account.smtp.secure_password
The instructions are slightly different for other email providers. For Gmail, you will need to generate an app password to store in your keystore. Instructions on how to do that can be found here. These same instructions are needed for ElastAlert.
ElastAlert
Feel free to install this via pip and run it locally. However, we recommend using a docker image because all dependencies are already installed there and there are usually fewer issues.
The directory structure is as follows:
elastalert/
├── rules/
│ └── frequency.yml
├── config.yml
├── docker-compose.yml
└── google-auth.yml
`frequency.yml`
name: "logs-threshold"
type: "frequency"
index: "logs"
is_enabled: true
num_events: 1000
timeframe:
minutes: 5
timestamp_field: "@timestamp"
alert_subject: "Watch logs-threshold has exceeded the threshold"
alert_text: "logs index has received more than 1000 documents over the last 5 minutes"
filter:
- query:
match_all: {}
alert:
- "email"
email:
- "person@example.com"
smtp_host: "smtp.gmail.com"
smtp_port: 587
smtp_auth_file: "/opt/elastalert/google-auth.yml"rules_folder: /opt/elastalert/rules run_every: minutes: 1 buffer_time: minutes: 15 es_host: host.docker.internal es_port: 9200 writeback_index: elastalert_status
`docker-compose.yml` can easily be converted to a run command:
version: '3.3'
services:
elastalert2:
container_name: elastalert
restart: always
volumes:
- '$PWD/config.yaml:/opt/elastalert/config.yaml'
- '$PWD/rules:/opt/elastalert/rules'
- '$PWD/google-auth.yml:/opt/elastalert/google-auth.yml'
image: jertel/elastalert2`google-auth.yml`:
user: "person@example.com" password: "****************"
Conclusion
It is fairly easy to convert to ElastAlert, and with some basic knowledge of Python, you can implement a lot of the simple use cases. The basic use cases are limited though, so any complicated watches will take time to convert and could be difficult. But if Watcher is the only feature you are using, as is the case with many users, then there really should be no reason why you should not switch over to ElastAlert to reduce your licensing costs.
