In this blog we will explain how to configure alerts in Grafana for your oVirt environment and provide an example alerts dashboard that you can import, use and edit to your needs.
The alerts can be configured to send you notifications by different channels as described in Grafana List of supported notifiers.
This blog focuses on how to configure Grafana to send alerts by email.

  1. Configure Grafana SMTP settings:

    Note: the SMTP setting in Grafana is very similar to the SMTP settings in oVirt, and the same SMTP server can be used in both.
    oVirt SMTP documentation: Configure SMTP for email notifications.
    • Log in to your engine virtual machine.
    • Edit: 
      /etc/grafana/grafana.ini

      And change below options:

      [smtp]
enabled = true
host = smtp.gmail.com:587
user = ********@gmail.com
password = ********
    • Restart the Grafana service:

      # systemctl restart grafana-server
  2. Enable your Gmail account to receive connections from external programs:

    • These steps are suggested only for demo purposes. More secure configuration, such as Google 2-Step Verification should be used in production environments.
    • Open your browser and access your Gmail account.
    • After the login access Less secure app access settings, and select the option to enable the use of less secure applications:

    • Access Allow access to your google account settings, and select the option to unblock your account:

  3. Add a notification channel to Grafana:

    • Log in to Grafana.
    • Access the Alerting menu and select Notification channels:

    • Click Add channel:

    • Set the name of the channel, choose the channel notification method (Email in this case), enter addresses of the recipients, and select your notification settings.
      Click Test and wait for confirmation “Test notification sent”:

    • Click Save.
  4. Import the example alerts dashboard (optional):

    • Note: The example alerts dashboard is compatible with oVirt 4.4.7 and above.
    • Copy the content of the dashboard json file from: alerts-dashboard.json.
    • Access the Create menu and select Import:
       
    • Paste the copied json in the Import via panel json box and click Load:

    • Set the dashboard parameters (name, folder, uuid, data source) and click Import.

  1. Create your own dashboards and Alert Rules:

    • Template variables are not supported in alert queries.
    • Only time based queries are supported in alert queries.
    • Click Edit in the panel where you want to configure the Alert:

    • Enter the Alert option and click Create Alert:

    • Set the Rule name and frequency for evaluation of your threshold:

      Note: If an alert rule has a configured For and the query violates the configured threshold it will first go from OK to Pending. Going from OK to Pending Grafana will not send any notifications. Once the alert rule has been firing for more than the For duration, it will change to Alerting and send alert notifications.
    • Set the rule Conditions based on the number of queries in your graph panel:

    • No Data and Error Handling settings:

    • Choose the Notification channel, and set the notification message:

      Note: channels can be set to appear automatically when an alert role is created by entering the channel settings (mentioned in step 3), and marking the channel as default. 
    • Test the alert by clicking: Test rule.
    • Save the dashboard when done setting the alert.