Vitrage Specs

Aodh Notifier

Sun, 03 Jan 2021 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/aodh-notifier

The Evaluator performs root cause analysis on the Vitrage Graph and may determine that an alarm should be created, deleted or otherwise updated. Other components are notified of such changes by the Vitrage Notifier service. Among others, Vitrage Notifier is responsible for handling Aodh Alarms.

This blueprint describes the implementation of Vitrage Notifier for notifying Aodh on Vitrage alarms.

+------------------+           +------------------+          +------------------+
|   Aodh           <--+        |                  |          |                  |
+------------------+  | Update |      Vitrage     |  Raise   |      Vitrage     |
                      +--------|                  <----------|                  |
+------------------+  | Alarm  |      Notifier    |  Alarm   |      Evaluator   |
| Other components <--+        |                  |          |                  |
+------------------+           +------------------+          +------------------+

Problem description

Vitrage should be capable of creating, deleting and otherwise updating alarms as requested by the Evaluator Engine. The notifier is responsible for ensuring these updates are executed. Specifically we will start here with Aodh alarms.

Main challenges:

There is no way to define a ‘custom alarm’ in Aodh
Vitrage alarms are based on resources. There is a need to pass the resource information to Aodh
Several alarms of the same type can be triggered at the same time, each for a different resource. For example, in case there is an alarm on a host, Vitrage will raise a deduced alarm on every instance in this host.
How can someone ask for notifications on updates of Vitrage alarms?

Proposed change

The Vitrage Notifier will be separate from the Evaluator, as the two will have different demands of scale and other performance considerations. The Vitrage Notifier will supply an API used by the Vitrage Evaluator, containing create/delete/update alarm.

In Aodh, Vitrage alarms will be defined as event alarms, this seems like the most appropriate option. The resource id will be defined in the alarm query.

Vitrage deduced alarms will look like this:

Property	Value
alarm_actions	[]
alarm_id	4a3cb988-a620-4bf3-87f7-077c751c408f
description	Instance is unreachable
enabled	True
event_type	vitrage.alarm.instance_unreachable
insufficient_data_actions	[]
name	vitrage_instance_unreachable_1
ok_actions	[]
project_id	5542b27142154f30b32dea6238aa81aa
query	[{field’: ‘resource_id’, ‘type’: ‘’, ‘value’: ‘b0bf3635-d9e8-4624-9793-7aac82948c0a’, ‘op’: ‘eq’}]
repeat_actions	False
severity	moderate
state	alarm
type	event
user_id	8ab65ef808b245e3ba234b7b3554cb94

In this example, Vitrage triggers a deduced alarm that an instance is unreachable due to a failure in the public switch (which was detected by Nagios). There will be several alarms with the same event_type and different instance ids in their query.

There are two options how to trigger Vitrage alarms in Aodh, none is perfect.

Alternative 1

Vitrage will create an event alarm in Aodh. Then, it will send a notification to the message bus. The notification will be converted to a Ceilometer event, which will trigger the Aodh alarm.

The exact notification and event format are still TBD.

The main problem with this solution is that the Aodh alarm will be created on-the-fly and triggered immediately, so it will be impossible for another project to register a web-hook on the alarm before it is triggered. It will be possbile to see Vitrage alarms in list-alarms, but not to be notified when they are first triggered.

Alternative 2

Vitrage will create an event alarm in Aodh, with ‘alarm’ state. The event itself will never be sent, so the alarm state will remain ‘alarm’.

The problem with this solution is that Aodh will not send a notification about the alarm being triggered. But since in Alternative 1 it is also impossible to register on the alarm, there is no real difference between the two options.

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

For Alternative 1 - there is a need to define the notification->event configuration

For Alternative 2 - None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: idan-hefetz

Work Items

None

Dependencies

None

Testing

This blueprint requires unit tests and Tempest tests.

Documentation Impact

For Alternative 1 - there is a need to document the notification->event configuration

For Alternative 2 - None

References

Vitrage wiki page: https://wiki.openstack.org/wiki/Vitrage

Vitrage use cases: https://github.com/openstack/vitrage/blob/master/doc/source/vitrage-use-cases.rst

Parametrized Template

Fri, 24 Apr 2020 00:00:00

StoryBoard link: https://storyboard.openstack.org/#!/story/2004056

This spec suggests a method to quickly create Vitrage templates based on a template that contains parameters.

Problem description

Vitrage template language is powerful, yet a bit verbose and it’s quite easy to make syntax mistakes when creating a new template from scratch. Moreover, in real production environments, we see that many templates have a very similar structure like “an alarm on a host is propagated to its instances”. We would like to make it easier to generate the most common templates.

Another motivation for this feature is for Heat usage. A Heat user should be able to easily define a Vitrage template for auto-healing or auto-scaling its stack based on specific alarms. For more information, see Heat_spec and Heat_template.

Proposed change

Parts of a template, like alarm names, resource states, etc. can be written as parameters. When creating a new template, the user will have to assign actual values to each parameter that doesn’t have a default value. The concept is similar to the way that parameters are used in Heat.

A Parameters section can be added to standard templates. Each parameter will hold two optional fields:

description
default

All values will be strings.

The parameters can be used inside the templates, using a get_param(param) syntax.

Vitrage template add API will include a new option to create a template based on a template with parameters. The template will be added to the database only if all parameters are set.

Example for a template with parameters

metadata:
  version: 2
  type: standard
  name: host_affects_instance_prototype
  description: Template prototype for scenarios where a Zabbix alarm on a host affects its instances
parameters:
  host_alarm_rawtext:
    description: Zabbix rawtext of the alarm on the host
  instance_alarm_name:
    description: Name of the alarm to be created on the instance
  instance_alarm_severity:
    description: Severity of the alarm to be created on the instance
    default: WARNING
  host_state:
    description: New state to be set for the host
    default: SUBOPTIMAL
  instance_state:
    description: New state to be set for the instance
    default: SUBOPTIMAL
definitions:
  entities:
   - entity:
      category: ALARM
      type: zabbix
      rawtext: get_param(host_alarm_rawtext)
      template_id: host_alarm
   - entity:
      category: ALARM
      type: vitrage
      name: get_param(instance_alarm_name)
      template_id: instance_alarm
   - entity:
      category: RESOURCE
      type: nova.instance
      template_id: instance
   - entity:
      category: RESOURCE
      type: nova.host
      template_id: host
  relationships:
   - relationship:
      source: host_alarm
      relationship_type: on
      target: host
      template_id : alarm_on_host
   - relationship:
      source: host
      relationship_type: contains
      target: instance
      template_id : host_contains_instance
   - relationship:
      source: instance_alarm
      relationship_type: on
      target: instance
      template_id : alarm_on_instance
scenarios:
  - scenario:
     condition: alarm_on_host and host_contains_instance
     actions:
      - action:
          action_type: raise_alarm
          action_target:
            target: instance
          properties:
            alarm_name: get_param(instance_alarm_name)
            severity: get_param(instance_alarm_severity)
      - action:
          action_type: set_state
          action_target:
            target: instance
          properties:
            state: get_param(instance_state)
  - scenario:
     condition: alarm_on_host and host_contains_instance and alarm_on_instance
     actions:
      - action:
          action_type: add_causal_relationship
          action_target:
            source: host_alarm
            target: instance_alarm
  - scenario:
     condition: alarm_on_host
     actions:
      - action:
          action_type: set_state
          action_target:
            target: host
          properties:
            state: SUBOPTIMAL

CLI Example

vitrage template add --path ./host_affects_instance_prototype.yaml --param 'host_alarm_rawtext'='High CPU load on host' --param 'instance_alarm_name'='High CPU load on instance'

Or:

vitrage template add --path ./host_affects_instance_prototype.yaml --params 'host_alarm_rawtext'='High CPU load on host', 'instance_alarm_name'='High CPU load on instance'

Alternatives

Hold a template prototype catalog

Define templates of type prototype that will be added to Vitrage database like other templates, but will be ignored by the evaluator.

There will be APIs to:

Add a template prorotype
View all template prototypes
Add a template based on an existing prototype name

Advantages:

At runtime, Vitrage will hold a template prototype catalog that can be used by the end users.

Disadvantages:

Since users will not add new templates very often, the advantage is not very important.
It will be harder to use the prototype ability in Heat, since the HOT template will have to depend on runtime information of Vitrage.
We will have to decide how to handle prototype updates. The same HOT template will behave differently if the prototype in Vitrage has changed.

Implement this feature in Heat

Instead of adding template prototypes in Vitrage, the prototyping option can be quite easily implemented in Heat. The main disadvantage is, of course, that the feature will not be available in Vitrage.

The entire Vitrage template can be written inside a Hot template, in one of two ways:

As part of the Vitrage Template resource in Heat. The disadvantage is that the Heat resource will be tightly coupled with Vitrage, and any syntax change in Vitrage should be reflected in Heat.
As a comment inside the Vitrage Template resource in Heat. The disadvantage is that writing a long yaml file as a comment is less clear and may result in syntax and alignment errors.

Data model impact

None. The template parameters will be given actual values, and standard templates will be added to the database.

REST API impact

Parameters should be added to vitrage template add function call, if the path points to a template with parameters . All parameters that are required in the template must be set.

There will be two ways to specify the parameters:

param: Followed by a key=value pair. There can be few param pairs.
params: Followed by a list of key=value parameters, separated by “,”.

Versioning impact

These changes will be part of Vitrage template version 3.

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None. In the future this feature may be used to write a template editor UI.

Implementation

Assignee(s)

Primary assignee:: ifat_afek

Work Items

Introduce Vitrage template version 3
Support template loading and parameters injection for template parameters
Support template validation for templates with parameters
New parameters for template add API
Documentation and tests

Dependencies

None

Testing

Unit tests, functional tests and tempest tests

Documentation Impact

The new template format will be documented

References

None

Prometheus Datasource Labels Mapping

Wed, 27 Feb 2019 00:00:00

StoryBoard link (task #28682): https://storyboard.openstack.org/#!/story/2004682

This blueprint describes the method of mapping Prometheus’s alert labels into Vitrage, in a way that will allow Vitrage to identify the resource that the alarm was raised on.

Problem description

Vitrage holds an entity graph with resources and alarms. Each alarm should be connected to its resource in the graph, in order to support alarm correlation. In Prometheus, an alert is based on a metric, where each metric can have different Labels. The labels contain enough information to identify the resource, however for each alert Vitrage may need to use one or more different label(s).

The purpose of this blueprint is to define a way for Vitrage to easily determine the identification method for every Prometheus alert.

In the use cases that are described below, we would like to create a Prometheus alarm vertex in Vitrage and connect it with an on edge to the right resource in the graph.

Prometheus alert structure

A Prometheus alert contains several fields, including:

annotations - annotations including title and description of the alert.
labels - a list of one or more labels. The labels are generated from alert rule and alert metrics that the alert is based on.
status - the current status of the alert: firing or resolved.

An example of a Prometheus alert:

{
    "annotations": {
        "description": "The average amount of CPU time spent in idle mode, per second, over the last minute (in seconds)",
        "title": "High average CPU time on idle mode"
    },
    "endsAt": "2018-12-30T09:43:52.589431274Z",
    "generatorURL": "http://devstack-rocky-release-4:9090/graph?g0.expr=100+%2A+%281+-+avg+by%28instance%29+%28irate%28node_cpu_seconds_total%7Bjob%3D%22node%22%2Cmode%3D%22idle%22%7D%5B5m%5D%29%29%29+%3E+20&g0.tab=1",
    "labels": {
        "alertname": "AvgCPUTimeOnIdleMode",
        "instance": "135.248.18.109:9100",
        "severity": "warning"
    },
    "receivers": [
        "vitrage"
    ],
    "startsAt": "2018-12-26T15:22:07.589431274Z",
    "status": "firing"
}

A full description of Prometheus alert structure can be found in prometheus_alert_description

Alerts based on libvirt metrics

All libvirt alerts have the following labels:

instance: holds the hostname/ip of where the exporter is running
domain: libvirt name that the exporter is scraping

Alert based on libvirt CPU metrics

Prometheus alert example

{
    "annotations": {
        "description": "Test alert to test libvirt exporter.\n",
        "title": "High cpu usage on vm"
    },
    "endsAt": "2018-12-30T09:44:05.91446215Z",
    "generatorURL": "http://devstack-rocky-release-4:9090/graph?g0.expr=rate%28libvirt_domain_info_cpu_time_seconds_total%5B1m%5D%29+%2A+10000+%3E+13&g0.tab=1",
    "labels": {
        "alertname": "HighCpuOnVmAlert",
        "domain": "instance-00000004",
        "instance": "135.248.18.109:9177",
        "job": "libvirt",
        "severity": "critical"
    },
    "receivers": [
        "vitrage"
    ],
    "startsAt": "2018-12-26T15:23:05.91446215Z",
    "status": "firing"
}

Vitrage resource

Vitrage resource can be uniquely identified by the instance and domain labels.

Alert based on libvirt network metrics

Prometheus alert example

{
    "annotations": {
        "description": "Another test alert to test libvirt exporter.\n",
        "title": "High traffic on bridge"
    },
    "endsAt": "2018-12-30T09:43:50.91446215Z",
    "generatorURL": "http://devstack-rocky-release-4:9090/graph?g0.expr=rate%28libvirt_domain_interface_stats_receive_bytes_total%5B5m%5D%29+%3E+0&g0.tab=1",
    "labels": {
        "alertname": "HighTrafficOnBridge",
        "domain": "instance-00000004",
        "instance": "135.248.18.109:9177",
        "job": "libvirt",
        "severity": "critical",
        "source_bridge": "br-int",
        "target_device": "tap456ab233-f4"
    },
    "receivers": [
        "vitrage"
    ],
    "startsAt": "2018-12-26T15:22:05.91446215Z",
    "status": "firing"
}

Vitrage resource

Short term: raise the alarm on the node or instance. Vitrage resource can be uniquely identified by the instance and domain labels.
Long term: Vitrage should hold a resource for br-int and the alarm should be connected to that resource. Vitrage resource can be uniquely identified by the instance, domain, source_bridge and target_device labels.

Node metrics

Prometheus alert

All Node metrics have a instance that holds the address of exporter. The exporter can scrape metrics from the instance it is running on. In this case ‘’instance`` label represents the resource address. Also, It can scrape different metrics not from the instance (e.g. network metrics). In this case instance is just an address of the exporter and other labels indicates to the resource.

Alert based on node CPU metric

Prometheus alert

CPU metrics are scraped from the instance so instance label represents the resource address.

Prometheus alert example

{
    "annotations": {
        "description": "The average amount of CPU time spent in idle mode, per second, over the last minute (in seconds)",
        "title": "High average CPU time on idle mode"
    },
    "endsAt": "2018-12-30T09:43:52.589431274Z",
    "generatorURL": "http://devstack-rocky-release-4:9090/graph?g0.expr=100+%2A+%281+-+avg+by%28instance%29+%28irate%28node_cpu_seconds_total%7Bjob%3D%22node%22%2Cmode%3D%22idle%22%7D%5B5m%5D%29%29%29+%3E+20&g0.tab=1",
    "labels": {
        "alertname": "AvgCPUTimeOnIdleMode",
        "instance": "135.248.18.109:9100",
        "severity": "warning"
    },
    "receivers": [
        "vitrage"
    ],
    "startsAt": "2018-12-26T15:22:07.589431274Z",
    "status": "firing"
}

Vitrage resource

Vitrage resource can be uniquely identified by the instance label.

Alert based on node filesystem metric

Prometheus alert example

{
    "annotations": {
        "description": "\"Consider ssh'ing into the instance and removing files or clean\ntemp files\"\n",
        "device": "/dev/vda1",
        "mount_point": "/",
        "runbook": "troubleshooting/filesystem_alerts_inodes.md",
        "title": "High number of inode usage",
        "value": "92.42%"
    },
    "endsAt": "2018-12-30T09:43:52.589431274Z",
    "generatorURL": "http://devstack-rocky-release-4:9090/graph?g0.expr=node_filesystem_files_free%7Bfstype%3D~%22%28ext.%7Cxfs%29%22%2Cjob%3D%22node%22%7D+%2F+node_filesystem_files%7Bfstype%3D~%22%28ext.%7Cxfs%29%22%2Cjob%3D%22node%22%7D+%2A+100+%3C%3D+100&g0.tab=1",
    "labels": {
        "alertname": "HighInodeUsage",
        "device": "/dev/vda1",
        "fstype": "ext4",
        "instance": "135.248.18.109:9100",
        "job": "node",
        "mountpoint": "/",
        "severity": "critical"
    },
    "receivers": [
        "vitrage"
    ],
    "startsAt": "2018-12-26T15:22:07.589431274Z",
    "status": "firing"
}

Vitrage resource

Short term: raise the alarm on the node or instance. Vitrage resource can be uniquely identified by the instance label.
Long term: Vitrage should hold a resource for ext4 and the alarm should be connected to that resource. Vitrage resource can be uniquely identified by the instance, device and fstype labels.

Proposed change

A configuration file that maps the Prometheus labels to a corresponding Vitrage resource with specific properties (id or other unique properties). The mapping will most likely be defined by the alert name and other fields.

Prometheus configuration file structure

The configuration file contains a list of alerts. Each alert contains key and resource.

The key contains labels which uniquely identify each alert.

The resource specifies how to identify in Vitrage the resource that the alert is on. It contains one or more Vitrage property names and corresponding Prometheus alert labels.

Configuration file example

alerts:
- key:
    alertname: HighCpuOnVmAlert
    job: libvirt
  resource:
    instance_name: domain
    host_id: instance
- key:
    alertname: HighTrafficOnBridge
    job: libvirt
  resource:
    instance_name: domain
    host_id: instance
- key:
    alertname: AvgCPUTimeOnIdleMode
  resource:
    id: instance
- key:
    alertname: HighInodeUsage
    job: node
  resource:
    id: instance

Alternatives

None

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

TBD

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: 7mode3294 (Muhamad Najjar)

Work Items

Load configuration file and use it in the Prometheus transformer.
Documentations and tests.

Dependencies

None

Testing

Unit tests, functional tests and tempest tests

Documentation Impact

The new configuration will be documented

References

Prometheus datasource: https://github.com/openstack/vitrage/tree/master/vitrage/datasources/prometheus
Prometheus alerting rules: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
Prometheus libvirt exporter: https://github.com/CanonicalLtd/prometheus-openstack-exporter
Prometheus node exporter: https://github.com/prometheus/node_exporter

Service List API

Wed, 27 Feb 2019 00:00:00

StoryBoard link: https://storyboard.openstack.org/#!/story/2004897

This spec adds a new api for listing all vitrage services and their status.

Problem description

In a production cloud environment, Vitrage will have multiple services deployed on multiple hosts. This will enable an admin to find these services and get details like:

what is the node on which vitrage service is running,
what is the running status of vitrage service.
How long the vitrage services are running successfully.

Proposed change

A new api command vitrage service list will be added that will list all the vitrage services that are currently running, where are they running, their status and how long are they running.

Example of a response

[
    {
      "Created At": "2019-02-10T11:07:15+00:00",
      "Hostname": "controller-1",
      "Process Id": 23161,
      "Name": "ApiWorker worker(0)"
    },
    {
      "Created At": "2019-02-10T11:07:15+00:00",
      "Hostname": "controller-1",
      "Process Id": 23153,
      "Name": "EvaluatorWorker worker(0)"
    },
    {
      "Created At": "2019-02-10T11:07:15+00:00",
      "Hostname": "controller-1",
      "Process Id": 23155,
      "Name": "EvaluatorWorker worker(1)"
    },
    {
      "Created At": "2019-02-10T11:07:15+00:00",
      "Hostname": "controller-1",
      "Process Id": 23157,
      "Name": "EvaluatorWorker worker(2)"
    },
    {
      "Created At": "2019-02-10T11:07:15+00:00",
      "Hostname": "controller-1",
      "Process Id": 23158,
      "Name": "EvaluatorWorker worker(3)"
    },
    {
      "Created At": "2019-02-10T11:07:33+00:00",
      "Hostname": "controller-1",
      "Process Id": 23366,
      "Name": "MachineLearningService worker(0)"
    },
    {
      "Created At": "2019-02-10T11:07:35+00:00",
      "Hostname": "controller-1",
      "Process Id": 23475,
      "Name": "PersistorService worker(0)"
    },
    {
      "Created At": "2019-02-10T11:07:15+00:00",
      "Hostname": "controller-1",
      "Process Id": 23164,
      "Name": "SnmpParsingService worker(0)"
    },
    {
      "Created At": "2019-02-10T11:14:30+00:00",
      "Hostname": "controller-1",
      "Process Id": 25698,
      "Name": "vitrageuWSGI worker 1"
    },
    {
      "Created At": "2019-02-10T11:14:30+00:00",
      "Hostname": "controller-1",
      "Process Id": 25699,
      "Name": "vitrageuWSGI worker 2"
    },
    {
      "Created At": "2019-02-10T11:07:32+00:00",
      "Hostname": "controller-1",
      "Process Id": 23352,
      "Name": "VitrageNotifierService worker(0)"
    }
]

CLI Example

+----------------------------------+------------+--------------+---------------------------+
| Name                             | Process Id | Hostname     | Created At                |
+----------------------------------+------------+--------------+---------------------------+
| ApiWorker worker(0)              |      23161 | controller-1 | 2019-02-10T11:07:15+00:00 |
| EvaluatorWorker worker(0)        |      23153 | controller-1 | 2019-02-10T11:07:15+00:00 |
| EvaluatorWorker worker(1)        |      23155 | controller-1 | 2019-02-10T11:07:15+00:00 |
| EvaluatorWorker worker(2)        |      23157 | controller-1 | 2019-02-10T11:07:15+00:00 |
| EvaluatorWorker worker(3)        |      23158 | controller-1 | 2019-02-10T11:07:15+00:00 |
| MachineLearningService worker(0) |      23366 | controller-1 | 2019-02-10T11:07:33+00:00 |
| PersistorService worker(0)       |      23475 | controller-1 | 2019-02-10T11:07:35+00:00 |
| SnmpParsingService worker(0)     |      23164 | controller-1 | 2019-02-10T11:07:15+00:00 |
| vitrageuWSGI worker 1            |      25698 | controller-1 | 2019-02-10T11:14:30+00:00 |
| vitrageuWSGI worker 2            |      25699 | controller-1 | 2019-02-10T11:14:30+00:00 |
| VitrageNotifierService worker(0) |      23352 | controller-1 | 2019-02-10T11:07:32+00:00 |
+----------------------------------+------------+--------------+---------------------------+

Note: The cloud operator must pre-install zookeeper or other tooz backend component. Otherwise, an exception will be raised when users call the service REST API.

If vitrage is running in k8s cluster then this api might be redundant. Since k8s handles pods health and topology. We might make the service api communicate with the k8s api in this case to get all vitrage services and their statuses.

Data model impact

None.

REST API impact

New api will be added to vitrage to list the services.

Versioning impact

None.

Other end user impact

In order to support the api we will need a backend to store the information. We will use the tooz library that supports multiple backends see Tooz.

Deployer impact

The deployer must pre-install zookeeper or other tooz backend component In order to support the API.

When deploying a container then hostname must be changed so api will be readable.

Developer impact

We need to think what to do in case of a container deployment. docker by default has a hostname of the container id but it can be changed.

We might use an optional environment variable (e.g HOST_HOSTNAME) for host name if exist in case of a container that can be passed to the container.

Horizon impact

None.

Implementation

Assignee(s)

Primary assignee:: Eyal

Work Items

add tooz support
add new API to vitrage
add service list to vitrage client
Documentation and tests

Dependencies

Depends on the tooz library with a backend configured.

Testing

Unit tests, functional tests and tempest tests

Documentation Impact

The new api will be documented

References

None

Short Template Format

Wed, 27 Feb 2019 00:00:00

StoryBoard link: https://storyboard.openstack.org/#!/story/2004871

This spec suggests a shorter format for Vitrage template files. The new format should be shorter, read faster and written more easily.

Problem description

Vitrage template language is powerful, yet a bit verbose and it’s quite easy to make syntax mistakes when creating a new template from scratch. We would like to make it easier to generate any template.

For more information on the current format, see Template_format.

Proposed change

Several simplifications that will result in a shorter template file while preserving it’s capabilities.

Remove definitions level nesting, entities will directly replace it

Remove relationships, these will be represented inline in the condition, example usage condition: host_ssh_alarm [ on ] host

entities will be changed from a list to a dictionary, template_id is removed and the entity key will be used instead.

In actions the field action_type is removed and the action key will be used instead

In each action action_target nesting level is removed

In scenarios removed the scenario nesting level

Action set_state will be replaced with set_suboptimal, set_error and set_ok.

raise_alarm action will have a new optional field causing_alarm this will add a causal relationship between the new alarm and the causing alarm, de-necessitating the use of add_causal_relationship

Example for a short format template

metadata:
 version: 3
 name: zabbix alarm for network interface and ssh affects host instances
 description: zabbix alarm for network interface and ssh affects host instances
entities:
 host_network_alarm:
  type: zabbix
  rawtext: host network interface is down
 host_ssh_alarm:
  type: zabbix
  rawtext: host ssh is down
 instance:
  type: nova.instance
 host:
  type: nova.host
scenarios:
 - condition: host_ssh_alarm [ on ] host
   actions:
     - set_suboptimal:
        target: host
 - condition: host_network_alarm [ on ] host AND host_ssh_alarm [ on ] host
   actions:
     - add_causal_relationship:
        source: host_network_alarm
        target: host_ssh_alarm
 - condition: host_ssh_alarm [ on ] host AND host [ contains ] instance
   actions:
     - raise_alarm:
        target: instance
        alarm_name: instance is down
        severity: WARNING
        causing_alarm: host_ssh_alarm
     - set_error:
        target: instance

Example for an equivalent template in the previous format

metadata:
 version: 2
 type: standard
 name: zabbix alarm for network interface and ssh affects host instances
 description: zabbix alarm for network interface and ssh affects host instances
definitions:
 entities:
  - entity:
   category: ALARM
   type: zabbix
   rawtext: host network interface is down
   template_id: host_network_alarm
  - entity:
   category: ALARM
   type: zabbix
   rawtext: host ssh is down
   template_id: host_ssh_alarm
  - entity:
   category: ALARM
   type: vitrage
   name: instance is down
   template_id: instance_alarm
  - entity:
   category: RESOURCE
   type: nova.instance
   template_id: instance
  - entity:
   category: RESOURCE
   type: nova.host
   template_id: host
 relationships:
  - relationship:
   source: host_network_alarm
   relationship_type: on
   target: host
   template_id : network_alarm_on_host
  - relationship:
   source: host_ssh_alarm
   relationship_type: on
   target: host
   template_id : ssh_alarm_on_host
  - relationship:
   source: host
   relationship_type: contains
   target: instance
   template_id : host_contains_instance
  - relationship:
   source: instance_alarm
   relationship_type: on
   target: instance
   template_id : alarm_on_instance
scenarios:
 - scenario:
     condition: ssh_alarm_on_host
     actions:
      - action:
          action_type: set_state
          action_target:
            target: host
          properties:
            state: SUBOPTIMAL
 - scenario:
     condition: network_alarm_on_host AND ssh_alarm_on_host
     actions:
      - action:
          action_type: add_causal_relationship
          action_target:
            source: host_network_alarm
            target: host_ssh_alarm
 - scenario:
     condition: ssh_alarm_on_host AND host_contains_instance
     actions:
      - action:
          action_type: raise_alarm
          action_target:
            target: instance
          properties:
            alarm_name: instance is down
            severity: WARNING
      - action:
          action_type: set_state
          action_target:
            target: instance
          properties:
            state: ERROR
 - scenario:
     condition: ssh_alarm_on_host AND host_contains_instance AND alarm_on_instance
     actions:
      - action:
          action_type: add_causal_relationship
          action_target:
            source: host_ssh_alarm
            target: instance_alarm

Alternatives

None.

Data model impact

None. Templates will be stored in the same data model.

REST API impact

None

Versioning impact

These changes will be part of Vitrage template version 3.

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None.

Implementation

Assignee(s)

Primary assignee:: idan_hefetz

Work Items

Introduce Vitrage template version 3
Support template validation and loading
Documentation and tests

Dependencies

None

Testing

Unit tests, functional tests and tempest tests

Documentation Impact

The new template format will be documented

References

None

Vitrage-graph fast failover

Tue, 13 Nov 2018 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/vitrage-fast-failover

vitrage-graph high availability should meet these requirements:

Single active instance of vitrage-graph (managed by pacemaker).
Initialize quickly upon failover without requesting updates.
In case of a long downtime, vitrage-graph startup will request collector updates

Problem description

Vitrage-graph is active standby. Currently on a failover, vitrage-graph needs to pull all the data again from the collector data-sources. This takes a considerable amount of time, in which data is inconsistent. As we wish to continue working with an in-memory graph (due to performance), vitrage-graph service will remain active-standby. Therefore, downtime must be minimized in failover events.

Proposed change

after every get_all, vitrage-graph stores a full entity graph snapshot in the db, so the majority of events do not need to be replayed.

Vitrage-graph sends each processed event to vitrage-persistor so these are stored in the order of handling.

Upon init vitrage-graph queries the db table graph_snapshots, fetching the latest entry, it will be used if it is not older than snapshot_interval.

Init with a snapshot - on failover

Unpickle stored snapshot to get the graph.
Run the processor on all the events (from events table) that occurred after the snapshot.
Enable the evaluators.
Process all the events that are waiting in the message bus.

Init without a snapshot - a fresh start (This is the current behaviour).

Start with a new empty graph.
RPC to Collector to run get_all for all drivers, then process the events.
Process all the events that are waiting in the message bus.
Enable evaluator and iterate all graph.

Alternatives

Using a persistent graph database can improve vitrage-graph high availability as fail-over will be quick due to running active-active. This may be a preferred solution in terms of high availability, but overall, when comparing performance compared to in-memory networkx, the degradation is not reasonable

Data model impact

May require minor changes, TBD.

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

This will be enabled by default. Deployer may disable in by adding the following to vitrage.conf [persistancy] enable_persistancy=false

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: idan-hefetz
Other contributors:: None

Work Items

None

Dependencies

None

Testing

Additional tempest will be added for fail-over, as persistence is already covered by existing tempest. Unit tests will not be affective here as changes are mostly in the init process and scheduler. This feature mostly reuses existing (tested) functionality.

Documentation Impact

None

References

None

kubernetes Datasource Driver - get_all implementation

Tue, 13 Nov 2018 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/k8s-datasource

This blueprint describes the kubernetes datasource, and its implementation for get_all nodes (VM’s).

Problem description

Kubernetes nodes should be added to Vitrage Graph via Vitrage Datasources. This requires writing a Datasource for kubernetes.

The datasource should support get_all: periodic query all kubernetes nodes.

Proposed change

Kubernetes datasource will be configured with:

Poll interval in seconds (default: 600)

Every poll-interval seconds, Kubernetes Driver will call kubernetes client to retrieve all nodes. The nodes will be converted to Vitrage datasource events and passed to Vitrage Graph queue.

Example of kubernetes client return call for list_nodes()

apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: 2017-11-29T07:24:59Z
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: regionOne
      failure-domain.beta.kubernetes.io/zone: zone0
      is_control: "true"
      kubernetes.io/hostname: bcmt-control-01
    name: bcmt-control-01
    namespace: ""
    resourceVersion: "10714282"
    selfLink: /api/v1/nodes/bcmt-control-01
    uid: 68011206-d4d6-11e7-9c63-fa163e2e2123
  spec:
    externalID: 41c40aab-80e9-4bb6-a280-27976bfc811f
    providerID: openstack:///41c40aab-80e9-4bb6-a280-27976bfc811f
    taints:
    - effect: NoExecute
      key: is_control
      timeAdded: null
      value: "true"
  status:
    addresses:
    - address: 172.16.1.12
      type: InternalIP
    - address: 10.5.138.49
      type: InternalIP
    - address: bcmt-control-01
      type: Hostname
    allocatable:
      cpu: "2"
      memory: 3779500Ki
      pods: "110"
    capacity:
      cpu: "2"
      memory: 3881900Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: 2018-02-15T12:08:58Z
      lastTransitionTime: 2018-02-14T13:39:53Z
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: 2018-02-15T12:08:58Z
      lastTransitionTime: 2018-02-14T13:39:53Z
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: 2018-02-15T12:08:58Z
      lastTransitionTime: 2018-02-14T13:39:53Z
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: 2018-02-15T12:08:58Z
      lastTransitionTime: 2018-02-14T13:39:53Z
      message: kubelet is posting ready status
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - 172.16.1.4:5000/gcr.io/google-containers/hyperkube-amd64
      - 172.16.1.4:5000/gcr.io/google-containers/hyperkube-amd64:v1.7.4
      sizeBytes: 615424570
    nodeInfo:
      architecture: amd64
      bootID: 883c98a9-17ea-40f9-af7d-a448ad817249
      containerRuntimeVersion: docker://1.12.6
      kernelVersion: 3.10.0-514.21.1.el7.x86_64
      kubeProxyVersion: v1.7.4
      kubeletVersion: v1.7.4
      machineID: 10783ea106f742728fede153a98b035d
      operatingSystem: linux
      osImage: Red Hat Enterprise Linux Server 7.3 (Maipo)
      systemUUID: 41C40AAB-80E9-4BB6-A280-27976BFC811F

Relevant data will be extracted:

Creation Timestamp.
Name.
Addresses (IP’s)
kubernetes ID (uid)
provider
providerID

Alternatives

None

Data model impact

New vertices will be added to the entity graph. This might be duplicate vertices. (VM’s from Nova and kubernetes). Proposed solution is resource equivalence. (planned for future work)

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

Kubernetes driver should be configured.(get access to master node)

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: Idan-kinory

Work Items

None

Dependencies

None

Testing

This blueprint requires unit tests.

Documentation Impact

Datasource configuration.

References

Datasource main blueprint: https://blueprints.launchpad.net/vitrage/+spec/k8s-datasource

Vitrage-Collector on demand

Tue, 13 Nov 2018 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/rpc-collector

A simpler implementation of Vitrage high availability, collector service should meet these new requirements:

collector should be active-active.

Dependency between the collector and graph services should be removed.

vitrage-graph should be able to request updates from the collector when needed.

Problem description

The current implementation, where vitrage collector and vitrage graph are always restarted simultaneously, is not desired, but is due to the lack of a better triggering event for get_all. In addition message bus may overload occasionaly due to this behaviour, when vitrage-graph is down and does not consume the messages.

Proposed change

Collector receives synchronous RPC and retrieves a list of all the events.

vitrage-collector implementation:

Remove timers for get_all and get_changes
get_all and get_changes can be called by RPC
get_all and get_changes receive a list parameter datasource_names.
Will return all the entities from all the specified data-sources
Will open a thread to write all the events to the Persistor message queue in the background.
Remove dependency between the collector and graph services (.service files)

vitrage-graph implementation:

Should manage the RPC calls to get_all and get changes.
Add appropriate timers calling get_all and get_changes by RPC
List of events should be processed using an event coordinator (before TwoPrirityQueue)
A snapshot should be taken after each get_all once the events have been processed. So the majority of events do not need to be replayed

Alternatives

None

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

vitrage-collector could be restarted individually from vitrage-graph

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: idan-hefetz
Other contributors:: None

Work Items

None

Dependencies

None

Testing

New behaviour is already covered by existing tempest

Documentation Impact

None

References

None

Nova Entity Transformer

Thu, 14 Jun 2018 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/nova-entity-transformer

When an entity event is being introduced into Vitrage graph, it must be transformed into a type which can be handled by the graph. The Entity transformer is responsible for converting an entity event into entity wrapper. Each entity type has his own transformer.

All transformers should be written in python and added to vitrage.conf. The Entity Transformer Engine allows a service provider to extend support for other entities by writing a designated transformer for each entity type.

The first transformers to implement are:

Nova Instance transformer
Nova Host transformer

Problem description

In Vitrage graph, each (instance of an) entity is represented as a vertex. Therefore, the output of a transformer is a vertex for the given entity (an “entity vertex”). In addition, the transformer also returns a list of (vertex, edge)-pairs. The vertex in each pair describes a neighbor with limited properties, and the edge represents the connection between both vertices, describing their relationship.

There are different Entity Types that should be supported by Vitrage:

Openstack types (Nova.instance, Nova.host and etc.)
Non-Openstack types (Nagios tests, Physical resources and etc.)

Note: in Vitrage Graph, each vertex contains a dictionary of key-value pairs which represents the entity properties. Similarly, each edge contains a dictionary of key-value pairs which represents aspects of the relation between two vertices.

Proposed change

When the Entity Processor receives a new entity event, it asks the Entity Transformer to convert the event into an object which the processor can then enter into the Vitrage Graph.

Transformer Operation: When receiving an entity event, the Transformer Engine first recognizes the entity type and accordingly activates the corresponding transformer. Each transformer inherits from the Transformer base class and implements the three methods:

Method that returns an entity wrapper
Method that returns key fields and their order.The key fields are mandatory
Method that returns a key by given an entity event

Output Object

The transformer returns an Entity Wrapper, which is a tuple containing an entity vertex and a list of (vertex,edge) pairs that describe the entity neighbors (relationships).

Entity (source) vertex description:

Mandatory properties:

key - For Openstack entities this is the Openstack ID. For non-Openstack
entities this is an ID which will generated by Vitrage
Type - Resource Physical Resource Alarm Tests Results
Sub Type - Alarm Name, host, instance, switch and etc.
Entity Name
Is Deleted- needed for graph maintenance and marks items that can be
gathered by the garbage collector

Optional properties (vertex metadata)

State
Project ID

The optional properties list is flexible and can be changed as needed. In addition, each entity type can have its own relevant properties.

(vertex, edge) Pairs:

The pair describes a entity’s neighbor vertex and their relationship. Relationships can be both physical, virtual or (in the future) logical

Target vertex description:

The vertex in the pair must have sufficient data to help specify uniquely which vertex in the Vitrage Graph will be connected to this entity. Currently, the minimal information needed for this is:

ID - For each entity, must have the data which vertex it connects
Type - Resource Physical Resource Alarm Tests Results
Sub Type - Alarm Name, host, instance, switch and etc.

Edge description:

Source ID Entity ID
Target ID - For each entity, must have the data which vertex it connects
Relation Type - contains, run, attached and etc.

Event Type:

The type of the event as it happened. Possible options:

CREATE - New entity is created
UPDATE - The entity has been updated
DELETE - When the entity is deleted

Alternatives

None

Data model impact

TBD

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

TBD

Developer impact

TBD

Horizon impact

None

Implementation

Assignee(s)

liat har-tal

Work Items

None

Dependencies

None

Testing

All code will be tested

Documentation Impact

None

References

Vitrage project <https://wiki.openstack.org/wiki/Vitrage>

Synchronizer

Thu, 14 Jun 2018 00:00:00

The synchronizer is a vitrage inner module which supplies both a snapshot (all the entities) of openstack services via poll requests, or, change notifications for entities which comprise the same OS services, via push notifications.

Problem description

Enable vitrage and to maintain the most up-to-date view of OS services, Nagios, etc., it samples.

Proposed change

periodically samples OS services, Nagios, etc., and produces up to date entities once changed (in contrast to deltas)

design

interacting with the module

get all notifications

list of entity types

change notifications

registration:
supply a list of entity types and a callback function for the synchronizer to invoke for sending information back concurrent immediate execution (a call for this function is would not block the caller)done by the vitrage graph manager
push notifications

synchronizers response

either list-all and notifications response contains entities
each entity must contain:keyparent/child keytypetimestamp?CRUD action (new entity/updated/…)state (active/shutdown/…)project (role)

synchronizer plugins

kinds of plugins:

Message bus plugin

Collector plugin

+-------------------------------------------------------------------------------------------------+
|                                                                                                 |
|       vitrage graph    ^                                                      ^                 |
|                        |                                                      |                 |
+-------------------------------------------------------------------------------------------------+
                         |                                                      |
+------------------------------------------------------------------------------------------------+
|                        |          synchronizer                                |                |
|                        |                                                      |                |
|                        |                                                      |                |
|               +--------+-----------------------+     collect          +-------+------------+   |
|               |                                |     notifications    |                    |   |
|               |     worker                     +---+-------------+    |  worker            |   |
|               |                                |   |             |    |                    |   |
|               +--------------------------------+   |             |    +------------------+-+   |
|                          notification              |             |          collect      |     |
|                                                    |             |          notifications|     |
|                                                    |             |                       |     |
| +------------------------------------------------------------+  +v----------+  +------------+  |
| |                         collector plugin         |         |  | msg bus   |  |  msg bus|  |  |
| |                                                  |         |  | plugin    |  |  plugin v  |  |
| |                                                  |         |  |           |  |            |  |
| |                                                  |         |  |           |  |            |  |
| |                                                  |         |  |           |  |            |  |
| |                                                  |         |  |           |  |            |  |
| |                                                  |         |  |           |  |            |  |
| |                                                  |         |  |           |  |            |  |
| |             +----------------------------+       |         |  |           |  |            |  |
| |    baseline:|hash0|hash1|hash2|hash3|... | <-----+         |  |           |  |            |  |
| |             +----------^-----------------+       |         |  |           |  |            |  |
| |                        |if                       |         |  |           |  |            |  |
| |                        |hash(item)!=hash1 ==>    |         |  |           |  |            |  |
| |                        |propagate item as a      |         |  |           |  |            |  |
| |                        |change notification      |         |  |           |  |            |  |
| |                        |                         |         |  |           |  |            |  |
| |            +-----------+---------------+         |         |  |           |  |            |  |
| |            |sampling  ^             <------------+         |  |           |  |            |  |
| |            +----------|----------------+                   |  |           |  |            |  |
| |                       |                                    |  |           |  |            |  |
| |                                                            |  |           |  |            |  |
| |                       |                                    |  |           |  |            |  |
| +------------------------------------------------------------+  +-----------+  +------------+  |
|                         |                                                                      |
+------------------------------------------------------------------------------------------------+
                          |
              +-----------+-------------------+
              |      OS service               |
              +-------------------------------+

samples OS services, Nagios, etc.
via each OS service REST API
method of collection: retrieve OS service elements list. For the purposes of:change notifications - as part of the periodic collection of the latest snapshot we’d use this method of collection against OS services which doesn’t propagates change notification.run on its own collection subprocesses how to discover a change once a snapshot is collected, we’d like to know which entity was changed from the latest time a snapshot was taken in order to enable this, we’d keep a baseline - a data structure which contains for each OS service entity (such as a vm instance), its ID + it latest collection timestamp or a hash which represents its latest state.by comparing the latest snapshot of elements against the baseline, we’d know for which entity we’d like to propagate a notification for.collect deltas for services which reveals this functionality for the purposes of change notifications this is the easy case, where change notifications are simply passed on to whoever registered for them
deployment - as a library

consumer flow

new plugin:

write my_plugin.py:
my_plugin(synchronizer.BasePlugin): … register my_plugin class path in conf file install an egg which contains a my_plugin.py restart vitrage service
initial configuration:
- configure nova, heat, etc. to send message bus notifications for vitrage too (consumption of msg bus notification by the listener removes it from the queue)
- configure active entity types
- configure message bus credentials for the msg bus listener
- configure nagios credentials
register for ‘change notification’ - for a streaming change notifications, upon synchronizer initialization, the consumer would supply the queue on which the notifications would be returned to upon synchronizer initialization which would spawn subprocesses which would call the ‘get all’ periodically in order to supply the graph with the latest snapshot.

Alternatives

None

Data model impact

TBD

REST API impact

TBD

Versioning impact

TBD

Other end user impact

None

Deployer impact

TBD

Developer impact

TBD

Horizon impact

None

Implementation

Assignee(s)

TBD

Work Items

TBD

Dependencies

TBD

Testing

Unit Tests - Tox Integration Tests - Tempest

Documentation Impact

TBD

References

TBD

Vitrage Template Validation API

Thu, 14 Jun 2018 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/template-validate-api

An API for validating templates

Problem description

We would like to be able to validate a single template (or several templates) through api before uploading it to Vitrage.

Proposed change

Create API to validate Vitrage templates in terms of content and syntax.

By given a full path to template file, validate a single template.
By given a full path to directory, validate all template files inside it.

The template validate API returns a result that contains the following fields:

status - validation succeeded/failed
file path - the full path to the template file
description
message - error message
status code

REST API impact

Template Validate

Validate Vitrage template(s)

POST /

Headers

X-Auth-Token (string, required) - Keystone auth token
Accept (string) - application/json
User-Agent (String)
Content-Type (String): application/json

Path Parameters

None.

Query Parameters

path (string(255), required) - the path to template file or directory

Request Body

None.

Request Examples

POST /v1/template/?path=[file/dir path]
Host: 135.248.18.122:8999
User-Agent: keystoneauth1/2.3.0 python-requests/2.9.1 CPython/2.7.6
Content-Type: application/json
Accept: application/json
X-Auth-Token: 2b8882ba2ec44295bf300aecb2caa4f7

Response

Status code

200 - OK
400 - Bad request

Response Body

Returns a JSON object that is a list of results. Each result describes the full validation (syntax and content) of one template file.

Response Examples

{
  "results": [
    {
      "status": "validation failed",
      "file path": "/tmp/templates/basic_no_meta.yaml",
      "description": "Template syntax validation",
      "message": "metadata is a mandatory section.",
      "status code": 62
    },
    {
      "status": "validation OK",
      "file path": "/tmp/templates/basic.yaml",
      "description": "Template validation",
      "message": "Template validation is OK",
      "status code": 4
    }
  ]
}

Implementation

Assignee(s)

liat har-tal <liat.har-tal@nokia.com>

Dependencies

Depends on the template validation blueprints

Testing

Tempest tests also need to be added in order to test:

Validate single template
Validate several templates

Documentation Impact

The new api should be documented

References

None

Get Aodh Alarms From Message Bus Notifications

Thu, 14 Jun 2018 00:00:00

Get Aodh alarms immediately via notification from the message bus.

Problem description

Currently Aodh datasource queries Aodh alarms periodically. It is not in time and it is also not efficient, since most query will get no change when the interval of alarm evaluation in Aodh is larger than the interval of pulling Aodh alarm in Vitrage.

Proposed change

Aodh update_method in vitrage config file will be configured with:

/etc/vitrage/vitrage.conf

[aodh]
update_method = push

Add a new notification topic in Aodh config file:

/etc/aodh/aodh.conf

[oslo_messaging_notifications]
topics = vitrage_notifications

Vitrage listener will get the alarm events from the message bus. Aodh driver will filter the needed event types, enrich the events and then send them to the queue for the Aodh transformer to create, update or delete the entity vertex.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

performance improvement of getting Aodh alarms

Other deployer impact

Config update_method as ‘push’ in Aodh section in vitrage config file.

Add the vitrage_notifications topic in oslo_messaging_notifications section in Aodh config file.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: dongwenjuan <dong.wenjuan@zte.com.cn>

Work Items

None

Future lifecycle

None

Dependencies

None

Testing

Unit tests and tempest tests.

Documentation Impact

Documentation will be modified to describe how to configure the notification topics in Aodh when using devstack to deploy OpenStack env.

References

None

Vitrage Multi Tenancy Support

Thu, 14 Jun 2018 00:00:00

When a tenant uses Vitrage APIs we would like to show him only the data that is relevant to him.

Problem description

Vitrage needs to show in its API an CLI only the data relevant to that tenant (it can’t show all the data due to irrelevancy and privacy of each tenant). Thus for each datasource and entity we need to know what relevant data to show for that tenant. We would also like to show all of the data if someone adds the all_tenants property.

Proposed change

Here is a description, for each of the Vitrage APIs, how it should behave for each tenant:

Get alarms:

Find all the alarms with the requested project_id (if the project is admin then show also alarms that has no project_id property)
Find all the resources with the requested project_id and return all the alarms that are attached to them.
Merge the results from the previous steps and return it.

Get RCA:

Find all the alarms that this alarm has caused, recursively. When reaching an alarm that is not of same project_id (or on resource of same project_id), stop the recursion, including this last alarm in the response.
Find all the alarms that caused this alarm, recursively. When reaching an alarm that is not of same project_id (or on resource of same project_id), stop the recursion, including this last alarm in the response.
Merge the results from the previous steps and return it.

Get Topology:

Find all the connected components for project_id.
For each component, select 1 entity and find all paths (without cycles) to “openstack.cluster” entity. Add all of the vertices in the path to the component.
Merge all the components and return it.

Remark: API and CLI needs to behave the same.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: alexey_weyl <alexey.weyl@alcatel-lucent.com>

Work Items

None

Future lifecycle

None

Dependencies

None

Testing

None

Documentation Impact

None

References

None

Static Data Source Configuration

Thu, 14 Jun 2018 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/static-datasource-config-format

The configuration of static data source has a lot in common with entity and relationships definition in evaluator template. This blueprint proposes a refactoring to reuse the template format and parsing methods in static data source. By doing so, we may reduce the work in maintenance and bring in new features more easily.

Problem description

Currently the configuration of static data source use a dedicated format, which has a lot of overlapping with the evaluator templates.

In static configuration, there are entities and their relationships

- entities
  - {entity}
  - {entity}

In each entity

- name:
  id:
  relationship:
    - {relationship}
    - {relationship}

In evaluator templates we define: entities, relationship and scenarios. Each scenario has a condition and actions.

- definitions
  - entities
    - {entity}
    - {entity}
  - relationships
    - {relationship}
    - {relationship}

Though serving different purpose, they both

Describe entities and relationships
Use a dedicated key (id/template_id) to reference the items
Include a source entity and target entity in relationship

The main differences between the two are

Evaluator templates defines a topology and scenarios based on it
Static config defines a topology and adds it to the graph

We may define the static configuration using the same format as the evaluator templates. And then simulate an entity discovery from the same file.

By reusing the template parsing engine and workflow, we may reduce the work in maintenance and bring in new features more easily.

Proposed change

Refactoring the static data source template and use the same parser as in evaluator template.

Discover entities from the static data source template.

For backward compatibility, static data source will take over the control of the default configuration folder /etc/vitrage/static_datasource which was used by static physical datasource.

Both legacy format and new format will be placed in the same folder. static datasource parse the file and check the existence of meta to decide which engine to use. If not found, proxy the job to static physical datasource and print a deprecation warning.

Static physical datasource will be disabled by default and throws exception if running standalone.

Alternatives

Fix the issues found in static data source without refactoring the format. This will keep best back-compatibility but will cause redundant work with scenario evaluator.

Data model impact

None

REST API impact

None

Versioning impact

Backward compatibility with old format will be kept.
This introduces a new feature and a minor version incremental is required.

Other end user impact

New format of static data source configuration should be applied by the end user.

Deployer impact

Old parser will be kept but a deprecated warning will be prompt.

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: yujunz
Other contributors:: None

Work Items

Reuse the parser of evaluator template in static data source configuration.
Discover entities from the configuration.
Add deprecated warning on old format.

Dependencies

None

Testing

The changes shall be covered by new unit test.

Documentation Impact

New format of the template shall be documented.

References

None

Vitrage ID

Thu, 14 Jun 2018 00:00:00

Vitrage ID will be standard generated UUID.

Problem description

Currently Vitrage ID is actually a set of properties. This can create duplicates and is very misleading. Furthermore it will impair us once we have history, since same alarms can happen multiple times. Thus, it should be the same as any other service in Openstack and provide an ID based on Openstack UUID. generating algorithm.

Proposed change

Changing Vitrage ID from it’s current algorithm to an OpenStack compliant “UUIDUtils” generated UUID.

All the documentations and examples in Vitrage project should be updated to use an OpenStack compliant UUID.

A few Mock Json files exist for “mock api” requests purposes, for example, alarms.sample.json. They contain the “old” Vitrage ID. The “mock” api should be deleted, and the same should go with these file. We can also just fix the examples in the mock.

Alarm IDs: When creating Vitrage ID for an alarm, we also need to put it in the metadata when updating AODH / other. Afterwards, when getting the updated alarm back, we will to update the alarm ID in Vitrage, In case we will have simultaneous multiple alarm engines, we might need to have an “ServiceName / ID” map for the alarm, and the Alarm ID in Vitrage will be “Vitrage ID”.

Template IDs should be changed back from a calculated String to generated uuid, in scenario_repository.

Datasources:

key / value tests : fix field names.
Transformers: No change is needed in the Transformers.
Processor: Checking if an entity exists in the graph: The entity is currently queried in the Graph according to its Vitrage ID. Instead, it will be queried according to the parameters set. If the entity exists, it’s original Vitrage ID will be used. Otherwise, a new UUID will be generated for vitrage ID via openstack UUIDUtils’ generate_uuid.

Update all necessary tests.

No changes are needed in Evaluator Action / Recipes or in Consistency enforcer.

Performance/Scalability Impacts

Performance needs to be tested, since after the development of this blueprint, all graph queries will use parameters instead of a single index.

As long as Vitrage uses an in memory graph database, starting from this change, standard HA will be buggy, to say the least. An entity’s Vitrage ID will have different values in each HA “instance”. Using Pacemaker equivalent HA (stonith) will solve this.

Implementation

Assignee(s)

Primary assignee:: doffek <dany.offek@gmail.com>

Dependencies

Aodh : Need to change the notification from Vitrage to Aodh.

Testing

Unit tests and tempest tests.

Documentation Impact

All documentation regarding the creation of Vitrage ID will be updated.

References

None

Integration with Mistral

Thu, 14 Jun 2018 00:00:00

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

http://creativecommons.org/licenses/by/3.0/legalcode

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/integration-with-mistral

Support executing Mistral workflows from Vitrage.

Problem description

Vitrage provides insights about the state of the cloud, but is not meant to be a policy engine. In order to take corrective actions, for example, we need to integrate an external engine like Mistral - the OpenStack workflow engine.

Proposed change

It will be possible to define in Vitrage templates that under certain conditions, a Mistral workflow should be executed. This gives the user the power to decide, for example, that different corrective actions should be taken based on the root cause of the problem (as identified by Vitrage).

Note that this blueprint is based on the external-actions blueprint, that handles the more general case.

Examples

- scenario:
   condition: host_down_alarm_on_host
   actions:
    - action:
       action_type: execute_mistral
       properties:
        workflow: wf1

Alternatives

Discussed in the external-actions blueprint.

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee: ifat-afek

Work Items

Implement the Mistral notifier
Update the documentation

Dependencies

None

Testing

The implementation will be covered by unit tests and tempest tests.

Documentation Impact

The new action should be documented

References

None

Add action list panel for entity click action

Thu, 14 Jun 2018 00:00:00

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

http://creativecommons.org/licenses/by/3.0/legalcode

The URL of the launchpad blueprint:

https://blueprints.launchpad.net/vitrage/+spec/add-action-list-panel

Problem description

The Vitrage Dashboard’s Entity Graph provides users with visual convenience. As a result, the cloud administrator or Vitrage users can easily identify the different situations for each entity. However, the Entity Graph is now provided for visual functionality and still has a small range of actions that can be taken in context.

Proposed changes

For each entity in the current entity graph, the user can see information about the entity through a click action. We will add an action list panel that provides multiple actions to the user through existing click actions. Users can click an entity to see a list of available actions from the drop-down menu. The drop-down menu is located at the bottom of the existing Info panel and configures the action list based on a setting file. The user can select one of these action lists and enter the specific parameters required to execute the action through the additional UI.

Examples of actions that can be provided:

Execute Mistral : If the user selects Mistral, a drop-down menu that is included in the new UI displays a list of workflows currently stored in Mistral. When the user selects a workflow from the list, the user can enter various parameter values (for example, Workflow_name, Workflow_input, params). Then ask them to run the workflow from Mistral server to the API. The functional scope of Mistral will be expanded in the future.
Performance Test : The user can perform tests by clicking entities such as Nova, Neutron, and others displayed in the Entity Graph. Like Mistral, the user can view a list of test scenarios in a new UI, enter various parameter values for testing, and then request performance testing through the Openstack Rally API. The range of performance support for this feature will be expanded in the future.
Launch web page of monitoring tool : The web of the monitoring tool monitoring the VM or the physical node entity is displayed.
Open related UI for other projects : If the user requests an action from other project through the action list panel, the results of the action request(e.g. Execute Workflow, Performance Test) can be checked through the UI of other project. Therefore, this action opens the UI related with the action in the other project(e.g. Workflow’s ‘Workflow Executions’ panel from Mistral) in a new tab to see the results of the request.

The overall workflow is as follows:

+-----------------------+
|   Vitrage Dashboard   |
|                       |
| +------------------+  |
| |                  |  |
| |   Entity Click   |  |        +-----------------------+
| |                  |  |        |                       |
| +---------+--------+  |        |    Other Project's    |
|           |           |        |                       |
+-----------------------+        +-----------^-----------+
            |                                |
         (1)|                                |(3)
            |                                |
+-----------v-----------+        +-----------+-----------+
|                       |        |                       |
|   Action list Panel   +-------->  Action Parameter UI  |
|                       |   (2)  |                       |
+-----------+-----------+        +-----------------------+
            |
         (4)|
            |
+-----------v-----------+
|                       |
|  Other Project's UI   |
|                       |
+-----------------------+

Users can click an entity to see a list of actions that can be performed through the drop-down at the bottom of the Info panel. The action list is composed of the settings file configured by the user. This means that the list of actions depends on user’s environment. Also, depending on the entity type, the action may be restricted.
When the user selects an action from the action list, a detailed action list(e.g. Workflow list, Test scenario list) and a UI for inputting parameters are displayed. The user can then select detailed actions and enter parameters in the corresponding UI.
Enter all the parameters in the parameter UI and press the OK button to request it from the other project.(e.g. Mistral, Rally)
The result of the user’s request can be viewed in a new tab with the related project’s UI.

An example configuration file is shown below:

[ACTION_LIST]
mistral = [Mistral Endpoint]
rally = [Rally Endpoint]
monitor_url = [Monitoring Tool URL]

If the user does not enter information for a specific action in the above mentioned setting file, the action list will not include the corresponding action. This determines whether the project is installed to request action. Therefore, the action list is configured based on the setting file, so if the user wants to receive the action, the user should input the information according to the user’s environment.

Alternatives

None

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

When the user clicks an entity in the entity graph, a panel is added to display a list of actions.
Additional actions for the action list can be configured by the user.
When the user selects an action, the UI for entering the required parameter values and selecting detailed actions(e.g. workflow list, test scenario) appears.

Implementation

Assignee(s)

Primary assignee:: MinWookKim <delightwook@dcn.ssu.ac.kr>

Work Items

Add a new panel for entity clicks in the entity graph.
See a list of actions that use the new panel.
The action list can be selected and requested from an other project.
Configure the settings file to organize the action list.

Dependencies

None

Testing

New UI. (action list UI, prameter UI)
Request API through action list panel and check other project action.

Documentation Impact

Configuration for additional action list, usage for adding actions should be documented.

References

https://wiki.openstack.org/wiki/Mistral

https://docs.openstack.org/rally/latest/

Datasource Scaffold

Thu, 17 May 2018 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/datasource-scaffold

A command line tool to generate skeleton of new datasource. A skeleton contains stubs of required classes and methods as described in the design specs, without detail implementation. It aims to bootstrapping the development of new datasource.

Problem description

The design specs has given detail instructions on how to add a new data source. However, there is much overhead to create it from scratch. Developers used to copy from existing datasource as a start. It is sometimes out of date and always contains many specific codes.

Proposed change

Create templates for the datasource skeleton with placeholders of names and render the Python source file on demand.

Example template in Jinja2:

from oslo_config import cfg
from vitrage.common.constants import UpdateMethod

{{ name|upper }}_DATASOURCE = '{{ name }}'

# define needed options
OPTS = [
    # Transformer with the path to your transformer classes
    cfg.StrOpt('transformer',
               default='vitrage.datasources.{{ name }}_datasource.transformer.'
                       '{{ name|capitalize }}Transformer',
               help='{{ name|capitalize }} transformer class path',
               required=True),

Providing name=foo, it will generate the skeleton source file in Python:

from oslo_config import cfg
from vitrage.common.constants import UpdateMethod

FOO_DATASOURCE = 'foo'

# define needed options
OPTS = [
    # Transformer with the path to your transformer classes
    cfg.StrOpt('transformer',
               default='vitrage.datasources.foo_datasource.transformer.'
                       'FooTransformer',
               help='Foo transformer class path',
               required=True),

Alternatives

Create and maintain a sample datasource to allow user to modify as base. In this way, developer is likely to miss some string replacement somewhere as we experienced in the abandoned patch set.

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: yujunz
Other contributors:: None

Work Items

Create datasource skeleton template - driver - transformer
Create unit test skeleton template - test driver - test transformer - mock configuration - mock driver - trace generator
Templates for datasource with different update methods

Dependencies

None

Testing

The changes shall be covered by new unit test.

Documentation Impact

How to use the generator will be documented.

References

Vitrage Alarm Counts API

Fri, 02 Feb 2018 00:00:00

Extend the Vitrage REST API to support a GET of the Active Alarm Counts, for each alarm severity level, in Vitrage.

Problem description

Provide REST API access to the Vitrage Active Alarm Counts in support of the Horizon blueprint, “Vitrage Alarm Banner in Top Navbar”.

Proposed change

Support the following REST API to Vitrage for calculating and returning the Active Alarm Counts for each alarm severity level:

GET /v1/alarm/count

Headers
        X-Auth-Token (string, required) - Keystone auth token
        Accept (string) - application/json

Path Parameters
        None.

Query Parameters
        None.

Request Body
        all_tenants - (boolean, optional) shows the alarm counts
                      summed across all tenants (in case the user
                      has the permissions).

Request Examples
        GET /v1/alarm/count HTTP/1.1
        Host: 135.248.19.18:8999
        X-Auth-Token: 2b8882ba2ec44295bf300aecb2caa4f7
        Accept: application/json

Response Status code
        200 - OK

Response Body
        Returns a JSON object containing the alarm counts for the
        different alarm severities.

Response Examples
        {
          "critical_alarm_count": 1,
          "major_alarm_count": 0,
          "minor_alarm_count": 1,
          "warning_alarm_count": 3
        }

NOTE: The vitrage CLI and client will be updated for this new API. e.g. “vitrage alarm count”

Alternatives

For performance reasons, maintain the Active Alarm Counts in Vitrage Entity Graph, and just return these counts when the REST API command is received.

Although decided against this due to:

keeping a counter, in addition to the graph, might be buggy (multi threading issues etc.),
calculating the counter means traversing once all of the vertices in the graph, get all alarms, and count. It shouldn’t be too expensive, it’s just like ‘get alarms’ api,
since the result of this api is used for ui query (and not for notification or corrective actions for example), the performance is not that critical.

Data model impact

None

REST API impact

Extending REST API with new GET /v1/alarm/count API.

Versioning impact

None … just extending API, no changes.

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

Horizon will use this API to populate the counts in its new “Vitrage Alarm Banner” in its Top Navbar; a proposed Horizon blueprint.

Implementation

Assignee(s)

Primary assignee:: gwaines
Other contributors:: None

Work Items

Implement new REST API in Vitrage API: GET /v1/alarm/count API, to calculate and return the Vitrage Active Alarm Counts for each alarm severity level,
Update Vitrage client for new API
Add the new “vitrage alarm count” CLI command

Dependencies

None

Testing

The changes shall be covered by new unit test and tempest test.

Documentation Impact

Update to Vitrage API Documentation; i.e. the new API will be added under https://github.com/openstack/vitrage/blob/master/doc/source/contributor/vitrage-api.rst

References

None.

Refactor execute-mistral action

Wed, 24 Jan 2018 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/refactor-execute-mistral-definition

The definition of the execute-mistral action should be changed, to better support the integration of Vitrage and Mistral.

Problem description

With the current execute-mistral action definition, it is possible to pass string values to the Mistral workflow, but not dynamic attributes like the id or ip address of the instance that was matched in the template condition. We should enhance the template language to support passing dynamic attributes to Mistral.

Another issue is that the structure of the action definition should be changed. In the current structure, optional input parameters to the workflow appear on the same level as the ‘workflow’ property, which is a mandatory part of the action definition:

- action:
    action_type: execute_mistral
    properties:
      workflow: evacuate_host
      timeout: 10
      force: false

Instead, we should gather all optional parameters (timeout and force) under an ‘input’ section.

Proposed change

The first part of the change is to create a new ‘input’ section, under which all input parameters of the workflow should be placed. A new versioning mechanism should be introduced to Vitrage templates, to allow validating and loading of both the old format and the new format. At the first stage both formats will be supported, but in a version or two we should deprecate the old format.

The second part is to define a way to describe which attribute of a specific entity should be passed to the workflow. The suggested solution is to use a syntax that is similar to the HOT template.

We will introduce a get_attr() function with the following parameters:

resource template_id: the id of the resource inside the template. Note that the resource must be part of the condition.
attribute: the name of the attribute to use. The attribute will be taken from the resource vertex in the graph, if the condition is met.

- scenario:
    condition: host_down_alarm_on_host
    actions:
      - action:
          action_type: execute_mistral
          properties:
            workflow: evacuate_host
            input:
              host_id: get_attr(host, "id")
              host_ip_addr: get_attr(host, "ip_address")
              timeout: 10
              force: false

Alternatives

One alternative is to replace the get_attr with a shorter syntax. In this case, we will refer to an entity attribute by the entity template-id and the attribute name. For example, host.ip_address will mark the ip_address attribute of the instance.

The example above will look like:

- action:
    action_type: execute_mistral
    properties:
      workflow: evacuate_host
      input:
        host_id: host.id
        host_ip_addr: host.ip_address
        timeout: 10
        force: false

This looks much nicer, but has two main disadvantages:

It is not clear, just by looking at the template, that this is a reference to an attribute. What if the user wants to pass a string which is “host.id”?
It is less generic. On the other hand, if we add get_attr as a function syntax, we will be able to add other functions in the future with a similar syntax.

Data model impact

None

REST API impact

None

Versioning impact

The suggested change is not backward-compatible with Pike. Vitrage templates should be enhanced to support versioning, and both the old version and the new one should be supported for now.

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: ifat-afek

Work Items

Support versioning in Vitrage templates. Allow per-version validators and loaders for specific actions.
Move optional input parameters under ‘input’ section
Support get_attr

Dependencies

None

Testing

The implementation will be covered by unit tests and tempest tests.

Documentation Impact

The changes in the action definition should be documented

References

Vitrage template format:

Add CRUD support for template addition

Wed, 24 Jan 2018 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/crud-templates

Adding CRUD support means that templates can be added/removed in real time when Vitrage is up.

Templates added/removed via API will be stored in the database so they remain after restarting Vitrage. sqlalchemy will be used for DB management.

Problem description

Currently, Vitrage templates are loaded from a specific folder during Vitrage startup. Adding/removing templates while Vitrage services are running requires restart of the vitrage-graph service.

Proposed change

Templates should be stored in the database instead of specific folder in file system, In that way they can be modified (add/delete) while vitrage is up.

The evaluator will preform a live update to the entity graph according to actions specified in the added/removed templates.

Template add:

Validate template.
Store template in database.
Notify evaluator.
Entity graph evaluation with new actions.

Delete:

Delete template from database.
Notify evaluator.
Entity graph evaluation to undo templates actions.

Update - not supported at this stage. In order to update template use add and delete template.

A few changes should be made in order to implement CRUD support

Support template Add/Remove commands in vitrageclient (Vitrage API)
API handler: Store/Delete Templates to and from the Database.
Graph cloning logic should be extracted to base class.
Add new DB table called templates

Template DB Table:

+----------------+--------------+------+-----+---------+-------+
| Field          | Type         | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| created_at     | datetime     | YES  |     | NULL    |       |
| updated_at     | datetime     | YES  |     | NULL    |       |
| id             | varchar(64)  | NO   | PRI | NULL    |       |
| status         | varchar(16)  | YES  |     | NULL    |       |
| status_details | varchar(128) | YES  |     | NULL    |       |
| name           | varchar(128) | NO   |     | NULL    |       |
| file_content   | text         | NO   |     | NULL    |       |
| type           | varchar(64)  | YES  |     | NULL    |       |
+----------------+--------------+------+-----+---------+-------+

Alternatives

None

Data model impact

A new database table.

REST API impact

PUT and DELETE methods will be added.

Versioning impact

None

Other end user impact

New CLI commands added:

Vitrage template add

Vitrage template delete

Deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: ikinory
Other contributors:: None

Work Items

API support for add/remove template
implement database table via SQLAlchemy.
implement queries to database.
Tests as explained in “Testing”.

Dependencies

None

Testing

API:

template add:
- add all types of templates : standard, equivalence, definition.
- add corrupted template and check for failed to add.
- add a folder of templates.
template delete:
- check all types of templates : standard, equivalence, definition.
template list.
template show:
- compare cli template content to original file content

e2e:

evaluate the added/ deleted templates on the entire graph.

test evaluator reload templates:

example:

1.raise trigger alarm (template is not loaded yet).

2.add the relevant template.

3.check action is executed.

This checks that the evaluators are reloaded and run on all existing vertices.

Documentation Impact

Template add and delete should be added. Modify template validate and list.

Changes should be added to:

API description: vitrage/doc/source/contributor/vitrage-api.rst

CLI description: doc/source/contributor/cli.rst

References

None

Snmp Parsing Service

Tue, 23 Jan 2018 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/snmp-support

This blueprint describes the implementation of SNMP parsing service for transforming SNMP alarm messages to alarm details and distributing them to corresponding datasource.

Problem description

The following use case should be supported:

Vitrage datasources module provides the ability to handle alarms from part of monitored systems, but currently there is no system that reports alarms by SNMP communication.

Proposed change

A SNMP service module is presented here, which provides service to parse alarms reported from SNMP managed system and sends them to the OpenStack message bus, for further processing by the datasources.

Since snmp service is a common service for alarm datasource, the service powers on just after api, graph and notifier service. After successfully powered on, SNMP parsing service can receive and decode alarm messages. Decoded alarm details are made up of alarm/object info and corresponding value, e.g. alarm_code. The SNMP parsing service parses alarm datasource info according to corresponding OID, then constructs message after marking datasource information and distributes messages to the RabbitMQ queue. According to the values of OID, the alarm datasource can extract information by decoded alarm details.

The configuration for snmp service: [snmp_parsing]

# snmp listening port (integer value) snmp_listening_port = xxx

# traps oid mapping yaml file path(string value) #oid_mapping = /etc/vitrage/snmp_parsing_conf.yaml

An example of config for snmp_parsing_conf.yaml:

oid: 1.3.6.1.4.1.3902.4101.1.3.1.12 # for example system: iaas_platform # for example datasource: new_datasource
oid: xxxx system: xxx datasource: xxx

Alternatives

None

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

None

Developer impact

The snmp parsing service does not support lost notifications at the moment. If one needs the solution in the future, the service should be enhanced.

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: xupeipei

Work Items

Add a new SNMP parsing service

Dependencies

None

Testing

The implementation will be covered by unit tests and tempest tests.

Documentation Impact

The new SNMP configuration should be documented

References

None

Webhooks

Wed, 03 Jan 2018 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/configurable-notifications

This blueprint describes the implementation of Vitrage Notifier for webhooks on Vitrage alarms and state changes.

Problem description

Vitrage should support webhooks for notfications, which are sent on raised alarms, deactivated alarms, state changes, RCA or other to any registered targets. Furthermore any registered recipient should supply a regex to filter the alarms sent to that recipient.

Proposed change

Needed definitions in Vitrage config file:

[DEFAULT]
notifiers = webhook

Vitrage listener will get the alarm events from the message bus and the webhook notifier will send http post notifications on raised deduced alarms and deleted deduced alarms.

The filtered notifications will be sent to the destinations that are written in the database, as configured via API requests.

The notifications will be sent only on alarms which meet the regex filter specified in the webhook specification.

The format of sent notifications will be hard coded.

As Vitrage notifiers are pluggable, you can write your own notifier and use it. Specifically in this case, you can inherit the webhook base class and implement your own webhook notifier.

Alternatives

None

Data model impact

New DB table, to represent registration details Preliminary columns :

ID
Date
Address
Headers
Filter

REST API impact

An API which supports adding, removing and listing webhooks

Versioning impact

None

Other end user impact

None

Deployer impact

To use webhooks one needs to define it in the Vitrage config file.

Developer impact

None

Horizon impact

Future support for webhooks in Horizon

Implementation

Assignee(s)

Primary assignee:: nivo

Work Items

Add DB table
Add API
Implement notifier
Update docs
Tests

Dependencies

None

Testing

This blueprint requires tempest tests and unit tests.

Documentation Impact

The usage of webhooks will be documented

References

Example on http post notifications in AODH http post request

DB Support

Tue, 28 Nov 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/db-support

There is a need in Vitrage for persistent DB. The main reasons are history and high availability, but there are additional uses for persistent DB, such as saving the registered rest notifications recipients, and improving performance using DB capabilities. We will use sqlalchemy for DB management, same as Nova, Heat, Aodh and other projects.

Problem description

Vitrage should support persistent db management.

Proposed change

The development will happen in a few stages. To enable faster development of Vitrage History / HA / Vitrage notifications, in the first stage only basic sqlalchemy support will be developed.

Note : A good example of sqlalchemy implementation can be found in :

https://github.com/openstack/aodh/blob/master/aodh/storage/impl_sqlalchemy.py

Basic configuration in vitrage.conf :

# The SQLAlchemy connection string to use to connect to the database.

# Example:

# connection = mysql://root:pass@127.0.0.1:8999/vitrage

#connection = <None>

# The SQL mode to be used for MySQL sessions. This option, including the

# default, overrides any server-set SQL mode. To use whatever SQL mode is set

# by the server configuration, set this to no value. Example: mysql_sql_mode=

# (string value)

# Additonal modes and their functionality :

# https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sql-mode-combo

#mysql_sql_mode = TRADITIONAL

# Timeout before idle SQL connections are reaped. (integer value)

#idle_timeout = 3600

# Minimum number of SQL connections to keep open in a pool. (integer value)

#min_pool_size = 1

# Maximum number of SQL connections to keep open in a pool. Setting a value of

# 0 indicates no limit. (integer value)

#max_pool_size = 5

# Maximum number of database connection retries during startup. Set to -1 to

# specify an infinite retry count. (integer value)

#max_retries = 10

# Interval between retries of opening a SQL connection. (integer value)

#retry_interval = 10

# If set, use this value for max_overflow with SQLAlchemy. (integer value)

#max_overflow = 50

# Verbosity of SQL debugging information: 0=None, 100=Everything. (integer

# value)

# Minimum value: 0

# Maximum value: 100

#connection_debug = 0

# Add Python stack traces to SQL as comment strings. (boolean value)

#connection_trace = false

# If set, use this value for pool_timeout with SQLAlchemy. (integer value)

#pool_timeout = <None>

# Enable the experimental use of database reconnect on connection lost.

# (boolean value)

#use_db_reconnect = false

# Seconds between retries of a database transaction. (integer value)

#db_retry_interval = 1

# If True, increases the interval between retries of a database operation up to

# db_max_retry_interval. (boolean value)

#db_inc_retry_interval = true

# If db_inc_retry_interval is set, the maximum seconds between retries of a

# database operation. (integer value)

#db_max_retry_interval = 10

# Maximum retries in case of connection error or deadlock error before error is

# raised. Set to -1 to specify an infinite retry count. (integer value)

#db_max_retries = 20

Step 1

Build a basic sqlalchemy implementation for connecting to the DB, Create a Vitrage schema in the db and required tables (from Vitrage history spec and Vitrage httppost registration spec). Each developer should add and test his own tables to the schema.

The schema and tables should be created dynamically from Vitrage, each time Vitrage loads. Vitrage will test existence of the database tables and if they do not exist, create them, using a “model” data representation. Testing validity of existing tables or even fixing them is not required in the first stage.

Cinder Data Model example : https://github.com/openstack/cinder/blob/master/cinder/db/sqlalchemy/models.py

Create DAO that will perform hard coded CRUD on the tables.

Step 2

Upgrade / versioning with alembic.
Generic sql support.
Pagination for big queries, such as all of the entity graph or events table.

Alternatives

Not today.

Data model impact

This whole design is one big Data Model impact.

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Proper Indexes should be applied, and, according to performance testing during development, multiply table data to smaller indexed tables for better data polling performance.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignees: danoffek. alexey_weyl.

Work Items

Create a basic SQLAlchemy implementation to connect to the DB.
Create Data Model for events and other tables.
Create simple CRUD methods for event storage.
CRUD templates
Tests as explained in “Testing”.

Future lifecycle

See “Step 2”

Dependencies

None

Testing

Unit tests:

Data selection queries
Data update
Table create with indexes
(should not run in the gate or regularly) Performance tests on large table data with multiple inserts per seconds over a period of an hour.

Tempest tests: Each developer should create his own DB tempest tests. Example: Create alarms / resources events, and poll the system afterwards. In case the data wasn’t stored in the events table properly, errors should be issued.

Documentation Impact

The DB configuration should be documented in an .rst file, and there should be a link to it from https://github.com/openstack/vitrage/blob/master/doc/source/installation-and-configuration.rst

References

Event Persistor

Tue, 28 Nov 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/event-persistor

The presistor service listens to the RabbitMQ2 (on a different topic) and asynchronously writes the events to a relational database. All events are stored after the filter/enrich phase.

Problem description

In order to support some of the main future use cases of Vitrage, including full HA support, alarm history and RCA history, we will need to store the events that arrive from the collector in a persistent database. Explained in Vitrage HA and History Vision.

Example of a use case for the stored data: Reconstructing the graph from the historic data that controlled by the processor, and will be used in two cases:

Upon failure, in order to initiate the standby vitrage-graph process

For RCA history

Proposed change

Add event table in Vitrage database. Both Datasource driver and Service listener (collector) passes the filtered/enriched events to the persistor via vitrage_persistor topic. Add a Persistor service which listens to vitrage_persistor topic and writes the events to event table.

Alternatives

None

Data model impact

The table will have the following fields:

Field	Description	Examples
id	INTEGER, Auto-Increment	19588
collector_timestamp	The time the event filtered/enriched in the driver	2017-10-09 09:19:50
payload	The enriched event sent from the collector	JSON representation of the event

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: 7mode3294
Other contributors:: None

Work Items

Add new topic vitrage_persistor in rabbitMQ2 for the Persistor.

Add event table in Vitrage database.

Both Datasource driver and Service listener (collector) passes the filtered/enriched events to the persistor via vitrage_persistor topic.

Add a Persistor service which listens to vitrage_persistor topic and writes the events to event table.

Dependencies

None

Testing

The implementation will be covered by additional unit tests and tempest tests.

Documentation Impact

None

References

https://docs.openstack.org/vitrage/latest/contributor/vitrage-ha-and-history-vision.html

Parallel evaluation of Vitrage templates

Tue, 28 Nov 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/parallel-evaluation

Currently Vitrage sequentially processes and evaluates incoming events. While it is a necessity to run sequential processing, template evaluation for a single event is independent and can be paralleled. The challenge is NetworkX in-memory graph held by a single process, thus preventing parallelism, this can be addressed by creating evaluator worker processes that maintain clones of the graph.

Problem description

Each and every graph change triggers evaluation of all relevant template conditions, thus run time of event processing is dependent upon the number of loaded templates.

Processor receives an event, from either the datasource or evaluator queue

Processor updates the graph

Evaluator is triggered to run relevant templates

Evaluation may result in an event writen to the evaluator queue

The above flow runs on in memory NetworkX graph in a single thread

Proposed change

One or more EvaluatorWorker processes will be added to vitrage-graph service. These processes keep their own graph instance. A new component EvaluatorManager manages the communication with the EvaluatorWorkers. Per each graph change resulted in the processor, it will request the EvaluatorManager to inform all the EvaluatorWorkers, so in effect these hold an identical graph clone. Each EvaluatorWorker runs a portion of the templates, writing its results to evaluator queue.

The flow will be as follows:

Processor receives an event, from either the datasource or evaluator queue

Processor updates the graph

EvaluatorManager is triggered, sending the event to the N EvaluatorWorkers, via N multiprocessing queues, then waits for their ack signal

Each EvaluatorWorker updates it’s own graph

In each EvaluatorWorker the evaluator is triggered to run a portion of the templates

Evaluation may result in an event writen to the evaluator queue

Alternatives

None

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

Each EvaluatorWorker holds a clone of the in memory Entity Graph, hence memory consumption will increase as the configured number of workers increases.

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: idan-hefetz
Other contributors:: None

Work Items

processor should not hold a ScenarioEvaluator
create EvaluatorManager
create EvaluatorWorker
change main in graph.py
change GraphService to handle these changes
Choose the best way to assign tasks to workers

Dependencies

None

Testing

The implementation will be covered by additional unit test

Documentation Impact

None

References

None

Add Nova reset-state notifier

Tue, 28 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/vitrage/+spec/add-nova-reset-state-notifier

When a host is marked as down, all the servers which are launched at the host should be with error status. Nova notifier will be extended to reset server state.

Problem description

As Vitrage works as the OPNFV Doctor Inspector component, when Vitrage receives alarm notifications from the Doctor monitor, it should map the physical resources to virtual resources and set their states appropriately. For the host down scenario, currently Vitrage only support to mark host down, the states of the servers which are launched at the host are still ‘Ok’ in Nova. And also in a real scenario, notifying Nova would help the user to get a clear picture of the state of its instances. So Nova notifier should be extended to call ‘reset-state’ API to reset server state.

Proposed change

Reuse ‘mark_down’ action type and set ‘instance’ as ‘action_target’ in Vitrage template which will call Nova api: ‘reset-state’ to set instance state.

Doctor Example

- scenario:
   condition: host_down_alarm_on_host and host_contains_instance and alarm_on_instance
   actions:
    - action:
       action_type: mark_down
       action_target:
         target: instance

Alternatives

None

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

To use the Nova notifier, there is a need to define it in the Vitrage config file, and in addition use the ‘mark_down’ action for instances in Vitrage template.

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: dong wenjuan <dong.wenjuan@zte.com.cn>

Work Items

Implement the ‘mark_down’ action for instances and tests
Modify the host_down_scenario template for calling Nova reset-state

Dependencies

None

Testing

Unit tests and tempest tests need to be added.

Documentation Impact

The usage of the ‘mark_down’ action for instances will be documented.

References

Doctor inspector design guideline Support external actions in Vitrage templates

Include template definitions from an external file

Tue, 28 Nov 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/definition-templates

Define special template files that contain only definitions (entities / relationships and not scenarios), that can then be included within other template files and used there to create new scenarios.

Problem description

A lot of templates were redefining the same entities and relationships for use in their scenarios.

Proposed change

Add a new file (or a set of files) into a “def_templates” directory. These files are the same as templates but do not contain scenarios or an “include” section (only definitions):

metadata:
 name: alarm_on_host_defs
 description: basic def_template example
definitions:
 entities:
  - entity:
     category: ALARM
     type: nagios
     name: host_problem
     template_id: alarm
  - entity:
     category: RESOURCE
     type: nova.host
     template_id: resource
 relationships:
  - relationship:
     source: alarm
     target: resource
     relationship_type: on
     template_id : alarm_on_host

Add an “include” section within templates, which states the name that should be included at it appears in the metadata of the definition template. Multiple definition templates can be added:

metadata:
 name: alarm_on_host_scenario
 description: basic template with an include section example
definitions:
 entities:
  - entity:
    ...
 relationships:
  - relationship:
     ...
include:
 - name: alarm_on_host_defs
 - name: ...
scenarios:
 - scenario:
    condition: alarm_on_host
    actions:
     - action:
        action_type: set_state
        properties:
         state: SUBOPTIMAL
        action_target:
         target: resource11

Alternatives

None

Data model impact

None

REST API impact

None. Should be addressed in a future template CRUD implementation

Versioning impact

None - Old template formats will still be supported. Introduces an alternative version.

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

Template UI should be changed to show definition template files.

Topology view

No impact

RCA view

No impact

Entity graph

No impact

Summary

No impacts

Implementation

Assignee(s)

Primary assignee:: nivolas
Other contributors:: None

Work Items

In scope:

Loading definition templates.
Validating definition templates.
Tests
Documentation

The following items are not in scope:

Definition templates with scenarios.
Recursive includes (a definition template can not include other definition templates).

Dependencies

None

Testing

The implementation will be covered by additional unit tests and tempest tests.

Documentation Impact

Documentation on how to define definition template files and when to use them

References

None

Proactive RCA fault model

Tue, 07 Nov 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/proactive-rca

This blueprint proposes a solution for proactive RCA. It aims to be an umbrella for all related blueprints.

Problem description

Currently, Vitrage relies on the monitored events for root cause analysis. It may deduce alarms on virtual resources, but it is always in a confirmed state. But in the real world, things could be more complicated. Suppose there are two possible causes (A or B) for fault C. When fault C is monitored, it is suspicious that A or B could happen and be the root cause. We need a way to take action in such case to find the root cause more proactively.

Proposed change

A fault model with deductive reasoning is proposed here to resolve the problem above.

Fault model

Given the template defined for the fault model above.

The underlying entity graph could be breakdown as

Deductive reasoning

Suppose we received a series of events in the following order:

t0: initial status, no faults
t1: fault C active monitored
t2: fault A active and fault B inactive is reported from diagnose action
t3: fault C inactive monitored
t4: fault A inactive returned from diagnostic action

Let’s illustrate the deductive reasoning in graphs with following legend.

cluster: aggregate fault
nodes: raw fault
edges:
- dashed: the target state is deduced from source state by Vitrage
- dotted: the target state is updated by diagnose action triggered by source state
border:
- dashed: suspect
colors:
- lightgrey: fault in undefined state
- red: fault in confirmed state
labels:
- Fault A: aggregated
- a_v: deduced by Vitrage
- a_m: monitored

t0: initial status

All faults undefined, no reasoning

t1: downstream fault monitored, upstream fault deduced

Vitrage deduces that fault A and fault B are active in suspect state
execute diagnose action to confirm suspect state

t2: diagnose action executed, upstream fault confirmed

monitored states updated by diagnose action
as a side-effect, Vitrage will deduce that fault C is active in parallel of monitored fault C

t3: downstream recovery monitored, upstream recovery deduced

from the fault model, we can deduce fault A and fault B are inactive
execute diagnose action to resolve status inconsistency between fault A deduced (inactive) and monitored (active)

t4: upstream recovery confirmed

Changes required

The main change would be allowing raise alarms in suspect status which can be used to trigger a diagnostic action.

Add support for diagnose action, like:
- an immediate pull from data source
- force a push from data source
- launch external tools to get state and post events to Vitrage API
Suspect states for alarm
- for deduced alarm, it is suspect if the source is from downstream
- for aggregated alarm, it is suspect if inconsistency is detected among underlying alarms (deduced and monitored)

Specially, it is not suspect if newer monitored state is different from old suspect deduced state. Because a suspect state means it could be either active or inactive. So it is reasonable to trust the latest update from monitor.

However, when we bring in proactive RCA, the entity numbers in the graph may grow a lot. We shall need to create deduced alarm for each monitored alarm and set suspect state in some condition. The relationships (edges) will also grow. So there are some additional work to be done to improve user experience, such as:

Aggregate underlying entity graph to simplify user view
Add new API for querying aggregated entity graph
Simplify the template definition by defining fault model

Backward compatibility

A fully functional proactive RCA fault model requires every fault can be monitored with a diagnose action. But the deductive reasoning procedure is also applicable for fault model lacking diagnose action or missing monitoring on some fault.

For example, if there is no diagnose action for confirming Fault A, the deductive reasoning will still continue once the status of Fault A get updated passively from monitor.

If there is no monitor for Fault A, then it will stay in suspect status. It helps the user to narrow down the scope of root cause to make manual investigation easier.

Implementation

Assignee(s)

Primary assignee:: yujunz
Other contributors:: no

Work Items

See dependent blueprints.

Dependencies

The implementation of proactive RCA depends on several blueprints. They will be listed below once got approved.

Testing

See dependent blueprints.

Documentation Impact

See dependent blueprints.

References

Support External Actions in Vitrage Templates

Wed, 16 Aug 2017 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/support-external-actions

Currently, the Vitrage templates support the following actions: raise alarm, set state, mark causal relationship and mark-down. The implementation of mark-down is patchy (update a property on the vertex and call the notifier).

We need a way to define in the template an external action that should be taken. One example is the mark-down. Another example is to execute a Mistral workflow or a Congress policy.

Problem description

The following use cases should be supported:

mark-down. This is an existing use case. If a mark-down action exists, Vitrage will call Nova mark-down API. The behavior should remain the same after the change.
Execute a Mistral workflow. The user should be able to execute a Mistral workflow based on Vitrage insights.
Execute other external actions, like a Congress policy.

Proposed change

Add a new action for each external engine we would like to support. The action will have metadata with parameters to be sent to the engine.

Examples

action:
  action_type: execute_nova
  metadata:
    api_call: mark-down
    host: host1

action:
  action_type: execute_mistral
  metadata:
    workflow: evacuate_host
    failed_host: host1

The idea behind adding a new external_* action for each engine is to emphasize that Vitrage supports a predefined set of external actions, and specifically it does not support runinng scripts.

The internal implementation will be identical for Nova, Mistral, Congress, etc. During the template loading phase, the execute_* action will be converted to a generic Execute action, with a notifier parameter. The template validator will verify that the wanted notifier is enabled in vitrage.conf, otherwise the template loading will fail.

When executing the Execute action, the evaluator will send an event to the message bus of the wanted notifier. This way, only the Mistral notifier will handle the execute_mistral actions.

Note that currently only the Vitrage graph sends notification to the notifier. This behavior will be changed, so notifications will be sent both from the graph (for deduced alarms and set state) and from the evaluator.

Alternatives

There are two alternatives:

Send Message bus notifications

Vitrage can have a message bus notifier, that will send notifications about raise/delete alarm, set state and mark causal-relationship. Mistral will be configured to execute certain workflows based on the notifications received from Vitrage.

Advantages:

No extra configuration needed in Vitrage
Can work with the current notifiers mechanism

Disadvantages:

Messgae bus notifications might get lost
Less powerful. In Vitrage template the user can decide to execute a workflow based on a combination of alarms or a specific topology. Once Mistral gets the notifications, the topology context is no longer there.
mark-down use case is not supported by this solution

Configuration Done on the specific notifier

A Mistral notifier will be written and receive notifications from the graph. It will be configured by a yaml file that will determine which workflow to execute per specific alarm.

Advantages:

No extra configuration needed in Vitrage
Can work with the current notifiers mechanism
No lost messages

Disadvantages:

Less powerful. In Vitrage template the user can decide to execute a workflow based on a combination of alarms or a specific topology. In Mistral notifier we no longer have this context.
mark-down use case is not supported by this solution

Calculate the action_target

Every “regular” action (raise_alarm, set_state, add_causal_relationship) has an action_target block. The action_target has no meaning for the external actions, since they are managed by an external engine that does not have a “target” terminology.

The problem is that the action_target is essential for the subgraph-matching calculations, in order to calculate the connected components.

The proposed solution

For external actions, we will automatically calculate an action target. The target will be selected arbitrarily from the set of possible targets. In complex conditions that involve {and, or, not} finding the target might not be trivial. Some more-complex conditions will not be supported for external actions, due to this reason.

The calculation method:

And condition - any vertex that is part of the condition can be a target
Not condition - no vertex that is part of the condition can be a target
Or condition - the target should be a vertex that appears in any “positive” part (i.e. one that does not have a ‘not’ in front of it) of the Or condition

Examples:

Note: In some cases it is impossible to find a valid target. They are marked with ‘/’.

The template validation should fail in these cases, for all kinds of actions.

Condition	Possible targets
a_in_error_status	a
a_contains_b	a, b
a_contains_b or a_contains_c	a
a_contains_b or a_contains_c or a_contains_d	a
a_in_error_status or a_contains_c or a_contains_d	a
a_contains_b or a_contains_c or b_contains_d	Not Supported
a_contains_b and a_contains_c	a, b, c
a_contains_b and a_contains_c and a_contains_d	a, b, c, d
a_contains_b or (a_contains_c and a_contains_d)	a
a_contains_b or (a_contains_c and b_contains_d)	a,b
not a_contains_b	Not Supported
not a_in_error_status	Not Supported
a_contains_b or not a_contains_c	Not Supported
a_contains_b or (a_contains_c and not a_contains_d)	a
a_contains_b or (not a_contains_c and not a_contains_d)	Not Supported

Alternatives to calculating the action_target

We can decide that an external action requires an action_target like all other actions. This will simplify the implementation, but is logically wrong. The action_target will serve as a “helper entity” for the subgraph-matching, and it should not be the end user’s role to help fixing implementation issues. As far as the external engine is concerned, the action_target has no meaning.

Data model impact

A new Execute action will be added to the evaluator.

REST API impact

None

Versioning impact

We should introduce a versioning mechanism to the templates. This will be done when modifying the implementation of mark-down.

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: ifat-afek

Work Items

Enhance the template language (template loading and validation)
Update the documentation
Execute the external actions from the evaluator

Dependencies

None

Testing

The implementation will be covered by unit tests and tempest tests.

Documentation Impact

The new action should be documented

References

None

OSProfiler Support

Wed, 16 Aug 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/osprofiler-support

OSProfiler is an OpenStack cross-project profiling library. It allows user to generate 1 trace per request which is processed in multiple services.

Problem description

It is quite common that Vitrage is integrated in a large system to provide a complete solution e.g. as inspector in OPNFV/Doctor for fault management.

When performance is a critical issue, it is very complicated to analyse the event processing workflow and locate the bottleneck in case something works slowly.

Proposed change

Add osprofiler support in vitrage and vitrage-client.

OSProfiler will help to generate a tree of calls (see example) which is intuitive to understand what exactly is going on.

Alternatives

Explained in why not cprofile and etc

Data model impact

None

REST API impact

Additional HTTP header inserted by profiler should be checked when it is enabled. Besides that, there is no impact on the business logic in api handler.

Versioning impact

None

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: yujunz
Other contributors:: None

Work Items

add configuration options for osprofiler
add initialization of osprofiler in service startup
add osprofiler wsgi middleware to trace HTTP calls
trace RPC calls

Refer to the changes proposed in similar topic in ironic for an overview.

Dependencies

None

Testing

Covered by unit test.

Documentation Impact

Add to developer guide on how to use osprofiler.

References

Keycloak support

Sun, 06 Aug 2017 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/keycloak-support

As part of an on going effort to make vitrage to be able to work also in a non OpenStack environment (in addition to the default OpenStack environment). We should be able to make vitrage work with a different authorization server instead of keystone. An optional authorization server can be Keycloak which is an open source Identity and Access Management solution aimed at modern applications and services

Problem description

Vitrage at the moment can only work in an OpenStack environment because it needs Keystone for authorization. We should support other authorization such as Keycloak.

Proposed change

New auth_mode in api section in Vitrage config file:

[api]
auth_mode = keycloak

New keycloak section with the auth_url in Vitrage config:

[keycloak]
auth_url = http://[keycloak server]:[keycloak port]/auth

The Vitrage server will use a new middleware which will authenticate with the Keycloak server once an api request is received.

A new auth plugin will be added to the vitrage client which will get the token from the Keycloak server and sent it with the api request.

Alternatives

None

Data model impact

None

REST API impact

When using the client we should use the keycloak-plugin

Versioning impact

None

Other end user impact

None

Deployer impact

To use the Keycloak Authorization there is a need to define it in the Vitrage config file.

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: eyalb1

Work Items

Create Keycloak plugin in client
Create Keycloak plugin in server

Dependencies

None

Testing

This blueprint requires unit tests.

Documentation Impact

The usage of the KeyCloak authorization will be documented

References

keycloak-config.rst

Define and handle equivalence among entities

Wed, 19 Jul 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/entity-equivalence

Define equivalence among entities to allow mapping two or more entities to same object and handle it properly in scenario evaluator. It is inspired by Idan Hefetz’s proposal about ZTE use case of alarm deduction, but designed in a generic way to allow extending to RESOURCE equivalence in future.

Problem description

Introducing entity equivalence will enhance the extensibility of Vitrage, making it suitable for more use cases, such as

early deduction of alarm before it is reported and deal with the real alarm followed
aggregation of equivalent alarms from multiple monitors
aggregation of resource information from multiple data sources.

Proposed change

Add a new file (or a set of files) that define equivalence between entities

metadata:
 name: entity equivalence example
equivalences:
 - equivalence:
    - entity:
       category: ALARM
       type: nagios
       name: host_problem
    - entity:
       category: ALARM
       type: zabbix
       name: host_problem
    - entity:
       category: ALARM
       type: vitrage
       name: host_problem
  - equivalence:
    ...

These definitions will take effect globally, i.e. for every other template

The evaluator will duplicate every scenario for every equivalent alarm automatically. For example, in case of the condition:

condition: nagios_host_problem_on_host and host_contains_vm

Two conditions will be created internally:

condition: nagios_host_problem_on_host and host_contains_vm
condition: zabbix_host_problem_on_host and host_contains_vm
condition: vitrage_host_problem_on_host and host_contains_vm

The idea is that the user will write a single condition, and all equivalent conditions will be created and evaluated automatically.

Equivalences should be defined explicitly. Including one entity in two or more equivalence definition will result in implicit chaining, thus is considered invalid. For example, if a eq b and b eq c are defined separately, it will logically result in an implicit a eq c. This will introduce unnecessary complexity in creating templates and should be restricted in validator.

Alternatives

Separate file vs embedded definition

Instead of creating a separate file, we may embed the equivalence definitions in templates by adding a new section equivalences. Entities that are equivalent to each other are grouped in arrays of their template_id

metadata:
 name: entity equivalence example
definitions:
 entities:
  - entity:
     category: ALARM
     type: nagios
     name: host_problem
     template_id: nagios_host_problem
  - entity:
     category: ALARM
     type: zabbix
     name: host_problem
     template_id: zabbix_host_problem
  - entity:
     category: ALARM
     type: vitrage
     name: host_problem
     template_id: vitrage_host_problem
  ...
 relationships:
  ...
 equivalences:
  - [nagios_host_problem, zabbix_host_problem, vitrage_host_problem]
scenarios:
  ...

In this way, there will be fewer duplication of entity definitions.

However, given the fact that once an equivalent edge is added between two alarms, then it logically means that they are equivalent in all other templates as well. Even if they are not specified this way in the other templates. Then template will be less clear without the equivalence information embedded in it.

The duplication of entity definition might be resolved by implementing an import feature in other blueprint.

Adding equivalent edge vs not

equivalent edges could be created between every two equivalent alarms. Since all related scenarios have been duplicated, This does not bring extra value in the evaluator.

The equivalent edge could be useful for future evolution such as alarm aggregation, UI optimization, alarm deduction. It may be implemented in those blueprints.

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

There are currently three views in vitrage-dashboard

Topology view

No impact

RCA view

More alarms and more causes edges

Todo

(yujunz) include example graph

Entity graph

separate vertices for equivalent alarms (nagios, zabbix, vitrage)
more edges (equivalent and on)

Summary

The impacts on RCA view and Entity graph will only be relevant to cases where both equivalence and vitrage-dashboard are used. We will handle it in future blueprints.

Implementation

Assignee(s)

Primary assignee:: yujunz
Other contributors:: None

Work Items

validate and parse equivalence definition in templates
duplicate scenarios in the scenario repository
no changes in sub-graph matching or the evaluator

The following items are not in scope

aggregation of equivalent alarms
add-equivalent action
support alarm equivalence in UI
implement causal tree model for alarm deduction enhancement
resource equivalence

Dependencies

None

Testing

The implementation will be covered by additional unit test

Documentation Impact

documentation on how to define equivalence and when to use it
declare limitation on resource equivalence
list known issues when use equivalence with vitrage-dashboard

References

Resource List API

Wed, 19 Jul 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/resource-list-api

An API to list the resources with specified type or all the resources.

Problem description

Currently Vitrage has the APIs for getting topology and alarms. But user may want to get specified resources which he cares about.

Proposed change

Add an API to list resources. If user specify the resource type, list the resources with the given type.

Alternatives

None

Data model impact

None

REST API impact

Resource List

Returns resource list

GET /v1/resources/

Headers

X-Auth-Token (string, required) - Keystone auth token
Accept (string) - application/json
User-Agent (String)

Path Parameters

None.

Query Parameters

None

Request Body

resource_type - (string, optional) the type of resource. defaults to return all resources.
all_tenants - (boolean, optional) shows the resources of all tenants (in case the user has the permissions).

Request Examples

GET /v1/resources/
Host: 127.0.0.1:8999
User-Agent: keystoneauth1/2.3.0 python-requests/2.9.1 CPython/2.7.6
Accept: application/json
X-Auth-Token: 2b8882ba2ec44295bf300aecb2caa4f7

Response

Status code

200 - OK
400 - Bad request

Response Body

Returns a list with all the resources requested.

Response Examples

[
  {
    "category": "RESOURCE",
    "is_placeholder": false,
    "is_deleted": false,
    "name": "vm-1",
    "update_timestamp": "2015-12-01T12:46:41Z",
    "state": "ACTIVE",
    "project_id": "0683517e1e354d2ba25cba6937f44e79",
    "type": "nova.instance",
    "id": "dc35fa2f-4515-1653-ef6b-03b471bb395b",
    "vitrage_id": "RESOURCE:nova.instance:dc35fa2f-4515-1653-ef6b-03b471bb395b"
  }
]

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

dong wenjuan <dong.wenjuan@zte.com.cn>

Work Items

Implement the API and tests
Implement the client and tests

Future lifecycle

None

Dependencies

None

Testing

Unit tests and tempest tests need to be added.

Documentation Impact

The new api should be documented

References

None

Resource show API

Wed, 19 Jul 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/resource-show-api

An API to show the details of the specified resource.

Problem description

As a user, I want to get the details of a specified resource.

Proposed change

Add an API to show the details of specified resource.

Alternatives

None

Data model impact

None

REST API impact

Resource show

Returns details of the resource

GET /v1/resources/vitrage_id

Headers

X-Auth-Token (string, required) - Keystone auth token
Accept (string) - application/json
User-Agent (String)

Path Parameters

vitrage_id

Query Parameters

None

Request Body

None

Request Examples

GET /v1/resources/`<vitrage_id>`
Host: 127.0.0.1:8999
User-Agent: keystoneauth1/2.3.0 python-requests/2.9.1 CPython/2.7.6
Accept: application/json
X-Auth-Token: 2b8882ba2ec44295bf300aecb2caa4f7

Response

Status code

200 - OK
404 - Not Found

Response Body

Returns details of the requested resource.

Response Examples

{
  "category": "RESOURCE",
  "is_placeholder": false,
  "is_deleted": false,
  "name": "vm-1",
  "update_timestamp": "2015-12-01T12:46:41Z",
  "state": "ACTIVE",
  "project_id": "0683517e1e354d2ba25cba6937f44e79",
  "type": "nova.instance",
  "id": "dc35fa2f-4515-1653-ef6b-03b471bb395b",
  "vitrage_id": "RESOURCE:nova.instance:dc35fa2f-4515-1653-ef6b-03b471bb395b"
}

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

dong wenjuan <dong.wenjuan@zte.com.cn>

Work Items

Implement the API and tests
Implement the client and tests

Future lifecycle

None

Dependencies

None

Testing

Unit tests and tempest tests need to be added.

Documentation Impact

The new api should be documented

References

None

SNMP Notifications

Wed, 19 Jul 2017 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/snmp-notifications

The Evaluator performs root cause analysis on the Vitrage Graph and may determine that an alarm should be created, deleted or otherwise updated. Other components are notified of such changes by the Vitrage Notifier service. Among others, one Vitrage Notifier is responsible for sending SNMP traps on Vitrage deduced alarms.

This blueprint describes the implementation of Vitrage Notifier for notifying SNMP on Vitrage deduced alarms.

Problem description

Vitrage should support registering for SNMP notifications, and sending traps on raised alarms and deactivated alarms to any registered targets.

Proposed change

Due to definition in Vitrage config file:

[DEFAULT]
notifiers = snmp

Vitrage listener will get the alarm events from the message bus and the SNMP notifier will send SNMP traps on raised deduced alarms and deleted deduced alarms.

The traps will be sent to destinations specified in consumers yaml file.

The traps will be sent only on alarms specified in yaml file which contains oid mapping for each alarm name.

The format of sent traps will be specified in another yaml file.

All those yaml files’ paths should be specified in Vitrage config:

[snmp]
consumers = <path to consumers yaml file>
alarm_oid_mapping = <path to alarm oid mapping yaml file>
oid_tree = <path to tree format oid configuration yaml file>

The SNMP notifier is pluggable, you can implement your own SNMP sender and use it (it must inherit from the base class), when there is a default implementation.

Alternatives

None

Data model impact

None

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

To use the SNMP notifier there is a need to define it in the Vitrage config file, and in addition create three yaml files and define them in Vitrage config file.

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: annarez

Work Items

Create SNMP notifier
- Create SNMP sender
  - create base class
Create unit test for SNMP sender
- test snmp notifier
- test snmp sender

Dependencies

None

Testing

This blueprint requires unit tests.

Documentation Impact

The usage of the SNMP notifier will be documented

References

notifier-snmp-plugin.rst

Template not operator Support

Wed, 19 Jul 2017 00:00:00

We would like the templates language to support the “not” operator in addition to “and” and “or” operators.

Problem description

Today the templates language support the “and” and “or” operators, but this is not enough. Many scenarios can’t be described by those two operators, and thus we would like to add a support for the “not” operator.

Proposed change

Remark: positive vertex = vertex that has an edge with “is_deleted”=False property.

negative vertex = the opposite of positive vertex. Meaning that the: vertex has only edges with “is_deleted”=True property.

The following changes needs to be done to support the “not” operator.

In the template validation. Check that the “Not” operator can appear only in the following way:

“Not” operator can appear before edges.
Scenario Evaluator:

check if “not” operator appears on an element, and add a property named “negative_condition” on it, and updated it’s “is_delete” property to True.
Networkx Algorithm:

In the subgraph_matching method. If the match is not a “negative condition” then perform regular subgraph_matching. Otherwise, if the match is a “negative condition” then perform subgraph_matching on a negative edges.

Performance/Scalability Impacts

The performance of the subgraph matching algorithm is a bit slower due to the steps above.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: alexey_weyl <alexey.weyl@nokia.com>

Work Items

None

Future lifecycle

None

Dependencies

None

Testing

Added tests that are checking different uses of the “not” operator.

Documentation Impact

Added documentation in Vitrage:: https://github.com/openstack/vitrage/blob/master/doc/source/not_operator_support.rst

References

None

Event API

Mon, 16 Jan 2017 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/support-inspector-sb-api

This blueprint defines an API for sending events to Vitrage datasources.

Problem description

As a reference implementation to OPNFV Doctor Inspector project, Vitrage has to implement the Inspector SB API. This is a REST API that each monitoring service should call in order to push events to the Inspector.

Proposed change

New API will be added to allow sending events to the Vitrage datasources. The events will be sent to Oslo message bus, and will be consumed by the one of the datasource drivers according to the type property.

For example, the Doctor datasource will handle events of type compute.host.down, while the Nova instance datasource will handle events of type compute.instance.delete.start. (Using this API for Nova is not a real use case, but can be used for debugging).

For the Doctor datasource, the events will contain the details defined in the Doctor specification. Future datasources may also use this API in order to send their own events to Vitrage.

REST API impact

POST /v1.0/event/

Post an event to Vitrage message queue, to be consumed by a datasource driver.

Headers

X-Auth-Token (string, required) - Keystone auth token
Accept (string) - application/json
User-Agent (String)
Content-Type (String): application/json

Path Parameters

None.

Query Parameters

None.

Request Body

An event to be posted. Will contain the following fields:

time: a timestamp of the event. In case of a monitor event, should specify when the fault has occurred.
type: the type of the event.
details: a key-value map of metadata.

A list of some potential details, copied from the Doctor SB API reference:

hostname: the hostname on which the event occurred.
source: the display name of reporter of this event. This is not limited to monitor, other entity can be specified such as ‘KVM’.
cause: description of the cause of this event which could be different from the type of this event.
severity: the severity of this event set by the monitor.
status: the status of target object in which error occurred.
monitorID: the ID of the monitor sending this event.
monitorEventID: the ID of the event in the monitor. This can be used by operator while tracking the monitor log.
relatedTo: the array of IDs which related to this event.

Request Examples

POST /v1/event/
Host: 135.248.18.122:8999
User-Agent: keystoneauth1/2.3.0 python-requests/2.9.1 CPython/2.7.6
Content-Type: application/json
Accept: application/json
X-Auth-Token: 2b8882ba2ec44295bf300aecb2caa4f7

{
    'event': {
        'time': '2016-04-12T08:00:00',
        'type': 'compute.host.down',
        'details': {
            'hostname': 'compute-1',
            'source': 'sample_monitor',
            'cause': 'link-down',
            'severity': 'critical',
            'status': 'down',
            'monitor_id': 'monitor-1',
            'monitor_event_id': '123',
        }
    }
}

Response

Status code

200 - OK
400 - Bad request

Response Body

Returns an empty response body if the request was OK. Otherwise returns a detailed error message (e.g. ‘missing time parameter’).

Implementation

Assignee(s)

Primary assignee:: ifat-afek

Testing

The changes will be tested by unit tests and tempest tests.

Documentation Impact

The new api should be documented

References

Collectd Data Source

Tue, 27 Dec 2016 00:00:00

This work is licensed under a Creative Commons Attribution 3.0 Unported

License.

http://creativecommons.org/licenses/by/3.0/legalcode

https://blueprints.launchpad.net/vitrage/+spec/collectd-datasource

This blueprint describes the datasource that will receive notifications from collectd.

Problem description

Vitrage should be able to accept a collectd notification.

Proposed change

The Collectd datasource will receive notifications in the following format:

{
    "host": "compute-1",
    "plugin": "ovs_events",
    "plugin_instance": "br-ex",
    "type": "gauge",
    "type_instance": "link_status",
    "message": "link state of "br-ex" interface has been changed to "WARNING,"",
    "severity": "WARNING",
    "time": 1482409029.062524,
    "id": "46c7eba7753efb0e6f6a8de24c949c52"
}

Upon receiving such a notification, the Collectd datasource will create a corresponding alarm in Vitrage. When receiving an ok notification, the alarm will be deleted.

In addition, a new evaluator template will be added in order to:

Create deduced alarms on the VMs running on the host
Modify the states of the host and the VMs to ERROR

Alternatives

None

Data model impact

None

REST API impact

None

Implementation

Assignee(s)

Primary assignee:: eyal bar ilan

Work Items

Implement the Collectd datasource
Write a template for creating deduced alarms on the VMs and calling Nova mark host down

Testing

The changes will be tested by unit tests

References

https://collectd.org/

Doctor Data Source

Sun, 25 Dec 2016 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/doctor-datasource

This blueprint describes the datasource that will receive notifications from OPNFV Doctor monitor

Problem description

In order for Vitrage to be accepted as a reference implementation for the OPNFV Doctor Inspector component, it should be able to receive alarm notifications from the Doctor monitor.

Proposed change

The Doctor datasource will receive notifications in the format defined by Doctor SB API (see the reference below):

{
    'event': {
        'time': '2016-04-12T08:00:00',
        'type': 'compute.host.down',
        'details': {
            'hostname': 'compute-1',
            'source': 'sample_monitor',
            'cause': 'link-down',
            'severity': 'critical',
            'status': 'down',
            'monitor_id': 'monitor-1',
            'monitor_event_id': '123',
        }
    }
}

Upon receiving such a notification, the Doctor datasource will create or delete a corresponding alarm in Vitrage, based on the ‘status’ field.

In addition, a new evaluator template will be added in order to:

Create deduced alarms on the VMs running on the host
Modify the states of the host and the VMs to ERROR
Call Nova force-down API to mark that the host is down

REST API impact

The Doctor monitor sends its alarms in REST format. Another blueprint discusses the SB API that should be added to Vitrage in order to support it. See https://blueprints.launchpad.net/vitrage/+spec/support-inspector-sb-api

Implementation

Assignee(s)

Primary assignee:: ifat-afek

Work Items

Implement the Doctor datasource
Write a template for creating deduced alarms on the VMs and calling Nova mark host down

Testing

The changes will be tested by unit tests, and later on also by Doctor test scripts.

References

Vitrage Entity Graph Consistency Validator

Thu, 22 Dec 2016 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/entity-graph-consistency-validator

The Entity Graph may have mistakes, such as an incorrect resource state, and we would like to detect and repair such errors.

Problem description

The Entity Graph is a living graph which is updated all the time with the data received from the synchronizer(s). The Entity Graph may have errors and become inconsistent compared to the state of the Cloud, which may occur due to: (1) The Vitrage Resource Processor may perform incorrect operations during graph updates, for example due to problems resulting from multi-threading. (2) Missing updates from the synchronizer(s).

Proposed change

We would like to check the Entity Graph each set interval (configurable) and validate the consistency of its resources and state. If the Entity Graph is incorrect, it will repair it. When needed, it might consult the synchronizer(s) to retrieve missing data.

Alternatives

None

REST API impact

None

Implementation

Assignee(s)

Primary assignee:: alexey_weyl <alexey.weyl@alcatel-lucent.com>

NetworkX Performance Improvement

Thu, 22 Dec 2016 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/networkx-performance-improvement

Many CRUD operations can be performed on the Entity Graph simultaneously. This can cause inconsistency regarding the resources and states on the graph.

Problem description

Many CRUD operations can be performed on the Entity Graph simultaneously. This can cause inconsistency regarding the resources and states on the graph.

Proposed change

We would like to have graph versioning, which will held two copies of the Entity Graph: (1) Stable copy of the last consistent graph, and (2) Dynamic copy of the graph which is updated by the Vitrage Resource Processor. Every set interval the changes inserted to the dynamic graph will be updated to the stable graph, resulting in the stable graph being up to date.

Implementation

Assignee(s)

Primary assignee:: alexey_weyl <alexey.weyl@alcatel-lucent.com>

Vitrage System Health Visualization

Thu, 22 Dec 2016 00:00:00

https://blueprints.launchpad.net/vitrage-dashboard/+spec/ui-system-health-sunburst

A macro view to show the general health of the system.

Problem description

Missing visualization of the system status.

Proposed change

We will present the system health in sunburest graph with color and labels that represnt their status

Implementation

Assignee(s)

Primary assignee:: alon-heller <alon.heller@alcatel-lucent.com>

Vitrage Resource Processor

Thu, 22 Dec 2016 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/vitrage-resource-processor

When Vitrage initializes we need to create the Entity Graph on which the Vitrage will run it’s algorithms (sub graph matching, BFS, DFS etc.) and perform the actions (RCA, deduced alarms etc.). After the initialization of the graph, the resources changes are being processed and pushed to the Entity Graph.

Problem description

Vitrage does not have a full Entity Graph of the resources and their state when it initializes. Vitrage needs to be consistent with the updated and new resources.

Proposed change

When Vitrage initializes we need to build the up to date entity graph, so we can run the different algorithms on it and perform the required actions. To perform we require the full resources list from the synchronizer. In order for the Entity Graph to be constructed, it will need updates on changes in the cloud (e.g., machine creation/termination), and guidance on how resources should be linked to each other (e.g., a virtual machine resides on a physical host, which in turn belongs to an availability zone, etc.). For this purpose, each synchronizer will also update a configuration file - the entity graph template - which specifies for each resource what and how to locate the specific resources to link to.

Implementation

Assignee(s)

Primary assignee:: alexey_weyl <alexey.weyl@alcatel-lucent.com> dan-ofek <dan.ofek@alcatel-lucent.com>

Template Validator

Thu, 22 Dec 2016 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/vitrage/+spec/template-validator

Vitrage Evaluator serves as workflow manager controlling the analysis and activation of templates and execution of template actions.

Template Validator ensures that a new template is correct. Meaning, it conforms with Vitrage Template Standard.

Problem description

Templates do not always meet the Vitrage Template Standard. For example, unsupported action, invalid alarm name, incorrect graph template and etc.

Proposed change

Template validator is a part of Vitrage Evaluator. It receives a template, runs over it and checks its correctness. If the template is valid, it notify the Evaluator Engine which inserts the template into the template DB. Otherwise, insertion is failed.

Alternatives

A template editor that prevent invalid templates.

Data model impact

None

REST API impact

The validator runs when a new template is added through API create call.

Nagios Synchronizer Plugin - get_all implementation

Sun, 27 Nov 2016 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/synchronizer-nagios-get-all

This blueprint describes the Nagios plugin for Vitrage Synchronizer, and its implementation for get_all nagios services (tests).

Problem description

Nagios test results should be added to Vitrage Graph via Vitrage Synchronizer. This requires writing a Synchronizer plugin for Nagios.

The plugin should support two modes:

get_all: query the results of all Nagios tests and send corresponding events to the Synchronizer
notifications: notify the Synchronizer upon a change in one of Nagios test results

This blueprint refers to get_all implementation.

As a first stage, we will support only Nagios 3.

Proposed change

The current Nagios plugin will support Nagios 3.x.

Nagios plugin will be configured with:

Nagios URI, for example: http://10.45.1.10/monitoring/nagios/cgi-bin/status.cgi?host=all&limit=0
Nagios credentials
Poll interval in seconds (default: 60)

Every poll-interval seconds, Nagios plugin will call Nagios URI to query the results of Nagios services (tests). It will parse the returned html (there is no REST API for Nagios 3.x), create Nagios events and send them to Vitrage Graph queue.

Alternatives

None

Data model impact

Nagios event will look like that:

Field	Description	Examples
resource_name	name of Nagios host	compute-0-0.local os-glance-00.local ilo.node14
resource_type	type of Nagios host	nova.host nova.instance switch
service	name of Nagios service (test)	CPU load check_ceph_health
status	the status of the service	OK WARNING CRITICAL UNKNOWN
last_check	last time the service was checked	2016-01-04 19:17:10
duration	duration since the last status change	1d 2h 55m 48s
attempt	how many attempts were made	1/3
status_info	additional information	OK - 15min load 1.66 at 32 CPUs
vitrage_entity_type	the source of information	nagios (constant value)

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

Nagios plugin should be configured with Nagios URI, credentials and poll interval.

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: ifat-afek

Work Items

None

Dependencies

None

Testing

This blueprint requires unit tests and Tempest tests.

Documentation Impact

Nagios Configuration should be documented

References

Nagios Configuration: https://github.com/openstack/vitrage/blob/master/doc/source/nagios-config.rst

Synchronizer main blueprint: https://github.com/openstack/vitrage-specs/blob/master/specs/mitaka/vitrage-synchronizer.rst

Heat Datasource Driver - get_all implementation

Thu, 21 Jul 2016 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/heat-datasource-get-all

This blueprint describes the Heat driver for Vitrage Datasource, and its implementation for get_all Stacks and Stack Resources.

Problem description

Heat stacks should be added to Vitrage Graph via Vitrage Datasources. This requires writing a Datasource driver for heat.

The driver should support two modes:

get_all: query all Heat Stacks.
notifications: notify the datasource upon a change in a stack definition or state

This blueprint refers to get_all implementation.

Proposed change

Heat driver will be configured with:

Poll interval in seconds (default: 600)

Every poll-interval seconds, Heat Driver will call Heat list all stacks API. The stacks will be converted to Vitrage datasource events and passed to Vitrage Graph queue.

Alternatives

None

Data model impact

Heat event will be sent to Vitrage Graph queue with the following properties:

Field	Description	Examples
stack_id	The stack UUID	e47e1be6-3598-46e6-bb63-0cc9a4e35ad7
name	The stack name	mystack
description	The stack description	mydescription
user_project_id	The ID of the tenant that owns the alarm	5542b27142154f30b32dea6238aa81aa
owner	The ID of the user that owns the alarm	5555527142154f30b32dea6238aa81aa
status	The stack status	in progress / failed
status_reason	The stack status reason	Stack create started ….

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

Heat driver should be configured

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: dan-offek

Work Items

None

Dependencies

None

Testing

This blueprint requires unit tests and Tempest tests.

Documentation Impact

None

References

Datasource main blueprint: https://github.com/openstack/vitrage-specs/blob/master/specs/mitaka/vitrage-datasource.rst

Vitrage Get Template List API

Thu, 30 Jun 2016 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/template-list-api

An API for list all templates loaded from templates lib, both those that passed validation and those that did not

Problem description

We would like to be able to list all templates loaded from /etc/vitrage/templates, both those that passed validation and those that did not before uploading it to Vitrage.

Proposed change

Create API to list all Vitrage loaded templates.

Valid template - template that passed validation and loaded into Scenario Repository.
Invalid template = template that did not pass validation.

The template list API returns a table with the following columns:

uuid - a unique id generated by Vitrage
name - template’s name
status - pass validation or not
status details
date - when template validation occurred (before template loading is executed)

Alternatives

None

Data model impact

None

REST API impact

Template List

Returns template list

GET /

Headers

X-Auth-Token (string, required) - Keystone auth token
Accept (string) - application/json
User-Agent (String)

Path Parameters

None.

Query Parameters

None

Request Body

None.

Request Examples

GET /v1/template/
Host: 135.248.18.122:8999
User-Agent: keystoneauth1/2.3.0 python-requests/2.9.1 CPython/2.7.6
Accept: application/json
X-Auth-Token: 2b8882ba2ec44295bf300aecb2caa4f7

Response

Status code

200 - OK
400 - Bad request

Response Body

Returns a table that is a list of all templates. Each row describes a template and its status.

Response Examples

::

uuid	name	status	status details	date
67bebcb4-53b1-4240-ad05-451f34db2438 4cc899e6-f6cb-43d8-94a0-6fa937e41ae2 0548367e-711a-4c08-9bdb-cb61f96fed04 33cb4400-f846-4c64-b168-530824d38f3e a04cd155-0fcf-4409-a27c-c83ba8b20a3c	vm_down_causes_suboptimal_application host_cpu_load_causes_vm_problem switch_connectivity_issues host_nic_down disconnected_storage_problems	failed pass pass pass pass	Entity definition must contain template_id field Template validation is OK Template validation is OK Template validation is OK Template validation is OK	2016-06-29T12:24:16Z 2016-06-29T12:24:16Z 2016-06-29T12:24:16Z 2016-06-29T12:24:16Z 2016-06-29T12:24:16Z

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

liat har-tal <liat.har-tal@nokia.com>

Work Items

None

Future lifecycle

None

Dependencies

None

Testing

Tempest tests also need to be added in order to test:

Get template list

Documentation Impact

The new api should be documented

References

None

Aodh Synchronizer Plugin - get_all implementation

Thu, 03 Mar 2016 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/vitrage/+spec/synchronizer-aodh-get-all

This blueprint describes the Aodh plugin for Vitrage Synchronizer, and its implementation for get_all Aodh alarms.

Problem description

Aodh alarms should be added to Vitrage Graph via Vitrage Synchronizer. This requires writing a Synchronizer plugin for Aodh.

The plugin should support two modes:

get_all: query all Aodh alarms and send corresponding events to the Synchronizer
notifications: notify the Synchronizer upon a change in an Aodh alarm definition or state

This blueprint refers to get_all implementation.

Proposed change

Aodh plugin will be configured with:

Poll interval in seconds (default: 60)

Every poll-interval seconds, Aodh plugin will call Aodh list all alarms API. The alarms will be converted to Vitrage Synchronizer events and passed to Vitrage Graph queue.

Alternatives

None

Data model impact

Aodh event will be sent to Vitrage Graph queue with the following properties:

Field	Description	Examples
alarm_id	The alarm UUID	e47e1be6-3598-46e6-bb63-0cc9a4e35ad7
name	The alarm name	CpuUtilAlarm1
description	The alarm description	Alarm when cpu is eq a avg of 300.0 over 60 seconds
enabled	Is the alarm enabled	true/false
project_id	The ID of the project or tenant that owns the alarm	5542b27142154f30b32dea6238aa81aa
repeat_actions	Should actions be re-triggered on each evaluation cycle	true/false
severity	The alarm severity	low/moderate/critical
state	The alarm state	ok/alarm/insufficient data
time_constraints	Optional. Evaluate the alarm within this time constraint	?
type	The alarm type	threshold/event
user_id	The ID of the user who created the alarm	8ab65ef808b245e3ba234b7b3554cb94
resource_id	Optional. The resource id in the query of the alarm rule	b0bf3635-d9e8-4624-9793-7aac82948c0a
sync_type	The source of information	aodh (constant value)

Aodh threshold alarms will include in addition:

Field	Description	Examples
state_timestamp	The date of the last alarm state changed	2016-03-01T17:13:08.847812
timestamp	The date of the last alarm definition update	2016-03-01T17:13:08.847812
threshold_rule	The threshold rule, for a threshold alarm	dictionary

Aodh event alarms will include in addition:

Field	Description	Examples
event_type	Event type for event alarm	compute.instance.update
timestamp	Does not exist in Aodh? current time will be used	2016-03-01T17:13:08.847812

Gnocchi alarms, combination alarms and composite alarms are still TBD.

TBD: how this information will be transformed to a Vitrage Graph vertex

REST API impact

None

Versioning impact

None

Other end user impact

None

Deployer impact

Aodh plugin should be configured

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: ifat-afek

Work Items

None

Dependencies

None

Testing

This blueprint requires unit tests and Tempest tests.

Documentation Impact

Aodh Configuration should be documented

References

Synchronizer main blueprint: https://github.com/openstack/vitrage-specs/blob/master/specs/mitaka/vitrage-synchronizer.rst

Evaluator Engine

Mon, 15 Feb 2016 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/vitrage/+spec/evaluator-engine

Vitrage Evaluator serves as workflow manager controlling the analysis and activation of templates and execution of template actions.

Evaluator engine is the main core of Vitrage evaluator which responsible for managing the templates, executing them over Vitrage Graph and running their action.

+---------------------+
|                     |
|                     <------------------+
|   Vitrage Graph     |                  |
|                     |          +-------+-------------------+
|                     |          | Vitrage Evaluator         |
+--------------+------+          |                           |
               |                 |                  +------+ |
               |                 |                  | RCA  | |
               +----------------->                  |      | |
                                 |                  +------+ |
                                 |                  +------+ |
+---------------------+          |                  |Deduce| |
|                     <----------+                  |Alarm | |
|   Vitrage Notifier  |          |                  +------+ |
|                     |          |                           |
|                     |          +---------------------------+
|                     |
+---------------------+

Problem description

Vitrage requires a component that is responsible for managing and executing templates, which are the basis for the different algorithms used in Vitrage, such as RCA.

Use cases:

When a new instance is added/removed/updated in the Vitrage Graph, find the templates relevant to this change and compare the pattern specified in each template to the new graph structure. For each pattern match found, execute the actions specified in the template.
When a new alarm is added/removed/updated in the Vitrage Graph, find the templates relevant to this change and compare the pattern specified in each template to the new graph structure. For each pattern match found, execute the actions specified in the template.

Proposed change

Managing Templates

Users can perform CRUD actions on templates, in order to make changes to the use cases Vitrage supports. Specifically, when adding a template it should be added and stored in graph format and saved in NetworkX (Graph-based in-memory DB). Similarly, this template can be modified or deleted as well. The functionality added here should have a corresponding set of API calls for users to perform these actions.

Executing Templates

When a change takes place in the Vitrage Graph, it informs the Evaluator Engine of the change. The engine will then search for templates relevant to this change and compare the pattern specified in each template to the new graph structure. For each pattern match found, execute the actions specified in the template.

Alternatives

None

Data model impact

The templates are saved in NetworkX graph-base in memory DB.

REST API impact

The functionality added here for managing templates should have a corresponding set of API calls for users to perform these actions.

Versioning impact

None

Other end user impact

None

Deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

TBD

Work Items

Dependencies

API with Vitrage Graph
API with Vitrage Notifier

Testing

TBD

Documentation Impact

TBD

References

TBD

Vitrage Get Topology API

Sun, 27 Dec 2015 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/vitrage-api-get-topology

Vitrage needs to supply an api for listing a full or partial resources and alarms topology.

Problem description

We would like to be able to retrieve the Vitrage topology, which will include both the resources and their appropriate alarms

Proposed change

Create API to retrieve the resources and alarms topology :

Retrieve the full topology.
List partial topology with filter :

Full connected components of a given element

Connected component of a given element, filtered according to edges of specific type(s) (i.e., labels and properties)

Alternatives

None

Data model impact

None

REST API impact

List Versions

Lists the supported versions of the Vitrage API.

GET /

Headers

X-Auth-Token (string, required) - Keystone auth token
Accept (string) - application/json

Path Parameters

None.

Query Parameters

None.

Request Body

None.

Request Examples

GET / HTTP/1.1
Host: 135.248.19.18:8999
X-Auth-Token: 2b8882ba2ec44295bf300aecb2caa4f7
Accept: application/json

Response

Status code

200 - OK

Response Body

Returns a JSON object with a ‘links’ array of links of supported versions.

Response Examples

{
    "versions": [
        {
           "id": "v1.0",
          "links": [
                {
                 "href": "http://135.248.19.18:8999/v1/",
                "rel": "self"
               }
          ],
          "status": "CURRENT",
          "updated": "2015-11-29"
        }
    ]

}

Get topology

Get the topology for the node. Its possible to filter the edges vertices and depth of the graph

GET /v1.0/topology/

Headers

X-Auth-Token (string, required) - Keystone auth token
Accept (string) - application/json

Path Parameters

None.

Query Parameters

edges (string(255), optional) - name of edges to filter.
vertices (string, optional) - name of vertices to filter.
depth (int, optional) - the depth of graph required.
graph type ([tree,graph], optional) - The type of graph required defaults to graph

Request Body

None.

Request Examples

GET /v1/topology/?graph_type=tree&depth=4&edges=vm&edges=host&edges=zone
&vertices=contains
Host: 135.248.19.18:8999
Content-Type: application/json
X-Auth-Token: 2b8882ba2ec44295bf300aecb2caa4f7

Response

Status Code

200 - OK
400 - Bad request

Response Body

Returns a JSON object that describes a graph with nodes and links. If a tree representation is asked then returns a Json tree with nodes and children.

An error of cannot represent as a tree will be return if the graph is not a tree. (400 - Bad request)

Response Examples

{
"children": [
    {
        "children": [
            {
                "children": [
                    {
                        "id": 16,
                        "name": "vm5",
                        "state": "ERROR"
                    },
                    {
                        "id": 23,
                        "name": "vm12",
                        "state": "SUBOPTIMAL"
                    }
                ],
                "id": 8,
                "name": "host4",
                "state": "SUBOPTIMAL"
            },
            {
                "children": [
                    {
                        "id": 26,
                        "name": "vm15",
                        "state": "SUBOPTIMAL"
                    },
                    {
                        "id": 19,
                        "name": "vm8",
                        "state": "ERROR"
                    },
                    {
                        "id": 12,
                        "name": "vm1",
                        "state": "RUNNING"
                    }
                ],
                "id": 5,
                "name": "host1",
                "state": "SUBOPTIMAL"
            }
        ],
        "id": 1,
        "name": "zone0",
        "state": "ERROR"
    },
    {
        "children": [
            {
                "children": [
                    {
                        "id": 21,
                        "name": "vm10",
                        "state": "RUNNING"
                    },
                    {
                        "id": 14,
                        "name": "vm3",
                        "state": "SUBOPTIMAL"
                    }
                ],
                "id": 10,
                "name": "host6",
                "state": "ERROR"
            },
            {
                "children": [
                    {
                        "id": 20,
                        "name": "vm9",
                        "state": "SUBOPTIMAL"
                    },
                    {
                        "id": 13,
                        "name": "vm2",
                        "state": "ERROR"
                    }
                ],
                "id": 4,
                "name": "host0",
                "state": "ERROR"
            },
            {
                "children": [
                    {
                        "id": 24,
                        "name": "vm13",
                        "state": "RUNNING"
                    },
                    {
                        "id": 17,
                        "name": "vm6",
                        "state": "SUBOPTIMAL"
                    }
                ],
                "id": 7,
                "name": "host3",
                "state": "ERROR"
            }
        ],
        "id": 2,
        "name": "zone1",
        "state": "SUBOPTIMAL"
    },
],
    "id": 0,
    "name": "node1",
    "state": "RUNNING"
}

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: dan-ofek <dan.ofek@alcatel-lucent.com>

Work Items

None

Future lifecycle

None

Dependencies

Depends on the blueprints of the graph and the synchronizer

Testing

Tempest tests also need to be added to test “get full topology” and “get partial topology” API.

Documentation Impact

The new api should be documented

References

None

Vitrage CLI

Mon, 21 Dec 2015 00:00:00

https://blueprints.launchpad.net/python-vitrageclient/+spec/vitrage-cli

Vitrage Project introduces a Root Cause Analysis (RCA) engine for organizing, analyzing and expanding OpenStack alarms & events.

In order to communicate with Vitrage a command line utility can be used that will issue some REST command to the API service.

The CLI will use a python-vitrageclient which is a client library built on the Vitrage API.

+-----------------+                     +-----------------+
|      *CLI*      |                     |                 |
|                 |                     |                 |
|  RCA            |                     |                 |
|                 |   HTTP/Vitrage API  |  vitrage api    |
|  CRUD Templates |+-------------------->                 |
|                 |                     |    service      |
|  Topology       |                     |                 |
|                 |                     |                 |
+-----------------+                     +-----------------+

Problem description

As a user I would like to be able to see the root cause of any alerts or events in the system. A command line utility will be used to communicate with Vitrage API service. The CLI will 3 types of commands:

RCA - find the root cause for an alert/event
CRUD Templates - Create/Read/Update/Delete Templates
Topology - get the topology of the system

Proposed change

The CLI and the vitrage client is part of a new project for Root Cause Analysis called vitrage

Alternatives

None

Data model impact

No data is stored or cached.

REST API impact

Will implement the api of vitrage-api service

Versioning impact

Discuss how your change affects versioning and backward compatibility:

None

Other end user impact

The User will be able to interact using any HTTP rest client. The User will also have a UI.

Deployer impact

A new project called Vitrage will deploy the vitrage client and CLI

Developer impact

None

Horizon impact

A new UI will be added to Horizon to support the Vitrage project a separate blueprint will be supplied.

Implementation

Assignee(s)

None

Work Items

None

Dependencies

None

Testing

All code will be tested.

Documentation Impact

None

References

Vitrage project the get topology api blueprint https://blueprints.launchpad.net/vitrage/+spec/get-topology-api

Vitrage Support for Deduced Alarms

Mon, 16 Nov 2015 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/support-deduced-alarms

Vitrage should support raising deduced alarms. When an alarm is raised on a certain resource, we might conclude that it results in problems in other related resources. In this case, we would like to raise deduced alarms on these other resources, even if we got no indication of a problem from other sources of information.

Problem description

There are cases where we know that a failure in one resource should result in problems in other resources. For example, high CPU load on a host may cause suboptimal performance of all instances running on this host, thus performance problems in the applications running on these instances. As far as Nova is concerned, all instances are up and running, and there is no indication of a problem.

Proposed change

The Vitrage Evaluator serves as workflow manager controlling the analysis and activation of templates and execution of template actions. One of its responsibilities is to listen to changes in Vitrage Graph, and upon a change execute the matching templates. This is a general mechanism that should work for all kinds of templates and perform several kinds of actions.

The aim of this blueprint is to make sure deduced alarms functionality works properly end to end.

Whenever a new alarm is raised, Vitrage Graph is updated with a new vertex for this alarm, connected to the relevant resource. Then, the vitrage evaluator engine looks for templates that could match this new alarm. If deduced alarms templates are found, the engine will try to find a full match for the entire template in Vitrage Graph. If found, the engine will ask the notifier to raise new alarms according to the template.

Example for a graph with deduced alarms:

Original alarm:
+-----------+               +------------+
|Host High  |     on        |            |
|CPU load   | +-----------> |  Host      |
|           |               |            |
+-----------+               +-----+------+
                                  |
                                  | contains
Deduced alarm                     |
to raise:                         |
+-----------+               +-----v------+
|Instance   |     on        |            |
|Suboptimal | +-----------> |  Instance  |
|Performance|               |            |
+-----------+               +------------+

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Horizon impact

None

Implementation

Assignee(s)

Primary assignee:: ifat_afek <ifat.afek@alcatel-lucent.com>

Work Items

The blueprint includes:

Define the exact syntax for deduced alarms templates
Call the notifier to raise alarms

Future lifecycle

None

Dependencies

Vitrage Graph
Vitrage Engine
Vitrage Notifier

Testing

This change needs to be tested by unit tests.

Documentation Impact

None

References

https://wiki.openstack.org/wiki/Vitrage

Vitrage RCA Support

Mon, 16 Nov 2015 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/support-rca

Vitrage should support RCA calculation. This feature will allow us to track the cause-and-effect path of an alarm raised in OpenStack. In order to track the local causal relationships between alarm-pairs we shall use one or more RCA templates which will specify which alarms cause which alarms.

Problem description

In case of a major failure in the system, we might get a lot of alarms, which will be hard to track. We would like to identify the root cause of the alarms, so the user can focus on understanding and fixing this alarm.

Proposed change

The aim of this blueprint is to make sure RCA functionality works properly end to end.

Whenever the Vitrage Graph is updated, we will calculate RCA and optionally connect alarm vertices with “causes” edges. When RCA relations are queried for a certain alarm (i.e. which alarm(s) caused it and which alarm(s) were caused by it), we will traverse the already-existing “causes” edges and return the RCA tree.

Example for a graph with causes edges:

 +---------------+              +-------------+
 |               |     on       |             |
 | switch alarm  | +----------> |   switch    |
 |               |              |             |
 +------+--------+              +-------+-----+
        |                               |
causes  |                               | attached
        |                               |
        v                               v

 +---------------+              +-------------+
 |               |     on       |             |
 | host alarm    | +----------> |   host      |
 |               |              |             |
 +------+--------+              +-------+-----+
        |                               |
causes  |                               | contains
        |                               |
        |                               |
        v                               v

 +---------------+              +-------------+
 |               |     on       |             |
 | instance alarm| +----------> |  instance   |
 |               |              |             |
 +---------------+              +-------------+

Alternatives

We could re-calculate the RCA relationship whenever someone queries it, but this would be inefficient. Calculating in advance and keeping the results in Vitrage Graph makes more sense.

Data model impact

None

REST API impact

The API is defined in a separate blueprint: https://blueprints.launchpad.net/vitrage/+spec/rca-api

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Performance should be tested. Most of the performance risk is in the common blueprints like https://blueprints.launchpad.net/vitrage/+spec/networkx-graph-driver (see also https://blueprints.launchpad.net/vitrage/+spec/networkx-performance-improvement). However, we will also need to have specific tests for RCA.

Other deployer impact

None

Developer impact

None

Horizon impact

We should develop horizon UI plugin for viewing the RCA relationship. This should be described in a separate blueprint.

Implementation

Assignee(s)

Primary assignee:: ifat_afek <ifat.afek@alcatel-lucent.com>

Work Items

The blueprint includes:

Define the exact syntax for RCA templates
Mark the causal relationship between two alarms. We would implement it using an action that adds a “causes” edge between the alarm vertices in Vitrage Graph.
Define and implement the method to query the RCA relations for a given alarm

Future lifecycle

None

Dependencies

Vitrage Graph
Vitrage Engine

Testing

This change needs to be tested by unit tests.

Documentation Impact

None

References

https://wiki.openstack.org/wiki/Vitrage

Vitrage Resource Processor

Wed, 11 Nov 2015 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/entity-graph-change-notifications

When the Entity Graph is changed, different services would like to be informed of the change, and perform operations accordingly.

Problem description

When the Entity Graph is changed, different services would like to be informed of the change, and perform operations accordingly. Such as: (1) When edges vertices are added removed, the Vitrage Evaluator will run the subgraph matching algorithm on the graph to find the templates which are matching and perform required actions. (2) When properties of existing edges vertices are modified, this can also impact subgraph matching outcome and thus the Vitrage Evaluator will need to be notified.

Proposed change

For each CRUD operation on the Entity Graph, we will support adding listeners that can be notified, and receive updates when each change occurs. The update must include information about each change.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: alexey_weyl <alexey.weyl@alcatel-lucent.com>

Work Items

None

Future lifecycle

None

Dependencies

None

Testing

None

Documentation Impact

None

References

None

NetworkX Graph driver

Tue, 10 Nov 2015 00:00:00

https://blueprints.launchpad.net/vitrage/+spec/networkx-graph-driver

Graph Driver is the defined API for access and manipulation of the underlying graph used for storing the Entity Graph. This API should be implemented for the NetworkX graph package and possibly for other graph tools, allowing Vitrage a seamless transition between different underlying graph implementations.

Problem description

Vitrage will have a graph data structure that will hold a list of physical and virtual entities and their relationships to one another, in what we call the Entity Graph. The specific implementation of this graph should be interchangeable allowing a stateless or state-full implementations.

Proposed change

1. Graph Driver

Create a Graph Driver, which defines a set of graph methods, to be implemented over NetworkX. This blueprint describes the addition of the Graph Driver and NetworkX Driver.

                                   +-------------------+
                                   |                   |
+------------------+       +------->   NetworkX Driver |
|                  |       |       |                   |
|      Graph       |  Impl |       +-------------------+
|                  |-------+
|      Driver      |       |       +-------------------+
|                  |       |       |                   |
+------------------+       +------->   Other Drivers   |
                                   |  (BulbFlow,etc..) |
                                   +-------------------+

Specification of the Graph Driver API:

Note that the entity graph is a property graph, where edges and vertices can also have a set of key-value properties, which can be added, updated and removed as well.

Graph CRUD

init
num_of_edges
num_of_vertices
copy //deep copy of the graph

Vertex CRUD

add_vertex
add_vertices
get_vertex
update_vertex
remove_vertex

Edges CRUD

add_edge
get_edge
update_edge
remove_edge

Graph algorithms

subgraph_matching (sub-graph isomorphism)
BFS
DFS

2. NetworkX Driver

NetworkX is a pure python library for graphs. It is stateless and suitable for operation on large real world graphs. The NetworkX Driver will implement Graph Driver

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: None.

Work Items

Create the GraphDriver skeleton

Implement Graph Driver for NetworkX

Testing of GraphDriver over NetworkX