telemetry-specs

Network Statistics From OpenDaylight

Tue, 13 Jun 2017 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/network-statistics-from-opendaylight

This feature proposes to add a ceilometer driver to collect network statistics information using REST APIs exposed by network-statistics module in OpenDaylight.

Problem description

OpenDaylight collects and stores network statistics information and exposes the same via its northbound REST APIs. An example of such statistics is number of packets received or transmitted by a port which may be created through neutron service.

A large portion of the network configuration received by OpenDaylight is from OpenStack. It will be good to have the network statistics readable from OpenDaylight via ceilometer APIs in a OpenStack-OpenDaylight deployment.

A driver already exists in ceilometer which collects statistics from OpenDaylight. But that ODL REST APIs used by the driver have been unfortunately removed or less maintained in current release of OpenDaylight which is Carbon.

We propose a v2 version of the driver which will utilize a new statistics module in OpenDaylight. To start with the v2 version of the driver will currently support only a subset of counters that were supported by old driver. Eventually this driver can be enhanced to support all counters. This v2 OpenDaylight ceilometer driver will enable counters to be stored with either MongoDB (or) with the Gnochhi collector thereby supporting both backends for ceilometer.

Proposed change

We propose to create a new driver in networking-odl in networking_odl/ceilometer/network/statistics/opendaylight_v2/

The driver will communicate with OpenDaylight to fetch statistics through REST APIs.

We wil support following meters:

* switch
* switch.ports
* switch.port
* switch.port.uptime
* switch.port.receive.drops
* switch.port.receive.errors
* switch.port.transmit.packets
* switch.port.receive.packets
* switch.port.transmit.bytes
* switch.port.receive.bytes
* port
* port.uptime
* port.receive.drops
* port.receive.errors
* port.transmit.packets
* port.receive.packets
* port.transmit.bytes
* port.receive.bytes
* switch.table.active.entries

New resource types will be added to /etc/ceilometer/gnocchi_resources.yaml:

- resource_type: switch
  metrics:
    - 'switch'
    - 'switch.ports'
  attributes:
    controller: resource_metadata.controller

- resource_type: switch_port
  metrics:
    - 'switch.port'
    - 'switch.port.uptime'
    - 'switch.port.receive.drops'
    - 'switch.port.receive.errors'
    - 'switch.port.transmit.packets'
    - 'switch.port.receive.packets'
    - 'switch.port.transmit.bytes'
    - 'switch.port.receive.bytes'
  attributes:
    switch: resource_metadata.switch
    port_number_on_switch: resource_metadata.port_number_on_switch
    neutron_port_id: resource_metadata.neutron_port_id
    controller: resource_metadata.controller

- resource_type: port
  metrics:
    - 'port'
    - 'port.uptime'
    - 'port.receive.drops'
    - 'port.receive.errors'
    - 'port.transmit.packets'
    - 'port.receive.packets'
    - 'port.transmit.bytes'
    - 'port.receive.bytes'
  attributes:
    controller: resource_metadata.controller

- resource_type: switch_table
  metrics:
    - 'switch.table.active.entries'
  attributes:
    controller: resource_metadata.controller
    switch: resource_metadata.switch

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

Users will have to configure OpenDaylight REST API endpoint information in resources attribute in configuration file of ceilometer ie /etc/ceilometer/polling.yaml. For eg:

sources:
  - name: odl_source
    interval: 600
    resources:
      - opendaylight.v2://127.0.0.1:8080/controller/statistics?
          auth=basic&user=admin&password=admin&scheme=http
    meters:
      - "switch"
      - "switch.ports"
      - "switch.port"
      - "switch.port.uptime"
      - "switch.port.receive.drops"
      - "switch.port.receive.errors"
      - "switch.port.transmit.packets"
      - "switch.port.receive.packets"
      - "switch.port.transmit.bytes"
      - "switch.port.receive.bytes"
      - "port"
      - "port.uptime"
      - "port.receive.drops"
      - "port.receive.errors"
      - "port.transmit.packets"
      - "port.receive.packets"
      - "port.transmit.bytes"
      - "port.receive.bytes"
      - "switch.table.active.entries"

Other end user impact

None.

Performance/Scalability Impacts

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: Deepthi V V
Ongoing maintainer:: Deepthi V V

Work Items

Add new ceilometer driver for OpenDaylight in networking-odl
Add support for new driver in devstack.
Provide support for switch.* meters for gnocchi as backend in ceilometer.
Unit tests for new driver

Future lifecycle

None.

Dependencies

Statistics module should be available in OpenDaylight Nitrogen release.

Testing

Unit Tests will be added to test the new driver.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

https://blueprints.launchpad.net/ceilometer/+spec/network-statistics-from-opendaylight

https://git.opendaylight.org/gerrit/#/c/59283/

Track cinder capacity notifications

Sun, 24 Apr 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/cinder-capacity-notifications

The goal of this blueprint is to capture the capacity notifications emitted by cinder service when it notifies storage capacity information to ceilometer.

Problem description

Cinder service collects storage capacity information about each pool/backend. These includes total/free/allocated/provisioned/virtual_free capacity info. Cinder service emits the formatted information as notifications periodically.

It’s better for ceilometer service to add a notification plugin to listen to the topic. If the information payload can be consumed and transformed into samples, it will be helpful for admin users to have an estimation of the future capacity planning.

Proposed change

Add new metrics for different capacity information carried by capacity payload from notifications emitted by Cinder. They will include:

CapacityTotalSize It is the total physical capacity of a pool/backend.
CapacityFreeSize It is the real physical available capacity of a pool/backend.
CapacityAllocatedSize It is the physical capacity allocated directly thru Cinder. Note: The capacity which is not allocated thru Cinder is not included in.
CapacityProvisionedSize It is the capacity which has been provisioned in a pool/backend.

Note: It includes the capacity both allocated directly thru Cinder and not. The provisioned capacity size is equal or greater than the allocated size.
CapacityVirtualfreeSize It is the apparent available virtual capacity means how much capacity can still provision besides the capacity which has been provisioned already in the pool/backend.

Note: It is different from free capacity, free capacity is related to real available physical capacity. For thin provisioning support, due to the max_over_subscription_ratio, the provisioned capacity can be much larger than the real physical capacity.

Please reference https://review.openstack.org/#/c/129342/ to get the detail explanation of above terminology.

Each metric extracts the capacity information which it is interested in from the notification payload. The admin users can utilize ceilometer statistics API to get the sum of each kind of capacity. Also they can utilize the horizon resource usage page to get the chart of the trend of each kind of capacity information.

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

No new impacts. Pre-existing concerns with capacity at the notification and storage handling layers remain.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: XinXiaohui
Ongoing maintainer:: XinXiaohui

Work Items

Add capacity.total.size metric
Add capacity.free.size metric
Add capacity.allocated.size metric
Add capacity.provisioned.size metric
Add capacity.virtual_free.size metric
Test coverage for the above metrics and samples validation.

Future lifecycle

In the future new types of capacity notifications maybe expected from the Cinder service to account for the statistics data. These will need to be handled later.

Dependencies

None.

Testing

Unit and integration Tests will be added to cover the necessary metrics and validate samples generated.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

https://etherpad.openstack.org/p/kilo-cinder-capacity-headroom

https://review.openstack.org/#/c/170380

Event To Sample Publisher

Sun, 24 Apr 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/event-to-sample-publisher

Problem description

Bracket events like compute.instance.create.latency created from BP: Transformers for events pipeline https://blueprints.launchpad.net/ceilometer/+spec/events-pipeline-transformers need to be published and stored as sample.

The bracket events created from transformers in events pipeline are new events/metrics with any kind of latency time spans like time.instance.creation or time.instance.life that is a delta time between compute.instance.create.start/end, compute.instance.create.end/delete.end, etc.

Proposed change

Add a new notifier publisher and use this publisher in event transformer, when there is a new transformed event from events pipeline transformers, convert it to notification, send it back to ceilometer notification listener, the notification payload is generated from event traits, then the sample pipeline will convert and publish it into sample.

For example, for compute.instance.create.latency event, Firstly, add this publisher(for example: tosamplenotifier://) to sink in event transformer for this event

- name: instance_create_bracketer_sink
  transformers:
      - name: "bracket"
        parameters:
          ........
  publishers:
      - tosamplenotifier://

then, add meter schema in meters.yaml

- name: 'compute.instance.create.latency'
  event_type:
    - 'compute.instance.create.latency'
  type: 'gauge'
  unit: 's'
  volume: $.payload.latency
  user_id: $.payload.user_id
  project_id: $.payload.tenant_id
  resource_id: $.payload.instance_id

The notification payload send back to message bus is generated from event traits, will be like this ::

{‘latency’: 0.355098, ‘user_id’: u’0f1b1e94ec2045af9f49f9b7e1d6b409’, ‘service’: u’network.yuntong-ThinkStation’, ‘resource_id’: u’67d1c4b3-84aa-42d1-a857-2d1481fe21dd’, ‘tenant_id’: u’c8ce7938e38b4612a8b3daab441b804c’, ‘request_id’: u’req-597a85ec-ebab-492f-8288-b6b72fc476b5’, ‘project_id’: u’c8ce7938e38b4612a8b3daab441b804c’, ‘publisher_id’: ‘compute.yuntong-ThinkStation’}

and finally, the compute.instance.create.latency sample will be like this ::

Resource ID	Name	Type	Volume	Unit	Timestamp
e8e8adf5-8ba1-4247-b1af-1e8c928563e7	compute.instance.create.latency	gauge	7.133763	s	2015-09-16T07:04:58.540929

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

This spec proposes additional publisher for event transformers in event_pipeline.yaml

Other end user impact

None.

Performance/Scalability Impacts

None

Other deployer impact

Configuration options in event_pipeline.yaml. Configuration options in meter.yaml.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: yuntongjin
Ongoing maintainer:: yuntongjin

Work Items

Add a new publisher that will send the convert notification to sample listener.

Future lifecycle

None

Dependencies

Transformers for events pipeline https://review.openstack.org/#/c/162167/

Testing

Extend messaging publisher testing.

Documentation Impact

Capture new meters which from event brackelet transformers.

References

https://blueprints.launchpad.net/ceilometer/+spec/events-pipeline-transformers

Unify the timestamp of samples generated by polling agents

Sun, 24 Apr 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/unify-timestamp-of-polled-data

Problem description

Many of the pollsters define the timestamp individually for each generated sample, which should really be timestamping based on when the data was polled and not when each sample is generated. We need to set the timestamp of the polled data to the timestamp when the polling starts for unity.

This should save overhead of calculating timestamp each sample and also allow for easier grouping of polled data.

Proposed change

Modify ceilometer.agent.manager.PollingTask.poll_and_notify method to set the timestamp of samples to the time when the data was polled:

# Get the timestamp when pollster starts to work
polling_timestamp = self._get_current_timestamp()

samples = pollster.obj.get_samples(
    manager=self.manager,
    cache=cache,
    resources=polling_resources
)

for sample in samples:
# Unify the timestamp of polled samples
sample.set_timestamp(polling_timestamp)

Drop all the timestamp generations when we build samples via pollsters, like: https://github.com/openstack/ceilometer/blob/6.0.0.0b2/ceilometer/image/glance.py#L113

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Save overhead of calculating timestamp each sample.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <yuywz>

Work Items

Update ceilometer.agent.manager.PollingTask.poll_and_notify method.

Drop all the timestamp generations when we build samples via pollsters.

Future lifecycle

None

Dependencies

None

Testing

Add unit test to test: The timestamp of samples generated during a polling cycle is unified.

Documentation Impact

None

References

Originally reported as a bug: https://bugs.launchpad.net/ceilometer/+bug/1491509 Blueprint: https://blueprints.launchpad.net/ceilometer/+spec/unify-timestamp-of-polled-data

Add pagination support for Aodh

Thu, 31 Mar 2016 00:00:00

https://blueprints.launchpad.net/aodh/+spec/support-pagination

This BP proposed to add pagination support for Aodh. User can use this feature to set limit, marker and sort when they query their alarm and alarm history.

Problem description

Currently when list alarm and alarm history, all the existed alarm and history will return at one time, and it is really not user friendly for users.

Proposed change

Allow Aodh user to use the general pagination mechanism with the help of limit, marker, sort_key, sort_dir optional parameters to list alarm and alarm history.

sort_key: Key used to determine sort order
sort_dir: Direction for with the associated sort key (“asc” or “desc”)
marker: The last alarm ID of the previous page. Displays list of alarms after “marker”.
limit: Maximum number of alarms to display. If limit == -1, all alarms will be displayed.

Alternatives

Keep the current anti-friendly situation.

Data model impact

None

REST API impact

New optional parameters limit, marker, sort_key, sort_dir will be added to GET /v2/alarms and GET /v2/query/alarms/history

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Zhenyu Zheng
Other contributors:: liusheng

Work Items

Add pagination support for alarm and alarm history;
Add the related support in alarm client.

Future lifecycle

None

Dependencies

None

Testing

Related tests will be added.

Documentation Impact

References about how to use pagination will be added.

References

‘Big Data’ SQL part two

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/bigger-data-sql

In step 1 of sql refactoring, we denormalised the data to capture and store data as close to its raw form as possible. This removed all the overhead caused by deprecated data design requirements. This blueprint further refactors the data model to organise data closer to api model.

Problem description

Currently, the Metering data model is completely denormalised. We store Samples, which are the raw data points, and Meters, which are the definition of said data points. This optimisation allows for good write performance but due to size of Sample table, can cause issues with read performance particularly with get_meter and get_resources. Specifically, joins can cause performance issues.

The current schema is as follows:

* meter - meter definition
    * id: meter id
    * name: meter name
    * type: meter type
    * unit: meter unit
* sample - the raw incoming data
    * id: sample id
    * meter_id: meter id (->meter.id)
    * user_id: user uuid
    * project_id: project uuid
    * resource_id: resource uuid
    * source_id: source id
    * resource_metadata: metadata dictionaries
    * volume: sample volume
    * timestamp: datetime
    * message_signature: message signature
    * message_id: message uuid

Proposed change

The proposed change is to re-implement a normalised model that is tailored to current API requirements. This means grouping Resource specific data in Resource table and Meter specific data in Meter Table.

the proposed schema is as follows:

* resource - resource data
    * internal_id: resource id (to handle assumption resources may not be
                   unique ie. diff user/project/source/meta per resource)
    * resource_id: resource uuid
    * user_id: user uuid
    * project_id: project uuid
    * source_id: source id
    * resource_metadata: metadata dictionary
    * metadata_hash: hash of metadata to allow comparison
* meter - meter definition
    * id: meter id
    * name: meter name
    * type: meter type
    * unit: meter unit
* sample - the raw incoming data
    * id: sample id
    * meter_id: meter id (->meter.id)
    * resource_id: resource id(->resource.internal_id)
    * volume: sample volume
    * timestamp: datetime
    * message_signature: message signature
    * message_id: message uuid

Alternatives

As a schema can be defined in quite a few ways, there are many alternatives in that sense.

Data model impact

Api model will remain unchanged. the sql backend model will change to proposal described above:

creating a new Resource table
moving appropriate values from sample table to resource table

These changes will require database migration.

REST API impact

We will need to adapt existing api model to interface with new backend schema but from user POV, there will be no change.

Security impact

None

Pipeline impact

None

Other end user impact

None, we will continue to store a new resource per sample

Performance/Scalability Impacts

The read performance should improve as we will not have a giant Sample table anymore but smaller, tailored Resource, Meter, and Sample tables. The write performance is not expected to degrade noticeably.

It is expected any degradation in write performance would be caught by existing tempest tests.

Use of Ilya’s performance tool will be used to verify there is improved read performance and negligible write performance degradation.[1]

[1] https://github.com/ityaptin/ceilometer/blob/master/tools/sample-generator.py

Other deployer impact

None

Developer impact

None, just a new schema to learn about

Implementation

Assignee(s)

Primary assignee:: chungg
Other contributors:: None
Ongoing maintainer:: chungg

Work Items

Migration script to add new attributes to Meter table and new Resource table
Modify impl_sqlalchemy get_meters, get_resource, record_metering_data, expirer and any other affected methods to use new schema

Future lifecycle

Most contributors know the sql backend to some degree. The community will maintain until v3 backend is phased in.

Dependencies

None

Testing

Existing test cases should cover change
Tempest test cases should cover performance degradation
Need to add test to handle data expiration

Documentation Impact

None

References

Discussion with Mike Bayer: https://etherpad.openstack.org/p/ceilometer-sqlalchemy-mike-bayer

Grenade Resource Survivability

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/grenade-resource-survivability

Integrated projects are required to participate in the Grenade upgrade testing harness. In addition to testing the upgrades themselves Grenade has facilities, called javelin, for testing survivability of resources through the upgrade process. Ceilometer needs to participate in this testing.

Problem description

To be certain that Ceilometer is robust across an upgrade it must be possible to process metrics and events from resources that exist before and after the upgrade. Grenade provides a feature named javelin which is designed to allow assertions that confirm resource A, present prior to the upgrade, is present after the upgrade.

In the Juno cycle Grenade is being updated to support javelin2 which, unlike the original javelin, should work well and has a declarative syntax for making assertions.

A previous spec describes adding basic Ceilometer support to Grenade. This spec is specifically about adding testing via javelin2.

Proposed change

Add support for Ceilometer resource checking to javelin2. This involves two types of changes (detailed below): Adding support for Ceilometer queries in the javelin code and adding Ceilometer specific entries to the resource definitions.

The main check that will be facilitated by javelin2 is ensuring the sanity of api queries with a time range that spans the entire window of time within which the Grenade test runs (e.g. -+12 hours from now).

Alternatives

It may be that Ceilometer queries do not map well to the resource model in use in javelin2. If this is the case then it might make sense to architect some other kind of before and after upgrade test specifically for Ceilometer. Doing so would be a shame, though. Better to make javelin2 more flexible.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

While Ceilometer has something of a reputation in this area, because it will already be running in the Grenade environment, no additional impact is expected by adding javelin tests.

Other deployer impact

None.

Developer impact

As Ceilometer features grow or change, adjustments in the javelin2 check tests may need to be made.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: chdent
Other contributors:: emilienm
Ongoing maintainer:: chdent

Work Items

Determine optimal form for handling ceilometer queries within javelin2. At a gross level there are two options: 1) Including the ceilometer queries as code within tempest.cmd.javelin itself, either built in or as plugin. 2) Representing the queries declaratively in a checks section of the resources.yaml file that could potenetially be used by other projects.
Add the queries (as described in Proposed change) in whatever form is chosen.

As a first pass, option 1 above is most expedient. If chosen the other options and the related review discussion should be considered for the future.

Future lifecycle

Ongoing maintenance of the Ceilometer portions of the javelin2 tests will be the responsibility of the Ceilometer project team.

Dependencies

These changes require javelin2 which was merged into Tempest on 30, May 2014.

Testing

Est quod est.

Documentation Impact

None.

References

Ceilometer blueprint for Grenade Upgrade Testing.
Javelin 2 blueprint, spec and code.

Grenade Upgrade Testing - Phase 1

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/grenade-upgrade-testing

Integrated projects are required to participate in the grenade upgrade testing harness. Ceilometer was integrated before these requirements were added but the requirements apply retroactively. Therefore, ceilometer must be added to the harness.

Problem description

Testing with grenade involves:

Installing a basic DevStack of the old version, confirming it with smoke and scenario tempest tests.
Establishing a javelin project which makes assertions about resources in the project.
Shutting down the original installation, upgrading to the new version of the code but not configuration, applying the relevant database migrations to update the schema, and repeating the tempest tests.
Testing the javelin project again to assert the integrity of resources across the upgrade.

This document considers the upgrade and tempest testing but not the javelin testing. Javelin will be addressed in another (forthcoming) document.

Prior to the Juno cycle ceilometer was neither a default service in DevStack nor enabled in tempest. The relevant configuration must be changed to allow testing in Grenade.

Proposed change

Enable ceilometer in Grenade (see Work Items for details).

Alternatives

Given the commitment to Grenade and the project graduation requirements, this is the way to go.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

These changes will mean a more positive upgrade experience for end users.

Performance/Scalability Impacts

There have been concerns that running ceilometer in the gate will negatively impact performance there. This had been true but recent changes in the sqlalchemy driver and database schema have shown enough improvement that ceilometer can be turned on.

Other deployer impact

None

Developer impact

Active testing will mean the discovery of bugs that someone will have to fix.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: emilienm
Other contributors:: vrovachev dbelova
Ongoing maintainer:: chdent

Work Items

Add upgrade calls to grenade.sh.
Add ceilometer upgrade scripts to grenade.
Add ceilometer directories to grenade cleanup procedures.
Ensure ceilometer data dumped between upgrades.

Note: These work items are captured in a pending patchset

Future lifecycle

As/when ceilometer expands to include additional services, it will be necessary to adjust devstack and grenade to manage (stop, start, cleanup) those services.

Dependencies

Other than those listed above, no additional dependencies.

Testing

These changes improve testing and will be validated there.

Documentation Impact

None.

References

None.

Ironic Notifications

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ironic-notifications

Work is in progress to get Ironic to emit notifications from data provided by IPMI sensors (such as cpu temp and voltage). Ceilometer needs to be updated to consume, transform and record these notifications.

Problem description

The Ironic project is doing work to emit notifications on the message bus containing IPMI sensor data. If Ceilometer processes these notifications other services will be able use queries and alarms to monitor and scale as required.

It is not possible to simply dump a message on the bus and for Ceilometer to handle it. Instead a notification plugin is required, in Ceilometer, to hear notifications at an exchange and topic and transform them into samples.

Proposed change

Add a new notification plugin for Ironic to transform sensor data into samples, using existing NotficationBase implementations as a model. To do this effectively a complete sample of the expected sensor data is required to determine the relevant types of samples.

Alternatives

Handling notifications is the standard and preferred method for gathering data into Ceilometer. The other option is polling which has known scalability issues, including over-frequent polling of IPMI. If Ironic is sending notifications it can continue to own the credentials for the IPMI access and also control the cadence with which sensors are polled. Using notifications also allows other services to consume the sensor data.

Data model impact

None.

REST API impact

There will be additional valid values in query parameters but no changes to API endpoints.

Security impact

None.

Pipeline impact

Unknown. The data being provided by the Ironic notifier is still being decided. Only when it is available will it be possible to determine what, if any, transformations may be usefully done in the pipeline.

Other end user impact

None.

Performance/Scalability Impacts

No new impacts. Pre-existing concerns with capacity at the notification and storage handling layers remain.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: chdent
Other contributors:: whaom
Ongoing maintainer:: chdent

Work Items

Establish expected data.
Create tests of transformation of sensordata to samples.
Create notification plugin to consume sensordata.
Create tests of notifications across fake bus.
Create sample query tests.

Future lifecycle

In the future new types of notifications are expected from the Ironic controller. These will need to be handled either by additional notification plugins or (hopefully) generic notification handling. The Ceilometer team will be responsible for collaborating with the Ironic team to ensure these are handled smoothly.

Dependencies

The primary dependency is work described in an Ironic spec. Once that is implemented, notifications will be present on the bus.

Testing

In addition to unit tests, Tempest tests which confirm that Ceilometer consumes notifications produced by Ironic would be useful. Such tests depend on there being a tool to provide sensor data in the testing environment. Such a tool would be similar to bm_poseur which fakes baremetal instances in devstack. In the event that such a tool is not available as long as the sample data used in unit tests is sufficiently representative then the tests themselves ought to be as well.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

Ironic spec
Review in progress for sending notification from Ironic.
Sample data

Alarm quotas

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/quotas-on-alarms

Currently, it is possible to create an unlimited number of alarms in Ceilometer. It would be useful to be able to specify a maximum number of alarms allowed per user or project in order to have some control and limit on the computing power required by Ceilometer.

Problem description

Presently, due to alarm evaluation process, the large number of alarms affects the performance of Ceilometer, but improvements to reduce this issue are underway with blueprint [1].

Even if performance wasn’t an issue, in order to prevent abuse from end users, the alarm quota mechanism must be implemented.

Proposed change

The proposed change will allow to express maximum number of alarms that can be set by user or project. These limits, if enforced, could be specified in Ceilometer’s configuration file (alarm section):

[alarm]
user_alarm_quota =
project_alarm_quota =

When a user adds an alarm, the following checks would occur:

Has the user filled his alarms quota ? If yes, the query is rejected.

Is the user’s project’s alarms quota filled ? If yes, the query is rejected.

Add the alarm.

Quotas don’t have to be set at every level. For example, setting a limit only at the project level will allow users to create as many alarms as they want as long as the sum of alarms for their specific project is less than the limit.

In order to provide backward compatibility, the default value for alarm quotas will be set to None.

Alternatives

An alternative solution, storing project/user quota in Keystone for all Openstack resources, was proposed during Havana development cycle [2]. This solution was since abandoned in favor of letting each component manage quotas on their own.

An alternative solution to implement quotas in Ceilometer would be to use the same mechanism that other Openstack components are using. This will imply the addition of an OS-QUOTAS API extension in Ceilometer, that will be used by the cloud operator for project/user specific quota assignments. These specific project/user quotas must be stored in Ceilometer backend, implying several backend specific quota storage implementations.

This alternative solution is more complex and could be implemented in future release cycles.

Data model impact

None

REST API impact

None

No API method is either added or changed. Nevertheless, the new error http response code (HTTP 403) will be returned when the alarm quotas are exceeded.

Security impact

None

Pipeline impact

None

Other end user impact

The impact of this feature for Heat Alarm and AutoScaling resources will be evaluated and fixed.

Performance/Scalability Impacts

None

Other deployer impact

The deployer can now define alarm quota in configuration file (alarm section):

[alarm]
quota_user_alarm =
quota_project_alarm =

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: arezmerita
Other contributors:: mhu-s

Work Items

Implement the alarm quota mechanism
Test the feature in unit tests

Future lifecycle

As mentioned in alternatives section, API extension for dynamic quota management can be implemented in future release cycles.

Dependencies

None

Testing

Tempest tests will be added to tests this feature

Documentation Impact

Ceilometer installation documentation will be updated

References

Ceilometer API RBAC - Granular Role-based Access Control

Wed, 10 Feb 2016 00:00:00

Current access control for the API presents a clear lack of granularity. In fact it is possible to have a “global” admin role or a simple “user” within the project and nothing in between.

With upcoming Keystone v3 enhancements we can expand the granularity of access control to allow non “global” admins, i.e. Domain Admins or Parent Project admins.

We will accomplish this by using a decorator on the API functions. The decorator will control access based on user/project roles and rules specified in the policy json file.

acl.py could be deprecated since its only function is to determine if a user is admin, and the decorator accomplishes this.

Example policy expansions:

Current policy.json only verifies user is admin:

{
    "context_is_admin": [["role:admin"]]
}

New rules allow separation of access control by method and expanded roles. Also compatible with Keystone v3 expanded functionality where domains are supported.

{
     "context_is_admin": [["role:admin"]],
     "admin_or_cloud_admin": [["rule:context_is_admin"],
              ["rule:admin_and_matching_project_domain_id"]],
     "telemetry:alarm_delete": [["rule:admin_or_cloud_admin"]]
}

Problem description

Ceilometer API currently supports all or nothing authentication for API calls.

This limits the functionality of the reporting API as managers of more than one project must either be given the admin role or re-scope each request to view multiple projects.

Use cases where this limits functionality include:

Resellers managing more than one project
Domain administrators managing more than one project
Support personnel who manage above cases

Proposed change

We propose to solve the problem by moving access control from calls to the ACL to applying a decorator to the API methods. Each publicly accessible API method would have a decorator pointing to a new RBAC module. The RBAC module decorator would use rules defined in policy.json to determine accessibility of methods by a caller.

This would allow fine-grained, role-dependent, method-specific access control.

It would also align Ceilometer API wit the Keystone V3 Domain capabilities as well as the hierarchical project proposal since both domain or project could be used in rule creation.

Alternatives

It would be possible to place an RBAC filter in front of the Pecan webserver. This filter would implement RBAC through calls to Keystone to verify roles, projects and domains.

While we believe this is a reasonable solution, it diverges from the way Ceilometer API is currently implemented. It would require insertion of an access control filter between the middleware and Pecan. It would also require significant code changes to the current RBAC scheme which handles access control within the API.

The proposed model will minimize code changes. It would simply add decorator statements to the external API methods, create an additional module, and add configuration changes to policy.json

Data model impact

None

REST API impact

None

Security impact

The new model would improve security because access control would become centralized in the decorator module. After ensuring each external method has a default decorator call, access control would remain as admin or project-only unless changed in the policy configuration file (policy.json).

The current model places calls to the ACL module in varying locations throughout the API code. For example, GET methods call _query_to_kwargs which eventually calls the ACL, and PUT methods call the ACL directly. This could lead to confusion on how to handle access control in new methods or how to change it for existing ones.

Security improvements include the ability to allow user/project combinations to be granted roles other than the powerful admin role and still accomplish meaningful activities. Removal of the maximum-privilege admin role and limiting access to the least possible set of operations is an improvement.

Possible security impacts involve the fact that this approach is more role-dependent. If complex rules are specified but the Keystone role granting privileges are not tightly controlled, there are more opportunities to grant unwarranted access to users.

In other words, with more roles and access schemes available there is more to manage.

We believe this risk is mitigated by the ability to ship the basic code with only the current context_is_admin rule enabled. Such configuration would not allow additional Keystone roles to grant new privileges unless the system operators explicitly added new rules to the policy file.

Pipeline impact

None

Other end user impact

This will have no direct impact on on python-ceilometerclient as roles and their associated rules would be established in keystone and interpreted by Ceilometer API. Nevertheless, the python-ceilometerclient will benefit from the increase security provided by the new policy support. For instance, collector agent (or any other ceilometer service) can have a special role associated with it disallowing other services (with admin status) to post data in the database.

If end users were to take advantage of the expanded RBAC capabilities, there would be end user impacts in securing appropriate user and project roles to match those defined in the policy file.

Performance/Scalability Impacts

None

Other deployer impact

By default there will be no impact during deployment. The change will ship with a configuration that preserves the current access control behavior. The operator can simply leave everything as is and expect the system to behave as it did before this change.

There will be an option to define and use different policy files if the operator wants to take advantage of the expanded access control capabilities offered by this fix. This fix will also offer compatibility with Keystone v3 features (again, optional not out of the box).

Deployment options:

Optional policy configuration changes to enable expanded access control
Change must be explicitly enabled to allow expanded access control, otherwise it defaults to current access control behavior.
No changes in the package deployment are required.
Existing policy definition files will continue to work as they currently do without any special changes.

Developer impact

Developers adding new Ceilometer API endpoints will need to add the appropriate access control mechanisms to exposed API methods.

Implementation

Assignee(s)

Primary assignee:: eap-x, fabgia
Other contributors:: eap-x, srinivas-sakhamuri
Ongoing maintainer:: fabgia, srinivas-sakhamuri

Work Items

Work items:

Implement RBAC validation module
Apply decorators to all external Ceilometer API methods (such as v2/meters, etc.)
Deprecate acl.py usage
Make policy.json rule additions if desired (optional)

Future lifecycle

The HP Ceilometer development team is actively interested in improving and maintaining API access control. We foresee a need by commercial cloud providers to configure access control for reselling and for private cloud management.

We would like to be actively engaged in cotinued development and maintenance of this feature for many cycles.

Dependencies

Keystone v3 adoption
No new libraries or programs required
No external dependencies

Testing

Existing API tests will ensure current access control functionality is preserved. New unit tests will cover expanded functionality.

Documentation Impact

Documentation around enabling the expanded RBAC features will be required.

References

Keystone V3 Policy: https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json

Ceph object storage meters

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ceph-ceilometer-integration

This spec proposes to add ceph object storage metering support to Ceilometer by polling mechanism, using the ceph radosgw (rados gateway) APIs, when the ceph is used as object storage, instead of swift.

Problem description

Currently, ceilometer doesn’t has the capability to get the meters from ceph object storage, when the ceph is used as an object storage, instead of swift object storage).

Proposed change

Ceph object storage (i.e ragosgw) provide a few REST APIs to get the meters like object/container list, object/container size, etc. Ceilometer can use these REST APIs to get the needful metering information.

In Ceilometer - polling mechanism is required, to get the meter details from the ceph object storage. This polling method could be similar to swift polling mechanism.

Add new pollster classes for ceph object storage to get object storage meter information into sample-list, using existing storage.objects implementation as a model used with swift object storage.

The ceph object storage poller names have the following pattern:

ceph.storage.objects
ceph.storage.objects.size
ceph.storage.objects.containers
ceph.storage.containers.objects
ceph.storage.containers.objects.size
ceph.storage.api.request

To implement the above - we will call the ceph object storage (i.e radosgw) REST APIs.

Alternatives

Ceph object storage can push the notifications for the meters to ceilometer, but currently ceph doesn’t support the push notifications support, so its a complex alternative to use ATM.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: swamireddy
Ongoing maintainer:: swamireddy

Work Items

Configure the setup (i.e setup.cfg) file based on the object storage used to get the metering information. If swift used, use swift collectors to get the metering information. If ceph used, use ceph collectors to get the metering information.
Implement the ceph object storage (radosgw) APIs calls for appropriately for each metering item. Here, the python swfitclient will be used to interact with ceph rados gateway.
Currently, ceph support the keystone authentication. To check if it works with ceilometer also.
Update/modify the ceph radosgw REST APIs appropriately, based on ceilometer requirements. But at present, planned to use the currently supported APIs from ceph radosgw, without major changes.
Add test coverage for the above pollster classes and meter samples validation.

Future lifecycle

In this spec, we have dealt with meters, which can be supported using the polling mechanism. But a few meters like object input/output bytes size need the push notifications support from ceph. If the push notifications supports in ceph, we add these meters also in ceilometer in future.

In the future new types of metering items are expected from the ceph object storage to account for the statistics data. These will need to be handled separately.

Dependencies

Ceph - radosgw REST APIs support required to do this. If ceph API is not supported, we may need to implement it in ceph - radosgw.

Testing

Unit tests will be added to cover the necessary pollster classes and validate them with appropriate meter samples generated.

The integration tests will be performed with help of 3rd party CI setup.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

None

Middleware-specific Ceilometer Package

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ceilometermiddleware

The swift middleware packaged in Ceilometer enables the generation of swift metrics which can be used to properly meter Swift activity. This cross project dependency causes issues during the upgrade chain.

Problem description

Currently the swift middleware exists in the Ceilometer package as it leverages the functionality of the pipeline. This process is rather heavyweight as it moves a lot of the processing (building samples, republishing to correct targets) to the Swift service where as it should really exist on Ceilometer’s. Additionally, having the middleware live in the Ceilometer package brings in a lot of additional dependencies unrelated to the middleware which is causing upgrade issues[1].

Proposed change

This spec is to propose branching off swift middleware into its own library: ceilometermiddleware. This work is similar to the keystonemiddleware library which split the auth_token middleware into its own library to avoid additional requirements of a larger package.

Additionally, the middleware will be modified to solely compute the required metric values as it does now and publish a single notification straight to the message queue rather than loading in the Ceilometer pipeline and pusblishing 3 separate samples as it currently does.

Alternatives

We can continue to publish via the pipeline but it is far too verbose and will not solve all dependency issues.
Have swift own its own metrics and have it exist swift package. This is dependent on swift accepting something not completely scoped to swift internal functionality.
Drop support of swift middleware meters (ie. we won’t test it but it’ll just existed if a deployer does want to try to use it)

Data model impact

We need to capture swift meters in notification agent.

REST API impact

None

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

There should be a performance increase we’d be using notifications which are async and we’d just be sending one notification (which would be picked up by a listener in ceilometer and parsed into the 3 samples we expect). this moves all the processing overhead to ceilometer’s services (where it should be)

Other deployer impact

None.

Developer impact

New middleware should be place in new library

Implementation

Assignee(s)

Primary assignee:: chungg
Other contributors:: osanai-hisashi
Ongoing maintainer:: chungg

Work Items

create ceilometermiddleware library
move edited swift middleware to new lib
add listener to ceilometer to pick up new notification and parse into three samples
create shell in middlware that exists in ceilometer to use new lib
update devstack to use new lib

Future lifecycle

look at statsd approach.

Dependencies

None.

Testing

Add new tests to ceilometermiddleware
continue to leverage existing tempest tests

Documentation Impact

reference new middleware

References

[1] https://bugs.launchpad.net/ceilometer/+bug/1403024

Declarative HTTP API Tests

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tests

This spec proposes a suite of “grey-box” tests for the Ceilometer HTTP API that are driven by one or several human-readable text files that declare URL requests alongside expected responses. These tests will provide end to end tests of API endpoints that in addition to confirming the functionality of the API will provide a cogent and readable overview of the HTTP API.

The suite of tests augments rather than replaces existing tests presuming that several small tools, each doing a focused job, are more effective than one job doing everything. In this case the balance of the focus is on traversing the breadth of the HTTP API to confirm it is in rude health and demonstrating expected behaviors, not validation of data provided by the API (unit tests ought to do that).

Transparency over the API ought to lead to better testing and ultimately better APIs.

Problem description

Existing Ceilometer HTTP API unit tests are enshrouded by and encumbered with Python code and traditional unit test infrastructure. These amount to noise when the purpose of a test is both to test and to make visible the testing of the HTTP API. What should be most visible is HTTP and the API. When these things are not visible, the effectiveness of the tests as tools for discovery by a developer who needs to make changes is limited.

The integration testing harness, Tempest, suffers from some of the same problems as above while also undergoing a refinement in purpose. Rather than testing all the things, the future Tempest will evolve to efficiently test the integration of most things. In this future projects which wish to affirm their systems in detail will need to do that testing themselves.

Tempest also more strictly requires that testcases have no knowledge of the internals. This proposal would be free of such constraints, which may prove convenient for test setup mechanics.

In addition, Tempest is now “branchless”, so that the same test branch is run against stable/{icehouse|juno} and master. This requires that tests for newly testable features either discover the availability of such features dynamically, or are explicitly configured to be skipped depending on the branch Tempest is being run against (neither option being ideal). This proposal would be free of such complications, as the tests would live in-tree and thus be branched in the same way as the rest of the codebase.

Proposed change

To address the shortcomings in HTTP testing (readability and completeness) a new target for tox will be defined that runs a suite of HTTP tests which are expressed as HTTP requests with expected responses. The exact format should be determined from discussion as these sorts of things always cause a lot of contention. Best to get that out of the way early.

This same suite of tests will also be registered as a check job to be run against patches proposed to the code review process.

In the ideal form the tests will run all the usual HTTP methods against every URL made available by all the active endpoints presented over HTTP by Ceilometer. This means that it should cover both the pending v3 API that will be presented by Gnocchi as well as the soon to deprecated older APIs.

To ensure the tests are lightweight and fast no web server will be used and only one of whichever lightweight datastore (mongo, sqlite3) for limited persistence. These tests are for testing the HTTP API aspect of Ceilometer and as such are not scenario tests: We don’t need to run these tests against every available datastore.

“No web server” can be achieved by using wsgi-intercept, WebTest or similar. The choice of tool should be driven by determining which is best able to facilitate easy translation from the declarative format to tests that can be handled by the runner.

The declarative format of the tests need to balance between easy for a human to write and read and easy for a computer to parse. There are two general classes of style:

Something that looks a bit like the output of curl -v. This has the advantage of looking just like HTTP on the wire but the disadvantage that it is hard to express partial expectations or partial requests.
A list of dictionaries that represent requests and expected responses. This is easy to express in a common format like YAML, can deal with partial information well, and handle a form of inheritance. The author has some good success with this format so tends to prefer it.

Assuming the latter format a possible implementation would include:

A test runner that parses YAML files to generate tests which will be captured by the testing harness. This runner, besides making the HTTP requests and processing responses, will be responsible for setup: establishing the necessary services and other prerequisites of the tests.
A directory including multiple YAML files each representing a transaction/scenario of some kind in the API or all the routes in one version of the API.

Note that this proposal makes no mention of client code because it is direct HTTP requests that are being tested, not calls to a client library. This is entirely the point: a client library hides HTTP and what we are trying to do here is expose HTTP interactions to the harsh light of review.

There are a few questions which will need to be resolved to reach a complete solution. It should be possible to resolve these in flight. The questions include:

What format for the tests?
How to handle authN and authZ in a minimal fashion? Ideally keystone won’t be running. If that’s the case, what needs to be done?
How to aggregate request and response groups into single tests? One option is for each YAML file to be treated as a single test.

Alternatives

There are several alternative options to a standalone suite of declarative HTTP tests within the Ceilometer code tree:

Propose a spec to qa for building a general purpose solution to the problem of declarative HTTP tests in-tree. Long term this would be a great outcome, but this outcome seems more likely if people are able to see a working version against which improvements can be made.
Don’t use a standalone suite, have the tests integrated with a larger set of in-tree functional and/or integration tests. While this may be a good idea from a management of complexity standpoint it has limited value in ensuring the tight focus of purpose that is required for something to be good at its purpose.
Don’t use declarative tests, instead continue with object-oriented unittests style that is the norm throughout many tests. Doing this will defeat the purpose of the proposal (exposing and expressing HTTP) so is not recommended.
Run the tests against various storage and web server scenarios to achieve maximum coverage across a variety of situations. While this may have value, it would lead to slow tests and increase complexity of the harness significantly. Slow tests discourage people running them and complexity leads to breakage.
Do nothing. Use existing unit tests and tempests test. This is not a good option as neither the existing unit tests nor Tempest tests present good (and readable) coverage.

Data model impact

None. The data model should be invisible to these tests.

REST API impact

These tests will not add to the API, but with luck will improve it.

Security impact

None.

Pipeline impact

None.

Other end user impact

Writing and running these tests may demonstrate that the API is confusing and hard to use which may then lead to making it better. This would be awesome.

Performance/Scalability Impacts

The additional gate check job may have some impact on performance there but these changes are not expected to impact the performance of Ceilometer itself.

Other deployer impact

None.

Developer impact

If a developer adds to or changes the HTTP API those changes will need to reflected in this test suite.

Implementation

Assignee(s)

Primary assignee:: chdent
Other contributors:: dbelova
Ongoing maintainer:: chdent

Work Items

Decide on the declarative format.
Determine extent or depth of testing (are web servers being used? other data store?).
Write harness.
Write first round of tests.
Integrate with tox and testr.
Create gate job.

Future lifecycle

As features are added and removed to and from the API tests for those features will need to be changed in this suite. Most of the time that should be done by the implementor of the feature but there will be times that larger cleanups are required.

Similarly if storage handling becomes a part of the test suite, then as new storage systems are implemented (or removed), they will need to be handled.

Dependencies

This work is self-contained but may add to the libraries required for testing (e.g. wsgi-intercept).

Testing

As a suite of tests, this system will test itself. It should, however, include some sanity tests within itself to make sure it is behaving.

Documentation Impact

None.

References

See links above and:

https://wiki.python.org/moin/PythonTestingToolsTaxonomy for prior art and potential formats.
http://lists.openstack.org/pipermail/openstack-dev/2014-October/049056.html related mailing list thread.

Dedicated Event Database

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/dedicated-event-db

Ceilometer provides the ability for metering, alarming, and eventing. As Ceilometer tracks more and more data, scalability issues will arise.

Problem description

Currently, metering and event data coexists on the same database. While related, there’s a logical separation in the metering and event models where the data in each model has its own unique data and structure; the metering model can be best described as a time series while the event model is closer to an entity attribute model. As the models are different in what they capture, it makes sense that deployers may choose to use different storage drivers to store each data set.

Proposed change

Similar to the work done to split alarming into its own database, this blueprint is to allow for event related data to be stored in its own database.

Alternatives

Continue on as status quo.

Data model impact

None, the model remains the same.

REST API impact

None, the api will remain the same. The only change is the api will now read from events database when event api is called.

Security impact

Policies and permissions will remain unchanged. The API will continue to function as before and give the same amount of access to data. To that effect, users will require an admin role to have access to data.

Pipeline impact

None.

Other end user impact

The end user will now be able to store event data in a different database from metering data. Alternatively, they can continue on as how Ceilometer currently functions and store event and metering data in the same database.

Performance/Scalability Impacts

This will probably improve scalability as it allows deployers to split event and metering data so a single database is not flooded by queries against disconnected data sets. There also remains the ability to write to the same db so the currently design/performance can continue as well.

Other deployer impact

A config option to specify an event database is added:

[database]
metering_connection=hbase://
alarm_connection=sqlite://
*event_connection=mongodb://*

Developer impact

Future event related work should be done in event submodule.

Implementation

Assignee(s)

Primary assignee:: chungg
Ongoing maintainer:: chungg

Work Items

Move event models to event submodule
Add support for event_connection option and create framework for drivers in event submodule
Move over event related code for each driver from storage module to event submodule

Future lifecycle

There is common code shared between alarm, event, and metering backends which can be refactored.

Dependencies

None

Testing

Existing testing should be sufficient. The only additional tests required are test the ability to define separate backends for each data set (alarm, event, metering) and to ensure the capabilities of each backend return the correct set of capabilities.

Documentation Impact

We need to update docs to highlight how to enable event specific backend.

References

dedicated alarm database blueprint: https://blueprints.launchpad.net/ceilometer/+spec/dedicated-alarm-database

ElasticSearch backend for Events

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/elasticsearch-driver

As we split our backends to store segregated data (alarm, metering, events), we have to ability to choose backends tailored to store said data.

Problem description

While SQL is able to store unstructured data, it’s true use case is to effectively store data in a defined schema and it’s relationships so that it’s easily queryable.

Events in OpenStack are for the most part schema-free; each notification message may contain any combination of attributes based on when and where the message originated from.

Proposed change

ElasticSearch is designed primarily to store unstructured real time data flows, similar to the events in OpenStack and has worked with relative success in other projects[1].

ElasticSearch will be added in as an alternative backend to the current offerings of HBase, MongoDB, and SQL. The current implementation of filtering attributes using a definition file will continue to be used. A Logstash implementation of capturing and processing events is not in the scope of this blueprint[2]

Additionally, the api will continue to match the events api currently offered rather than implementing logstash.

Samples backend is not in the scope of this patch as TSDBs offer arguably better support for capturing measurements. That said, alarms database could be feasible and may be included in subsequent patches.

Alternatives

As mentioned, existing solutions using MongoDB, HBase, and SQL exists. They will continue to be viable options.

Using Kibana to retrieve/analyze data will not be the default solution to query data. That said, deployers should be able to bypass Ceilometer’s api and use Kibana if desired.

It should be noted, ElasticSearch currently isn’t recommended as a primary storage engine [3][4]. There is debate around how consistent it is.

There is another solution to use Mongodb as a primary storage and pipe data to ElasticSearch for better querying.[5]

ElasticSearch parallels MongoDB as it is also based on JSON. The reason for having ElasticSearch as an alternative is because it allows us to create indices based on time so we can effectivley shard data as well as expire data. Also, in addition to INFO notifications, services also emit ERROR notifications which can be richer in textual information, and ElasticSearch is designed specifically for such cases.

Data model impact

No data model changes are required. Events will continue to have the following attributes:

* message_id
* event_type
* generated
* list of traits

REST API impact

None

Security impact

Nothing new

Pipeline impact

None

Other end user impact

None, unless ElasticSearch is chosen as the backend. Then ElasticSearch will need to be configured accordingly.

Performance/Scalability Impacts

None, as it’s an optional backend. ElasticSearch has the ability to cluster to provide HA and it is built to scale horizontally[6]

Other deployer impact

The ElasticSearch storage driver is to have feature parity with the rest of the currently available event driver backends. It will add an elasticsearch-py client dependency should the driver be selected.

Developer impact

Devs will need to use ElasticSearch client when modifying ElasticSearch driver

Implementation

Assignee(s)

Primary assignee:: chungg
Ongoing maintainer:: chungg

Work Items

write equivalent event driver functions and test.

Future lifecycle

Depending on evolution of events in Ceilometer, ElasticSearch driver will need to be updated to support new features (if feasible/logical). ElasticSearch may be extended to cover alarms (and samples) but is not currently in scope because time series databases are probably a better solution for measurements.

Dependencies

elasticsearch-py client library

Testing

testing will need to be done against a mocked db or a real ElasticSearch database

Documentation Impact

Update driver docs

References

[1] http://www.elasticsearch.org/case-studies/ [2] http://logstash.net/docs/1.4.2/ [3] http://www.bigdatamontreal.org/?p=305 - elasticsearch employee [4] http://aphyr.com/posts/317-call-me-maybe-elasticsearch [5] http://stackoverflow.com/questions/20080189/index-mongodb-with-elasticsearch/20120927#20120927 [6] http://www.elasticsearch.org/overview/elasticsearch

Adds disk IOPS metrics implementation in the Hyper-V Inspector

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/hyper-v-disk-iops-metrics

High latency between I/O requests can be signs of issues. Collecting disk latency metrics can help us detect those issues. Hyper-V Inspector can collect those metrics.

Problem description

Currently, when inspecting disk metrics, the Hyper-V Inspector cannot post any values for read_requests or write_requests, since this kind of metrics is not supported by Hyper-V. Instead, the Hyper-V Inspector can collect disk IOPS metrics, which will give users a useful alternative.

Proposed change

Add DiskIOPSPollster and PerDeviceDiskIOPSPollster pollsters, which create samples having the properties: * name: ‘disk.iops’ * type: ‘gauge’ * unit: ‘count/second’

Create the named tuple DiskIOPSStats, containing the field ‘iops_count’.

Add the method ‘inspect_disk_iops’ in the Inspector and implement it in the HyperVInspector. The method will return a DiskIOPSStats object.

The metric value will be fetched from Hyper-V VMs, located in the Msvm_AggregationMetricValue object (further referred to as metric object) associated with the VMs. The metric object’s ‘MetricDefinitionId’ must be equal to the ‘Id’ of Msvm_AggregationMetricDefinition object having the Caption ‘Average Normalized Disk Throughput’.

Hyper-V disk metrics were introduced in Windows / Hyper-V Server 2012 R2 (kernel version 6.3). They are not supported in the previous versions.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

Users will have to add the meter ‘disk.iops’ in the disk_source source in pipeline.yaml instead of ‘disk.read.requests’ and ‘disk.write.requests’.

Having ‘disk.read.requests’ and ‘disk.write.requests’ in pipeline.yaml has no effect, the HyperVInspector will return zeroes for those metrics since Hyper-V cannot collect those metrics, hence the need for ‘disk.iops’ metrics.

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

By default, the disk usage metrics collection is enabled in Nova, we just need to collect the data from the Hyper-V API.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <cbelu@cloudbasesolutions.com>

Work Items

Adds the DiskIOPSPollster and PerDeviceDiskIOPSPollster pollsters
Adds the ‘iops_count’ in DiskStats.
Adds the ‘DiskIOPSStats’ named tuple, having the field ‘iops_count’.
Adds the ‘inspect_disk_iops’ method in the Inspector and implement it in the HyperVInspector. This method will return a DiskIOPSStats object.
Adds related unit tests.
Updates ceilometer measurements document.

Future lifecycle

Once this feature is enabled, it needs tests and bug fixing in the next 2 releases to avoid regression.

Dependencies

None

Testing

Unit tests are needed to test the new pollsters and the implementation on the Hyper-V side.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

None

Integrating Kafka Publisher into Ceilometer Publisher

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/kafka-publisher

Apache kafka is messaging broker which enables the ability to publish real-time metering data to applications outside of OpenStack Projects with low-latency and high-throughput.

Problem description

Ceilometer monitors and collects data from OpenStack via notifications sent from existing services or by polling the infrastructure. These real-time collected data have great potential to be used for various kind of analysis. There are many external projects which analyze, serialize, and visualize these metering data, for example, Storm, Elastic Search, Mahout, Jubatus etc. In OpenStack, oslo.messaging library provides nice communication paths, however these communication paths are not so general especially in external to OpenStack projects. Thus, this blueprint is trying to publish ceilometer metering data to such external projects using Kafka, distributed messaging system, and achieve the full potential of ceilometer metering data.

Proposed change

To reduce the cost of implementing a Kafka publisher independently, Ceilometer has to provide the kafka publisher as a publisher plugin. This automates publishing metering data, and developers only have to configure the kafka publisher plugin in pipeline.yaml file like:

publisher:
    - kafka://<broker_ip>?topic=<topic_name>

This way any application that is trying to consume streaming ceilometer metrics via Kafka, can directly consume the ceilometer samples. For example, projects like monasca - https://github.com/stackforge/monasca-thresh can consume the ceilometer metrics that are published by the Ceilometer Kafka publisher.

Alternatives

Inside of the OpenStack, to communicate with each modules is achieved by oslo.messaging, however defacto usage of oslo.messaging is for OpenStack projects and Kafka is supporting for more general usages.

Data model impact

None

REST API impact

None

Security impact

Current security support of Kafka is none. Authentication of clients and ecrypting connections are under implementation. More information can be found in the references.

Pipeline impact

Provide new options to specify kafka publisher as a ceilometer publisher

Other end user impact

None

Performance/Scalability Impacts

The paper related to Kafka says that producers can publish about 50,000 messages/sec on the condition that message size is 200 byte and messages are sent one by one. However, this number is affected by parameters such as message size, the number of replication, batch size (kafka can send multiple messages together), and environments. Roughly, RabbitMQ can publish about 10,000 messages/sec, it means this kafka publihser might not be the bottleneck of performance.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <kshimamu>
Other contributors:: <yudupi>
Ongoing maintainer:: <kshimamu>

Work Items

adding a kafka publisher as a ceilometer publisher

Future lifecycle

None

Dependencies

Kafka python package

https://pypi.python.org/pypi/kafka-python

Testing

None

Documentation Impact

None

References

Apache Kafka Project

http://kafka.apache.org/

Kafka: a Distributed Messaging System for Log Processing

http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf

Kafka Security

https://cwiki.apache.org/confluence/display/KAFKA/Security

KAFKA 0.8 PRODUCER PERFORMANCE

http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/

Performance of Kafka

http://kafka.apache.org/07/performance.html

RabbitMQ Performance Benchmarks

http://blogs.vmware.com/vfabric/2013/04/how-fast-is-a-rabbit-basic-rabbitmq-performance-benchmarks.html

Track Network Services Notifications

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/network-services-notifications

The goal of this blueprint is to capture the notification events emitted by neutron network services during create, update and delete operations.

Problem description

Neutron emits notifications from the data provided by the network services plugin and agent components. These in particular are for FWaaS, LBaaS and VPNaaS. Ceilometer needs to be updated to consume and record these notifications. If Ceilometer processes these notifications other services will be able use queries and alarms to monitor and scale as required.

A notification plugin is required, in Ceilometer, to hear notifications at an exchange and topic and transform them into samples. We can leverage the existing Network Notification plugins and build on top of it.

Proposed change

Add a new listener classes for Network Services to transform neutron event data into samples, using existing NetworkNotificationBase implementations as a model. These listeners include create, update and delete events emitted by neutron for specific components of firewall, VPN or a Loadbalancer.

The Neutron event topic names have the following pattern,

<resource>.create.start
<resource>.create.end
<resource>.update.start
<resource>.update.end
<resource>.delete.start
<resource>.delete.end

For this implementation we will track the end events as those give us the most information with respect to event payload. Also end events are more informative to track the existence of a resource and its usage over time.

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

No new impacts. Pre-existing concerns with capacity at the notification and storage handling layers remain.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: pkilambi
Ongoing maintainer:: pkilambi

Work Items

Load Balancer Listener class to process create/update/delete events for pool, vip, member, health_monitor.
Firewall Listener class to process create/update/delete events for firewall, policy and rule.
VPN listener class to process create/update/delete events for vpnservice, ipsec_policy, ike_policy, ipsec_site_connections.
Test coverage for the above listener classes and sample validation.

Future lifecycle

In the future new types of notifications are expected from the Neutron services to account for the statistics data. These will need to be handled separately.

Dependencies

None.

Testing

Unit and integration Tests will be added to cover the necessary notification listener classes and validate samples generated.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

Rally Gate Check

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/rally-gate-check

Rally provides support for what it calls rally-gate jobs. These automate a set of non-voting, custom benchmark scenarios that effectively make it possible to observe the performance impact of changes. Ceilometer’s overall performance is the result of complex interactions in several systems, automating benchmarks of these systems will save time and provide useful information.

Problem description

At this time there are no generally available automated benchmarks for Ceilometer performance. Both developers and deployers would appreciate knowing if code changes will improve or damage performance.

Proposed change

A non-voting gate check job will be created to run a set of benchmark scenarios against each Ceilometer patchset. This job is controlled through multiple changes:

A rally-scenarios directory in the Ceilometer code tree containing a ceilometer.yaml file that describes the Rally scenarios to be used in the job.
An addition to projects.yaml that adds rally-jobs to the jobs list.
An update to zuul layout that adds gate-rally-dsvm-fake-ceilometer to the check list of the openstack/ceilometer section.

Rally includes a suite of pre-built scenarios which can run against Ceilometer. At the moment these primarily interact with Ceilometer over the API. This is not a requirement. Scenarios can do as much or as little as required and can be added as necessary as plugins in the rally-scenarios directory. As long the code is properly decorated by Rally code, the scenarios can be measured.

Initially this change will implement the basic structure for doing benchmarks, using simple API tests to flesh out and confirm the setup.

Once confirmed, additional scenarios will be created to measure:

Latency in polling and notification agents (getting samples into the pipeline).
Latency in ingestion (getting samples into the data store).

These latter measurements will require more complex scenario code that is responsible for creating and destroying VMs, volumes and other resources.

Scenario code which proves effective should migrate from plugins to the Rally codebase to allow general use.

Alternatives

Alternatives include:

The Rally team is automating performance checking to gather historical performance data to share with projects. This serves a similar but not identical purpose to the changes propsed here: it would help to evaluate the improvement of Ceilometer at large but not specific changes.
Use the Rally job as an experimental job, only run when a “check experimental” comment is left in gerrit.
Once this initial spec has been implemented and the principles involved validated in the ceilometer context, it may be worthwhile to implement what are called “SLA checks” via Rally. These allow a check job which can fail based on a severe degradation in performance. These are not being done now to first gain experience with Rally as well as to compartmentalize work in actionable chunks.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

Some end users may be able to use the benchmark information to make decisions about how they configure their installations. It seems unlikely, however, that they will want to review this data from gerrit.

Performance/Scalability Impacts

These tests will provide a safety net to guard against degradations in performance and provide data to identify which changes have caused issues if there is degradation.

Other deployer impact

None.

Developer impact

As features are added, the Rally scenarios may be insufficient and need to be updated. Ideally the scenarios will be high-level enough that this will be rare. When they are not it will be up to the feature developer to update the Rally scenarios and, as necessary, add Rally plugins.

Implementation

Assignee(s)

Primary assignee:: chdent
Other contributors:: boris-42, rediskin, sileht
Ongoing maintainer:: chdent

Work Items

In addition to the steps listed in the Proposed Solution section above the major chunk of work is in selecting and/or creating the necessary Rally tasks. Rally itself includes many sample scenarios and runners and includes a gate check in its own tree which includes several Ceilometer checks that may be used as starting point.

Future lifecycle

As stated in “Developer Impact” new features may require new scenarios.

Dependencies

Rally.

Testing

The proposal adds a new (non-voting) gate job which will act as its own test.

Documentation Impact

None.

References

An existing related patchset: https://review.openstack.org/#/c/132649/

API No Pipeline

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/api-no-pipeline

The ceilometer API server allows new samples to be posted via the API. These are then processed through the pipeline and then dispatched as required. This means the API server must use the pipeline code and have a pipeline.yaml file, raising the complexity of the code and the testing and install profile of the API. An alternative would be for the API server to publish notifications for the meters it receives which would then be processed by the notification agent (which has a pipeline of its own) and then dispatched.

Problem description

Including the transformation pipeline in the API server increases complexity on at least three dimensions for a single feature in the API (creating samples) which is not commonly used (most samples are produced via polling agents and notifications).

The use of the pipeline creates dependencies in code, testing and installation that could be avoided. In code we must include the pipeline modules and classes. In testing and installation we must have a pipeline.yaml.

In the installation case, if there are multiple API servers, the pipeline.yaml must be kept in sync across the hosts on which those servers are installed. That’s an unnecessary burden.

Proposed change

One way to resolve this complexity is to not use the pipeline in the API server at all. Instead when new samples are posted, format them as notifications and push them onto the notification bus to be retrieved by the notification agent. The pipeline within that agent will then do any expected transformations.

For convenience of testing, an optional flag will be add that can enable posting samples directly to storage to avoid notification-agent process.

Alternatives

An alternative which accomplishes the state goal would be:

A more complex solution would be create notifications on the bus that are targeted to a specific transformation service. Such a service would only listen for ceilometer generated measurement notifications, transform them according to a pipeline description and then dispatch them onward to storage or other processing. This model would skip the notification agent entirely.

Pro: extends the long term plan of decomposing to smaller pieces. Con: adds to the complexity of this change, increasing time to completion and risk of error.

Another alternative which are related but not quite the same include:

The issues with testing of the API requiring a pipeline.yaml could be addressed by having a code-based set of pipeline defaults which are used in the absence of a pipeline.yaml file (and could also be used to generated the default pipeline.yaml).

Data model impact

The proposal does not impact the structure of the data, but it would impact the flow of the data. Testing will be required to see what kind of impact this will have on the timeliness of data acquisition.

REST API impact

Samples created via the API will take a slightly different path between the API and their eventual storage.

An optional boolean parameter ‘direct’ will be added to the post-samples API that allow users posting samples directly to storage and avoid notification agent processing(will skip pipeline), this is mainly used for testing.

Security impact

None beyond existing concerns.

Pipeline impact

Pipeline transformations that had been happening in the api server will happen in the notification agent.

Other end user impact

None.

Performance/Scalability Impacts

This will potentially introduce some latency in the time between when a sample is posted to the API and when it lands in the data store.

Other deployer impact

Deployers of the API server will no longer need to deploy a pipeline.yaml alongside the API server.

And deployers need to config the ‘notification_driver’ in ceilometer.conf if need posting samples through notification-agent, such as:

notification_driver = messaging

Developer impact

This change may increase the amount of asynchrony in some tests. For example, the time between posting a sample and being able to retrieve that sample may become more unpredictable. As it is already unpredictable any tests which rely on immediate retrieval are bad anyway, so we should fix that.

Implementation

Assignee(s)

Primary assignee:: liusheng
Other contributors:: chdent
Ongoing maintainer:: liusheng, chdent

Work Items

Ensure existence of robust tests of posting meters that are transformed by the pipeline.
Replace pipeline-based publishing in the API code with a notifier.
Update API tests to deal with things like mocked pipelines that no longer exist.
API option to perform direct writing to storage

Future lifecycle

The members of the Telemetry program will do any required ongoing maintenance for this feature.

Dependencies

No new dependencies.

Testing

In addition to existing unit tests of the API, the new declarative http tests can be used to confirm that posted samples continue to have expected transformations.

Documentation Impact

Deployment documentation may need some slight updates to indicate the change in configuration file requirements for the api server.

References

None

Add the function of deleting alarm history

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/delete-alarmhistory

Currently the ceilometer-expirer doesn’t delete expired AlarmChanges. Remained AlarmChanges would be cause of wasted disk-space and slow response. This blueprint adds the function of deleting alarm history.

Problem description

Currently the ceilometer-expirer doesn’t delete expired AlarmChanges. So we need to add the function of deleting alarm history.

For doing so, a time-to-live (TTL) value needs to be specified. Currently the metering_time_to_live (aka time_to_live) value is used for deleting metering data. However, alarm history expirations shouldn’t be the same frequency as sample expiration.

Therefore, separate TTL value needs to be introduced for alarm history.

Proposed change

Add the function of deleting alarm history. This functionality will be put into the ceilometer-expirer.

New TTL, alarm_history_time_to_live, is added as a config option to database section.

Alternatives

It is possible to use the same time_to_live value for both sample and alarm history.

However their scale might be completely different, so the expiration frequency shouldn’t be the same.

Therefore, we will have separate TTL.

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

A new option should be set if deployer wants to enable this feature.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

honjo-rikimaru-c6
ZhiQiang Fan

Other contributors:

None

Ongoing maintainer:

ZhiQiang Fan

Work Items

“alarm_history_time_to_live” is added as a config option.
support mongodb back end
support sqlalchemy back end

Future lifecycle

None

Dependencies

None

Testing

This code will be tested in unit tests for expire.

Documentation Impact

None

References

[1] https://blueprints.launchpad.net/ceilometer/+spec/delete-alarmhistory

[2] bug report that addresse this issue https://bugs.launchpad.net/ceilometer/+bug/1289141

[3] ongoing patch for [2] https://review.openstack.org/#/c/87869/

Existence Meters Deprecation

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/deprecate-existence-meters

Problem description

As Ceilometer has expanded to capture more notifications from the OpenStack message ecosystem, the number of samples it generates grows even faster as many samples are derived from a single notification or polling request.

The “Existence of xyz” meters we store in our samples database represents a significant portion of the data we store. These meters however offer no useful measurement value and it’s true value is to capture the state of the resource at a given time – a value that is also available in the meters generated alongside the existence meters. Additionally, the volume=1 value is very confusing to consumers of data as for a while, Horizon used that value as the total number of values and often users would wonder why there was always 1 record of an instance, network, port, etc…

As we move to a more time-series focused storage for samples, the “volume=1” meters we collect has not just an impact on storage size, but also the overhead of rolling up and computing statistics on something as trivial and meaningless as the constant 1. Additionally, the rollup of samples will diminish the value of said meters as valid auditable datapoints.

At a high-level, Samples are the children of Events. Samples are a derived subset of an Event. Because of that, the Samples we create should capture an explicit datapoint of interest from an Event and not just be a shadow of an Event.

Proposed change

The “Existence of xyz” and “volume=1” meters can be better represented as what they really are: Events. The core event functionality was implemented in previous cycles and should be expanded to better cover these non-measurement meters.

This will offer better querying of data in Events, less storage of data in general, and less confusion as to what the volume attribute in a sample represents.

The proposed solution is to:

1. Mark 'volume=1' meters as available as Events in the documentation and
   add them to the event_definitions file. The vast majority of them are
   created from notifications so no work needs to be done aside from adding
   them to event_defintion. **COMPLETED IN KILO**
2. Change default pipeline to not enable these meters. NOTE: this does not
   affect existing solutions nor will it break anything. An option was added
   in Kilo to enable/disable meters. **COMPLETED IN KILO**
3. Some meters such as network related meters (LBaaS, VPNaaS, SDN,
   etc...) come from pollsters. We will need to convert these pollsters to
   republish information as events.  An example would be health check polls
   made for network meters[5]. Instead of creating a sample, it would create
   an event.
4. Henceforth, Samples will be measurements which can be aggregated and
   rolled up in meaningful ways (within gnocchi). Events will be the raw
   base that Samples come from and they should be stored accordingly. Both
   are time series driven data.

A rough list of meters to be dropped can be found at the correspond bug[1].

Alternatives

None. We could leave step 3 as optional as they are essentially healthcheck or existence events. We can also support a tool to migrate ‘non-metric’ meters to events but this is arguably not important.

Data model impact

None. Events already exist. This may have a side dependency on adding alarming on Events to maintain feature parity.

REST API impact

None. we may want to expand Events but nothing is directly impacted here.

Security impact

Audit data. This is an issue that already exists and will just be migrated from Samples to Events. It should be possible to add a tag to the event_definition file to mark data as audit. It is technically possible to redirect audit data to a separate db using event pipeline[4]

Pipeline impact

None.

Other end user impact

Users STILL have the option to keep these meters but we should advocate they use Events instead. As of Liberty, these meters will be disabled by default and remove completely as of M*. This has been documented in docs already.

Performance/Scalability Impacts

Less sample data. Less equivalent data in Events (because of trait filtering). Events are a bit more scalable in its design (maybe not the SQL backend).

Other deployer impact

We should turn on store_events option by default. This will probably require deployers to use event_connection option as it probably isn’t advisable to store sample and event data together but they could.

Developer impact

Learn events. Make events better.

Implementation

Assignee(s)

Primary assignee:: chungg
Ongoing maintainer:: chungg

Work Items

add pollster to Events functionality
turn on store_events by default

Future lifecycle

add alarming to Events.[3]

Dependencies

None.

Testing

None.

Documentation Impact

mark meters in measurement as available as Events and add a link to events.

References

[1] https://bugs.launchpad.net/ceilometer/+bug/1384874 [3] https://blueprints.launchpad.net/ceilometer/+spec/notifications-triggers [4] https://bugs.launchpad.net/ceilometer/+bug/1376915 [5] https://github.com/openstack/ceilometer/blob/master/ceilometer/network/services/fwaas.py

Using aggregation pipeline instead of map-reduce job in MongoDB

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/mongodb-aggregation-pipeline

Problem description

Currently, when we make a GET “/v2/meter/<meter_type/statistics” with MongoDB backend it starts a native map-reduce job in MongoDB instance. Tests and deep researching show that the job have a lack of performance in work with huge amount of samples (several millions and above). For example, job processes ~10000 samples per second on my test environment (16 GB RAM, 8 CPU, 1 TB disk, 15000000 samples). So job for 15M samples works ~1500 seconds. It’s longer than default api timeout, 1 minute.

Of course, with Gnocchi dispatcher we haven’t issue with statistics, but users which are going to use only MongoDB backend will have troubles with alarm work and making user reports.

Proposed change

Add an implementation of method get_meter_statistics via MongoDB aggregation pipeline framework.

From MongoDB docs: “This framework modeled on the concept of data processing pipelines. Documents enter a multi-stage pipeline that transforms the documents into an aggregated result. The most basic pipeline stages provide filters that operate like queries and document transformations that modify the form of the output document. Other pipeline operations provide tools for grouping and sorting documents by specific field or fields as well as tools for aggregating the contents of arrays, including arrays of documents. In addition, pipeline stages can use operators for tasks such as calculating the average or concatenating a string. The pipeline provides efficient data aggregation using native operations within MongoDB, and is the preferred method for data aggregation in MongoDB.”

My researches show that aggregation pipeline is faster than native map-reduce job to ~10 times. So processing of 15M samples in the same test environment works 128 seconds vs 1500 seconds with map-reduce.

Pipeline aggregation framework has a large functionality and amount of operators, that allows to provide support of all existence “statistics” features.

This implementation affects only performance of statistics request in Ceilometer MongoDB and doesn’t affects API or different backends.

Risks:

This framework have specified limits. It restricted by 100 MB RAM for stage otherwise it needs to write temporary files with intermediate stages results to disk. For avoiding failing caused by excessive memory using in MongoDB>=2.6 we can use the option allowDiskUse=True in aggregation command . This option allows to write intermediate staging data to temporary files. So, primary risks of this approach are a necessity of free space on disk and a slow performance of disk writing and reading.

Accordingly, researches and MongoDB docs, the “$sort” command creates the most amount of intermediate data for follow stages. So, in practice this stage prepares data whose size is close to new index size. In same time, the indexed fields sorting (like timestamp in our meter collection) does not need the any additional data in the disk. This request uses the existing index for sorting. Other commands works with processed and grouped data and use additional space only in worst case (huge amount of resources and group by resource_id in one request).

Despite to writing temporary file into disk the aggregation command in this case faster than Map-Reduce up to several times.

Also this MongoDB mechanisms have limit in size of finally document in 16 MB, same as map-reduce job.

Alternatives

Also we may improve performance of map-reduce job

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Improve performance of GET “/v2/<meter_name>/statistics” request.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ityaptin
Ongoing maintainer:: idegtiarev

Work Items

implement a get_meter_statistics function with aggregation pipeline framework

Future lifecycle

None

Dependencies

None

Testing

current tests are check correct work of “statistics” request

Documentation Impact

None

References

http://docs.mongodb.org/v2.4/core/aggregation-introduction/

Pollsters No Transform

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/pollsters-no-transform

The ceilometer polling agents currently perform two roles: They poll various services to generate samples and they optionally transform those samples, using pipelines defined by pipeline.yaml, to compose other samples eventually publishing them as measurements. This latter transformation functionality raises the complexity of the pollster code. An alternative would be for the pollsters to send notifications for the meters it creates and have them be processed by the notification agent (which has a transformation pipeline of its own) and then be published.

Problem description

Including the transformation pipeline in the pollsters increases complexity in the pollsters and duplicates functionality found in the notification agent.

Proposed change

One way to resolve this complexity is to not do transformations in the pollsters. Instead when new samples are polled, format them as notifications and push them onto the notification bus to be retrieved by the notification agent. The pipeline within that agent will then do any required transformations.

Besides clarifying the focus of the polling agents it is likely this change will make it easier to test and build performance improvements both in code and in deployments. This is murky speculation but it is challenging to test the theory without doing the work.

We also get an opportunity to review and audit the polling code, which is a useful thing to do on a regular basis.

Alternatives

The primary alternative is to leave things as they are. Obviously if we do this we lose the benefits described above.

Data model impact

REST API impact

No direct impact on the resources presented by the API.

Security impact

None beyond existing concerns.

Pipeline impact

Describing pipeline impact is somewhat complex because the word pipeline can be used:

to describe the overall process whereby a suite of samples is gathered and transformed
to identify the pairing of a source with a sink in the pipeline.yaml file
to point at the file pipeline.yaml which configures many aspects of gathering, transforming and publishing samples as well as transforming and publishing samples that have been produced from notifications

In this change transformation and publishing will be localized in the notification agents. Gathering and producing of samples will be done by polling agents, the api server (via posting meters) and the notification agents.

In order to ease migration the polling agents will continue to use the pipeline.yaml file to configure the sources they will poll and the intervals at which that polling will be done. The same file may be used, or a new file without any sinks. The idea is that both should work.

The most significant difference is that the existing validation which confirms that there is a pairing between source and sink will be removed. The polling agents may poll whatever they like. If the notification agent doesn’t want the samples that are generated it can simply drop them on the floor. This provides two benefits:

it makes the pipeline (on the polling side) less complex
it focuses authority in the notification agent

Other end user impact

None.

Performance/Scalability Impacts

There is hope that this will decrease the footprint of the pollsters making it easier to deploy many of them, thus increasing the overall performance of the polling system.

On the negative side there could be increased load on both the messaging and the notification agents but this could be compensated for by:

Having the pollsters publish their notifications on their own topic or even bus.
Deploying more notification agents (in a horizontal fashion).

Other deployer impact

Deployers will need to be aware of their options for deploying suitable numbers of notification agents.

Developer impact

Testing sample transformation from point of sample creation to point of storage will become more complex as it will traverse several services. With the advent of in-tree functional testing there is a place where this testing can and should be done. Having those tests will be very helpful in the long run despite the initial pain in setting them up.

Implementation

Assignee(s)

Primary assignee:: chdent
Other contributors:: you and all your friends
Ongoing maintainer:: chdent

Work Items

Ensure existence of robust tests of both polling and notification agents.
Replace source and sink handling in the pollster code with a simpler read of the source descriptions in pipeline.yaml (leaving open the future option of using a different file). Create samples will be republished as notifications.
Update documenation to reflect new architecture.

Future lifecycle

The members of the Telemetry program will do any required ongoing maintenance for this feature.

Dependencies

No new dependencies.

Testing

Testing as described above.

Documentation Impact

Deployment documentation will need to be updated to reflect the connection between the polling and notification agents, the option to use a custom messaging bus and other opportunities for customization.

References

None at this time.

Dynamic pipeline configuration using File Reloading

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/reload-file-based-pipeline-configuration

Currently there is no way to enable or disable meters without restarting ceilometer. There are cases where operators do not want to run all the meters continuously.

By enhancing Ceilometer to monitor pipeline file configuration, the application could be made to dynamically activate/deactivate meters, collection targets or similar functions, have distinctly different configurations for multiple nodes in different environments (i.e. dev/test/prod or HA scenarios) and could ultimately be updated with new collection targets “on-the-fly”.

Problem description

Currently, Ceilometer relies on pipeline configuration file for determining run-time parameters used in polling and notification handling.

There is no way to enable or disable meters without restarting ceilometer. There are cases where operators do not want to run all the meters continuously. In these cases, there should be a way to disable or enable them dynamically (without restarting ceilometer).

Meters in ceilometer are enabled by adding them in “setup.cfg:entry_points” configuration file and restarting ceilometer agent.

When ceilometer agent restarts, during initialization it reads the entry_points and creates and loads all pollster objects corresponding to meters present in setup.cfg.

There are disadvantages with this implementation:

Ceilometer might be running several hundreds of meters continuously and restarting it might impact other meters as it involves the deletion and re-creation of the hundreds of pollster objects, when the need is to poll a few meters or avoid polling a few meters.

The subsequent sections will cover an approach to dynamically reload and update agent pipeline configuration. (pipeline.yaml)

Proposed change

Summary:

File polling by agent - Each agent polls its local pipeline.yaml file for changes and re-configures pollsters/listeners on detecting a change.

By default, this will be turned off. Its an optional feature and can be turned on by specifying reload_pipeline_config = True in ceilometer.conf under the default section.

The proposed high-level flow:

Each agent daemon polls local pipeline configuration file at configurable interval.
The agents will reload and validate the configuration.
If validation succeeds, the new configuration is used.

Next steps for central-agent, compute-agent and ipmi-agent:

For polling agent, determine which pollsters are needed based on new configuration and need to run at what frequency (polling interval)
For polling-agent, all pollsters in flight are allowed to gracefully complete.
In HA mode, re-initialize the partition coordinator with new groups
Start the new set of pollsters.

Next steps for notification agent:

Existing listeners are stopped to allow for graceful termination.
Configure new listeners with the changed pipeline configuration.

Next steps for HA notification agent:

Pipeline listeners(pipeline sink-based internal queue listener) are gracefully terminated.
Pipeline listeners are re-configured with new pipeline.

Pros:

Preferred and Easiest approach

Cons:

It doesn’t centralize the pipeline definition and runs the risk of agents diverging on their pipeline definitions

This means we’re allowing any kind of error levels due to the fact file might be changed for one agent, and not changed for another one in the same coordination group.

Alternatives

File polling by separate daemon - Each agent uses a separate daemon that polls the pipeline.yaml for changes. On detecting a change, it signals (HUP) the agents to re-load the pipeline configuration.

Pros: Allows for future extensibility while keeping the agent to pipeline file contract unchanged. Cons: Same as Previous approach HA for the new daemon needed Using SIGHUP a potential loophole? (since anyone can send a SIGHUP and have the agent reload its configuration) More work for less gain
HUP-based pipeline reload by agent - The agent uses signal handler to handle the HUP signal and reloads pipeline from the file in response to the signal. The assumption on receiving a signal is that the pipeline file has changed, so the pipeline file can be changed manually or using config management tools - Chef/Puppet.

Pros: No API. Cons: File synchronization within coordination group is deployer responsibility Same as Approach 1 Using SIGHUP a potential loophole? (since anyone can send a SIGHUP and have the agent reload its configuration)
Use automated deployment tools - Puppet, Chef, Ansible to change pipeline definitions. While this automates changing pipeline definitions across multiple agents, it doesn’t bring the value-add of on-the-fly updates to the agent, without incurring a restart of the daemons.

Pros: No API. Cons: Is looking like the only pure admins approach. Will be more tricky for everyone not familiar with these tools (devs, DevOps, etc.) Not in ceilometer scope.
Use a combination of 2 and 3 - Use configuration management tool to automate the change in pipeline configuration across multiple remote agent nodes. Do not restart any processes though. Agents on each node poll the pipeline file and on detecting a change in the next polling, will update the pollsters and listeners. This approach has a dependency with ops on the config mgmt tools.

Pros: No API. Cons: File synchronization within coordination group is on deployer responsibility

Data model impact

REST API impact

Security impact

Pipeline impact

The use of the pipeline configuration will change from static to dynamic. This will affect the supported meters and their datapoint collection.

The impact to the system is expected to be minimal since changes to pipeline are expected to be low in frequency.

Other end user impact

Performance/Scalability Impacts

There could be a small window where message build-up occurs in oslo bus/ internal queues when the notification listeners are restarted.

Other deployer impact

Depending on the approach chosen, the deployer will have to synchronize pipeline file updates across multiple agent instances and then signal the agents to reload the configuration.

Developer impact

Implementation

Assignee(s)

Primary assignee:: rjaiswal
Other contributors:: TBD
Ongoing maintainer:: rjaiswal

Work Items

Implement timer to poll pipeline configuration file
Implement graceful termination of pollsters and listeners
Implement reconfiguring and reloading of pollsters in polling-agent
Implement reconfiguring and reloading of listeners in notification-agent
Unit Tests
Doc updates

Future lifecycle

We’ll want to continue to iterate on this for future cycles to gain additional functionality:

Use Tooz with publish-subscribe functionality to enable synchronization of pipeline configuration across multiple agent instances in a single coordination group.

Migration of event traits configuration

Dependencies

Relates to: https://blueprints.launchpad.net/ceilometer/+spec/dedicated-event-db https://review.openstack.org/#/c/119077/ (central and compute agents merge)

Testing

Add unit tests to exercise reloading of pipeline in agents

Documentation Impact

Update the relevant documentation about this new feature of reloading pipeline configuration

References

https://etherpad.openstack.org/p/liberty-ceilometer-pipeline-config

https://etherpad.openstack.org/p/configuration_via_data_store

https://review.openstack.org/#/c/171826/

https://wiki.openstack.org/wiki/Ceilometer/blueprints/Configuration-via-data-store

http://docs.openstack.org/developer/oslo.config/configopts.html#oslo_config.cfg.ConfigOpts.reload_config_files

Remove Eventlet from the API Server

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/remove-web-eventlet

As of Kilo, the WSGI application which provides the Ceilometer API can be run in two fundamentally different ways:

As a Python command that runs a Werkzeug-based web server that is monkeypatched to use eventlet.

As a WSGI application hosted by any WSGI server, often Apache + mod wsgi.

Hosting the server application in a dedicate WSGI host has performance, configuration and scaling advantages. Running the command line service, while convenient for simple testing, can result in difficult to diagnose bugs and unusual behaviors when it is monkeypatched to use eventlet. Since the Werkzeug server can (and as written, does) provide support for multi-process and multi-threaded interactions, the inclusion of eventlet should be removed.

Problem description

Eventlet monkeypatches the socket module to provide non-blocking network I/O. This can work well most of the time, but when unusual socket situations arise (for example the client frequently closes the connection before reading all the data the server would like to send) the produced tracebacks can be more difficult to use as debugging aids than those that are produced without eventlet in place. We want our tooling to be as useful as possible especially when there are situations that require debugging.

When using the Werkzeug service, using eventlet is fairly redundant as the web server is configured to run multiple processes.

Proposed change

We can resolve this problem relatively easily. At the moment eventlet monkey patching is turned on for everything that is imported under ceilometer.cmd (through ceilometer/cmd/__init__.py). This is overkill. We should evaluate which services benefit from it and turn it on only for those. We can start with the API server.

Further, for the sake of guiding people to the best solutions, we should update documentation and guidance to downstreams to indicate that using a WSGI host is the better way to host the API service in production.

Alternatives

Do nothing. This is a hygienic change that will benefit users and developers but is not strictly required.

Data model impact

None

REST API impact

None. If there is anything we’ve messed up.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

For production environments, the guidance to use a WSGI host will provide more options for adjusting configuration for scaling and performance.

Other deployer impact

Deployers who wish to take advantage of the more direct guidance to use a WSGI host will have to do what it says.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: chdent
Other contributors:: None
Ongoing maintainer:: chdent

Work Items

Separate eventlet and non-eventlet commands into two different module directories, starting with the api module.
Compare and contrast performance of the API server with and without eventlet paying specific attention to the impact on accessing the storage layer. Mike Bayer has pointed out that removing eventlet has little impact on a properly configured web server with a properly configured storage layer. We need to confirm this across the three styles of presenting the APU: WSGI host, Werkzeug with eventlet, Werkzeug without eventlet.
Update developer and admin documentation.

Future lifecycle

This is a core service which will receive ongoing maintenance from the Ceilometer project. Once the API server is not using eventlet we can see if any of the other console scripts would benefit from the change.

Dependencies

No new dependencies.

Testing

Existing API tests will cover these changes.

Documentation Impact

As mentioned above developer and admin documentation will need to be updated to reflect the new guidance about the best service configuration.

References

None

Spliting Ceilometer alarming

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/split-ceilometer-alarming

Ceilometer evolved from a simple meter gathering component to a lot of different component doing different things. The storage layer has been abstracted during Juno and Kilo and is going to be handled by Gnocchi. This spec proposes that the work continues so that the alarming subsystem of Ceilometer is split out as an autonomous component connected to the rest of Ceilometer and Gnocchi.

This should help with code review, maintenance, deployment, upgrade… as having several small service-oriented components to manage is going to be easier than a single monolithic project.

Problem description

Currently, Ceilometer features can be grouped in different components, mainly:

Metric polling and events storage (Ceilometer polling and notifications agents and collector)
Metric storage (Gnocchi)
Alarm evaluation and triggering

These all 3 components are currently merged in one code base and repository, whereas they are different service talking to each other. This leads to a bit of spaghetti code and architecture when digging the project.

Proposed change

We want to improve project management, code review, upgrade, etc, by spliting the alarming subsystem out of Ceilometer main repository and having it living its own life in its own repository.

The new repository and project will be called either aodh.

Alternatives

Do nothing and continue growing the project until it collapses.

Data model impact

The main data model impact was storing alarms in a different database than the samples and events. This change already been implemented in Juno.

REST API impact

The alarming API should not change and continues to be contained in /v2/alarms. However, the ceilometer-api endpoint and daemons is going to be replace by a specific REST API daemon only implementing the alarming subsystem. This means a new Keystone endpoint will need to be created and registered.

We’ll also provide a documentation about how to support configuring a proxy that can route requests to the Ceilometer API endpoint or the new alarm API endpoint.

Security impact

None

Pipeline impact

None

Other end user impact

As stated above, this will require a new Keystone endpoint to manipulate alarms.

Performance/Scalability Impacts

It’s easy to envision a better scalability if anything. It’ll be easier to scale the API endpoint for alarming only. Also, performances might be a little improved as the alarming codebase could be smaller.

Other deployer impact

The configuration file used by the alarming subsystem when it’ll be part of its own project/repository should be split out of ceilometer.conf. That means a new configuration file, but close to the current ceilometer.conf file, should be created.

Other than that, no option should be added or removed.

The daemon will remain the same and the API will be deployed in the same way – either with a (new) daemon either through the recommended WSGI interface.

The subsystem will be released independently than Ceilometer on its own schedule, like it’s already the case for Gnocchi and now other OpenStack projects. A coordinated release at the end of each cycle will happen too.

Developer impact

The new code repository hosting Ceilometer alarming will require a new core review team which will be composed of a copy of the current ceilometer-core reviewers.

Implementation

Assignee(s)

Primary assignee:: jdanjou

Work Items

Isolate Ceilometer alarming codebase from the other part of Ceilometer.
Create a new Python package and repository with that isolated codebase.
Publish this repository as openstack/<projectname>
Enable Tempest, devstack, grenade, etc, tests
Deprecate alarming code from Ceilometer
After Liberty: remove alarming code from Ceilometer
Contribute to the a downstream switch-over effort to migrate existing Fedora/Deb packages and puppet modules.

Future lifecycle

This Ceilometer subproject will be able to be released on its own like the rest of the OpenStack projects/components.

Work items 4 can be removed if the project agrees to not having a deprecation period. Otherwise, core reviewer will have to be careful that the no code changes/fixes are merged into Ceilometer rather than the new repository.

Dependencies

None

Testing

The testing coverage should remain the same, using unit tests and functional tests with Tempest.

Documentation Impact

The documentation will have to be updated to include this new component in lieu of the current Ceilometer alarm.

References

Liberty Summit Etherpad https://etherpad.openstack.org/p/ceilo-multi-identity
Name discussion http://eavesdrop.openstack.org/meetings/ceilometer/2015/ceilometer.2015-06-04-15.00.log.html

Support composite threshold rule alarm

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/composite-threshold-rule-alarm

This proposal want to support creating alarm that can specify composite threshold rule and try to replace combination alarm.

Problem description

The combination alarm was designed to handle alarm triggered against multiple conditions. A combination alarm need to depend on other alarms, the dependent alarms can be threshold alarms or combination alarms. This will form a dependency chain if the situation is complex. There are some potential problems of combination alarm evaluation, which have been described by ZhiQiang Fan in blueprint[1]:

dependency chain is too long, which causes the state of combination alarm will not be evaluated timely, in the worst case, it will cause the combination alarms never be updated.
dead loop will be introduced when update alarm, which causes the state of combination alarm will not be set properly.
dependency chain will be broken when delete dependent alarm.

For more information, please see: blueprint[1] and spec[2]

ZhiQiang Fan have made some efforts for these problems by resolve-alarm-dependency-chain proposal[2]. That proposal tried to check and resolve the dependency chain in both API and alarm-evaluator layers. To address the above problems, the implementation of that proposal will be complex, and, consider the alarm-evaluator working with partition coordinator, the situation is worse, and hard to be addressed by that proposal.

Proposed change

This proposal allow specifying multiple threshold rules when creating a threshold alarm, and the threshold rules combined with logical operators: and, or. following is a sample for the new alarm body:

{
    ......
    "threshold_rules": {
        "or": [{
            "type": "threshold" (or "gnocchi_threshold")
            "meter_name": "cpu_util",
            "evaluation_periods": 1,
            "period": 60,
            "statistic": "avg",
            "threshold": 0.8,
            "query": [{
                "field": "metadata.metering.stack_id",
                "value": "36b20eb3-d749-4964-a7d2-a71147cd8147",
                "op": "ge"
            }],
            "comparison_operator": "ge",
            "exclude_outliers": false
        },
        {
            "and": [{
                "type": "threshold" (or "gnocchi_threshold")
                "meter_name": "mem_util",
                "evaluation_periods": 1,
                "period": 60,
                "statistic": "avg",
                "threshold": 0.8,
                "query": [{
                    "field": "metadata.metering.stack_id",
                    "value": "36b20eb3-d749-4964-a7d2-a71147cd8147",
                    "op": "ge"
                }],
                "comparison_operator": "ge",
                "exclude_outliers": false
            },
            {
                "type": "threshold" (or "gnocchi_threshold")
                "meter_name": "disk.usage",
                "evaluation_periods": 1,
                "period": 60,
                "statistic": "avg",
                "threshold": 0.8,
                "query": [{
                    "field": "metadata.metering.stack_id",
                    "value": "36b20eb3-d749-4964-a7d2-a71147cd8147",
                    "op": "ge"
                }],
                "comparison_operator": "ge",
                "exclude_outliers": false
            }]
        }]
    }
}

The change items of this proposal:

Allow specifying a list of threshold rules and the threshold rules combined with two logical operators: “and”, “or”, see the above example. The combination of threshold rules can be multi-hierarchy to support complex scenario.
Alarm evaluator will resolve the alarm threshold rules combined with operators, and evaluate the threshold condition with statistics list queried from Ceilometer in proper order. In this process, the short-circuit logic should be considered to avoid unnecessary statistics query and comparison.
This proposal cannot be applied to EventAlarm, it mainly because event alarm evaluation is not periodical, the event alarm evaluator listening message bus and the evaluation action is passively triggered by external notifications. This is disadvantage if there is alarm usage that need to combine event alarm and threshold alarm, but I’d rather this situation should be handled in users end. That means if users want alarm fired when threshold reached and specify event occurred meanwhile, they can combine threshold alarm and event alarm in upstream application.
Also considered above disadvantage, the old combination alarm functionality will also be kept as supplementary for composite rules alarm, but need to add documentation to describe the potential problem.

These change will solve the above issues easily and hope to replace combination alarms. And also, this change will reduce the amount of alarms definition in our cloud, because currently, we may need to create the dependent alarms firstly if we want combination alarms. For end users, it is also convenient to show the alarm triggering multiple conditions than combination alarms (which need to query dependent alarms successively).

Alternatives

Adopt the proposal[2] and struggle to make it works with partition coordinator.

Data model impact

The “threshold_rules” will be a dict that can be include multiple threshold rules and the key of the dict will be “and” or “or”. But this has no effect on data model, because the rules is stored as json in db.

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: liusheng
Other contributors:: you
Ongoing maintainer:: liusheng

Work Items

Support Aodh API to allow creating alarms with this new threshold rules definition, and hopefully, the API can keep compatibility with former usage.
Implement the related changes in storage layer
Change the threshold evaluator to support multiple threshold rules and logical operators attached with an alarm.

Future lifecycle

None

Dependencies

None

Testing

The exist declarative tests and new tests with gabbi should be changed/added to test the alarm CRUD actions.
Unit tests should be added for new threshold evaluator process.

Documentation Impact

API documentation should be updated for adding description about this feature.
Users guide should be updated for add the this usage description.

References

[1] https://blueprints.launchpad.net/ceilometer/+spec/resolve-combination-alarm-dependency-chain

[2] https://review.openstack.org/#/c/98047/

Only support sqlalchemy in Aodh

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/only-support-sqlalchemy-in-aodh

This change want to deprecate others storage backends (mongodb, hbase) except sql and remove them in future O* cycle, because sql is totally competent for alarm data and we don’t need to maintain multiple storage backend.

Problem description

We support multiple storage drivers(habase, mongodb, sqlalchemy) in Ceilometer, that is mainly for the recording/query of performance based massive metric data. Aodh continue those drivers after alarming service splitting, but Aodh only has alarm related data which unlikely be a large number and sql can well meet the requirement.

Most of other projects in OpenStack only use sql as database. So we don’t need to keep the hbase and mongodb support in Aodh. The advantages of removing the hbase and mongodb support are:

make the code of Aodh more clean that reduce the maintenance works of storage drivers and related tests.
focus on the functionality and avoid considering multiple drivers support if new features coming.

Proposed change

Deprecate mongodb/Hbase support firstly and remove them in future cycle.

In current cycle:

deprecate mongodb and Hbase storage support and log warning messages if config mongodb or Hbase as storage driver
add alarm data migration tool for migrating data from mongodb/Hbase to sql

In future O* cycle:

remove the mongodb storage implementation
remove the hbase storage implementation
remove the mongodb and hbase related tests
remove the gate jobs based on mongodb and hbase

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

mongodb or Hbase as storage driver will be not recommended in deployment.

Developer impact

Developers will be happy, they won’t need to consider mongodb and Hbase storage support and related tests.

Implementation

Assignee(s)

Primary assignee:: liusheng
Ongoing maintainer:: None

Work Items

In current cycle:

send an email to do a user survey to get feedbacks about this change
deprecate mongodb and Hbase storage support and log warning messages if config mongodb or Hbase as storage driver
add alarm data migration tool for migrating data from mongodb/Hbase to sql

In future O* cycle:

remove the mongodb storage implementation
remove the hbase storage implementation
remove the mongodb and hbase related tests
remove the gate jobs based on mongodb and hbase
update the related docs

Future lifecycle

None

Dependencies

None

Testing

None

Documentation Impact

Update the documentation about these changes.

References

https://blueprints.launchpad.net/ceilometer/+spec/only-support-sqlalchemy-in-aodh

Rolling Upgrades

Wed, 10 Feb 2016 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/rolling-upgrades

As Ceilometer matures through each iteration and adds new features, users will be required to upgrade their existing environments while minimising the potential downtime required.

Problem description

Ceilometer currently provides 4 discrete services: polling agents, notification agents, collector service, and an api service. Each of them provide their own functionality and the flow of data and purpose of each component can often be lost on operators when upgrading services.

To ensure a smooth upgrade experience, we need to properly describe the upgrade path of the components.

Proposed change

Fortunately, due to the unilateral design of Ceilometer where work is done and handed off without worry, upgrading is actually a simple procedure and only requires proper ordering of upgrades.

Using the simple mantra of ‘never remove, never alter, only add’, we can ensure a new schema change is understandable by both old and new consumers.

There are two upgrade paths to handle – both require no code change:

Full upgrade of services

The database is upgraded using the above mantra.

The collector must be first taken offline. The new collector, that knows how to interpret the new payload, can then be started. It will disregard any historical attributes.

The notification agent can then be taken offline and upgraded with the same conditions.

The polling agents can be taken offline and upgraded. In this path, you’ll want to take down agents on all hosts before starting. After starting first agent, you should verify that data is again being polled.

The api service can be taken offline and upgraded at any point.

Partial upgrade of services

The database is upgraded using the above mantra.

The new collector services can be started alongside the old collectors and must know how to interpret the new payload. It will disregard any historical attributes.

The new notification agent can be started alongside the old agent if no workload_partioning is enabled OR if it has the same pipeline configuration. If not, the old agents must be loaded with the same pipeline configuration first to ensure the notification agents all work against same pipeline sets.

The new polling agent can be started alongside the old agent only if no new pollsters were added. If not, new polling agents must start only in its own partitioning group and poll only the new pollsters. After all old agents are upgraded, the polling agents can be changed to poll both new pollsters AND the old ones.

API service management is handled by WSGI so there is only ever one version of API service running

Note

Upgrade ordering does not matter in partial upgrade path. The only requirement is that the database be upgraded first. It is advisable to upgrade following the same ordering as currently described: database, collector, notification agent, polling agent, api.

Regarding new models, they will be pushed into their own unique queue similar to how Events and Samples are processed on completely separate queues today.

The above procedures will be added to OpenStack documentation.

Alternatives

oslo.versionedobjects - this seems like overkill and will add processing overhead to each and every sample
versioned queues - this does not have o.vo overhead but will require more memory to handle additional queues.
versioned payloads - this may simplify payload as we don’t need to carry historical fields but requires agents to understand each unique version

Data model impact

Depending on the amount of changes, this may increase the size of the model as we need to capture old attributes. We can choose to define a drop period if this becomes an issue where we stop support attributes from EOL builds.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

Same amount of processing but potentially larger payloads.

Other deployer impact

For those not consuming data from API, they will need to be aware of model changes if they want the latest and greatest.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: gordc
Ongoing maintainer:: everyone

Work Items

Add the above conditions to docs
Add testing support

Future lifecycle

If service requirements change, the above assumptions may not be enough.

Dependencies

None.

Testing

multi-node grenade testing
migration testing - https://review.openstack.org/#/c/234686/

Documentation Impact

This proposal is nothing but documentation.

References

[1] https://etherpad.openstack.org/p/mitaka-telemetry-upgrades

Event Alarm Timeout

Mon, 25 Jan 2016 00:00:00

https://blueprints.launchpad.net/aodh/+spec/event-alarm-timeout

This BP adds timeout mechanism for event-alarm. End users can specify a timeout, 0 (no timeout) by default, for each event-alarm. The alarm status becomes ‘TIMEOUT’ after timeout reached without receiving desired event.

Problem description

After event-alarm were introduced in Liberty, end users or operators could set alarm for desired event and get alarmed when it receive them. But in some circumstances, operator want to know otherwise: when desired event is not received.

For example, “compute.instance.create.end” is the final event sent to message bus to indicate success of instance creation. Not receiving it after a long time is a signal of creation failure, in which operator should be notified. Unfortunately, current event-alarm doesn’t support it.

Proposed change

When creating event-alarm, a new parameter ‘timeout’ is proposed to define a expiry time length, so that alarm gets fired when desired event is not received in expected time. Otherwise, alarm status becomes ‘TIMEOUT’.

Currently, 3 states are supported in alarm: ‘UNKNOWN’, ‘ALARM’ and ‘OK’, so a new state ‘TIMEOUT’ will be added to reflect timeout situation.

For timeout implementation, an ‘alarm.timeout.start’ notification is sent out by the AODH api process after the event-alarm is created. After receiving it, evaluator asks its timeout thread/process to handle timeout request. In this way, avoid new process in AODH api and make all alarm handling jobs inside evaluator.

Synchronization handling is critical in evaluator, as both evaluators original process and timeout process can change status for same alarm. To avoid complicated lock, timeout process just send a ‘alarm.timeout.end’ event with related alarm/project id to ‘alarm.all’ topic, where evaluator original process handle it along with desired event.

Each ‘alarm.timeout.*’ event should carry enough info in payload, including alarm_id, project_id, timeout, and desired event. These info could be used for evaluation and future partitioning infrastructure.

Each evaluator has only one timeout thread, which gets timeout requests from evaluator into a queue. Small footprint of timeout process is guaranteed, as it always sleeps unless one timeout happens. After wake up, it only does 2 things:

sends out ‘alarm.timeout.end’ event
pick up nearest timeout request and start sleeping for it

The final alarm status depends on the order of events. If ‘timeout.end’ event comes first, alarm status becomes ‘TIMEOUT’ and following desired event is ignored. Otherwise, alarm status becomes ‘ALARM’ and following ‘timeout.end’ event is ignored. In this way, synchronization is well handled with new ‘timeout.end’ event and simple logic in evaluator.

After adding timeout, event-alarm state transition changes as following:

EVENT - desired event arrive
TIMEOUT - timeout happen

Registered action is only triggered when:

transition between different states, like UNKNOWN => ALARM
receiving desired event again at ALARM state if repeat_actions is true

+-----------+            +---------+  EVENT
|           |            |         +---------+
|  UNKNOWN  +------------> TIMEOUT |         |
|           |  TIMEOUT   |         <---------+
+-----+-----+            +---------+
      |
      |
      |                  +---------+
      |                  |         +---------+
      +------------------>  ALARM  |         |
              EVENT      |         <---------+
                         +-+-----^-+  TIMEOUT
                           |     |
                           +-----+
                            EVENT

Also need changes in DB layer to get all the event-alarm with timeout. In following cases, we need restart timeout thread if the alarm has timeout, and previous timeout thread already exit:

reset the alarm state to ‘UNKNOWN’
enable the alarm via the ‘enabled’ attribute

Alternatives

There is another BP from Igor to illustrate a different high level design and usage model. Pls. check https://blueprints.launchpad.net/ceilometer/+spec/timeout-event-alarm-evaluator

It adds a new alarm type ‘event_timeout’ to track sequence of events, like: “compute.instance.create.start”, “compute.instance.create.end”. So this new alarm includes 2 events: start and end – only after receiving “start” event, timeout is created for “end” event. It is not as simple and flexible as this BP, which just adds timeout.

To handle the synchronization, an alternative is to lock critical section between evaluator timeout and original thread, so each of them need handle alarm data structure. But it’s tricky and bug prone.

Data model impact

None

REST API impact

API needs minor changes to add ‘timeout’ to ‘event_rule’. One simple event-alarm definition is as following:

{"alarm_actions": ["log://"],
 "ok_actions": ["log://"],
 "alarm_id": null,
 "enabled": true,
 "name": "alarm01",
 "repeat_actions": false,
 "state": "insufficient data",
 "event_rule": {"query": [{"field": "traits.name",
                           "type": "string",
                           "value": "cirros-0.3.4-x86_64-uec-ramdisk",
                           "op": "eq"}],
                "event_type": "image.update",
                "timeout": 10},
 "type": "event"}

Security impact

None

Pipeline impact

None

Other end user impact

End user need to know a new ‘timeout’ parameter when create event alarm.

Performance/Scalability Impacts

No obvious performance issue, because of small footprint of timeout thread. No obvious scalability issue, as timeout handling is done in evaluator who will support good partition.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: edwin-zhai

Work Items

Add new parameter ‘timeout’ for event-alarm creation in aodh-client
Add new alarm state ‘TIMEOUT’ for timeout expired alarms
Add new interface in aodh-client and DB layer to get all event-alarm with timeout
Modify AODH api’s event-alarm creating code to send out ‘alarm.timeout.start’ notification
Modify AODH event-alarm evaluator so that:
- spawn a new timeout thread to handle all timeout requests
- timeout thread works in a loop of sleeping for timeout seconds then sending out ‘alarm.timeout.end’ event
- set related alarm status as ‘TIMEOUT’ when receive ‘alarm.timeout.end’ event.
Add extra action to restart timeout thread when reset alarm state to ‘UNKNOWN’ or enable the alarm if previous timeout thread already exit

Future lifecycle

To be maintained by edwin-zhai for bug fixing and enhancement.

In future, we need timeout thread disaster-recovery capability, that is, no loss of timeout info when evaluator crash. Need store pending timeout requests in DB, and feed evaluator when restarting.

Dependencies

None

Testing

Add new test case besides current event-alarm test to cover timeout

Documentation Impact

Administrator Guide and Installation Guide in OpenStack Manuals should be updated to describe usage of ‘timeout’ parameter.

References

Blueprint Timeout mechanism for Event Alarm Evaluator https://blueprints.launchpad.net/ceilometer/+spec/timeout-event-alarm-evaluator

Enable LBaaS V2 for Ceilometer

Wed, 11 Nov 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/lbaas-v2-enablement

The goal of this blueprint is to support LBaaS v2 to poll the LoadBalancer data in ceilometer.

Problem description

According to the Neutron api documentations[4], the LBaaS v1 has been deprecated in the release Mitaka.

Based on the neutron LBaaS v2 documentation[1] and LBaaS v2 blueprints, LBaaS v2 apis are supported. LBaaS v1 and LBaaS v2 can not be run in the same time. The users can only choose one version of LBaaS to run in their environment.

According to the ceilometer admin guide[3], Ceilometer support to collect the data of LBaaS from Juno release.

When the users choose to use LBaaS v2 in their environment, they will meet the problem to use the ceilometer to collect LBaaS meters for the LBaaS v2 api is different from LBaaS v1.

For LBaaS v1 has been deprecated from the release Mitaka, Ceilometer will lost the function to collect the LBaaS meters if Ceilometer doesn’t support the LBaaS v2.

The design of LBaaS v1 and LBaaS v2 is different. LBaaS v2 introduces the meter ‘loadbalancer’ and it contains ‘vip’, so the meter ‘vip’ is not important to the LBaaS v2 users. And LBaaS v2 has not supplied with the APIs to achieve the data of the vips directly.

Meters are supported by LBaaS v1 and not supported by LBaaS v2.

Name Type Unit network.services.lb.vip Gauge vip network.services.lb.vip.create Delta vip network.services.lb.vip.update Delta vip

All the LBaaS APIs have changed in the version 2. We need to use the new APIs to collect the data.

The difference of APIs between LBaaS v1 and LBaaS v2.

Meter_Name v1 API network.services.lb.pool /v2.0/lb/pools network.services.lb.member /v2.0/lb/members network.services.lb.health_monitor /v2.0/lb/health_monitors network.services.lb.total.connections /v2.0/lb/pools/%s/stats network.services.lb.active.connections /v2.0/lb/pools/%s/stats network.services.lb.incoming.bytes /v2.0/lb/pools/%s/stats network.services.lb.outgoing.bytes /v2.0/lb/pools/%s/stats

Meter_Name v2 API network.services.lb.pool /v2.0/lbaas/pools network.services.lb.member /v2.0/lbaas/pool/%s/members network.services.lb.health_monitor /v2.0/lbaas/pools network.services.lb.total.connections /v2.0/lbaas/loadbalancers/%s/stats network.services.lb.active.connections /v2.0/lbaas/loadbalancers/%s/stats network.services.lb.incoming.bytes /v2.0/lbaas/loadbalancers/%s/stats network.services.lb.outgoing.bytes /v2.0/lbaas/loadbalancers/%s/stats

The difference of neutron client methods we use to collect the LBaaS between LBaaS v1 and LBaaS v2.

Meter_Name v1 method network.services.lb.pool list_pools network.services.lb.member list_members network.services.lb.health_monitor list_health_monitors network.services.lb.total.connections retrieve_pool_stats network.services.lb.active.connections retrieve_pool_stats network.services.lb.incoming.bytes retrieve_pool_stats network.services.lb.outgoing.bytes retrieve_pool_stats

Meter_Name v2 method network.services.lb.pool list_lbaas_pools network.services.lb.member list_lbaas_members network.services.lb.health_monitor list_lbaas_healthmonitors network.services.lb.total.connections retrieve_loadbalancer_stats network.services.lb.active.connections retrieve_loadbalancer_stats network.services.lb.incoming.bytes retrieve_loadbalancer_stats network.services.lb.outgoing.bytes retrieve_loadbalancer_stats

Proposed change

In order that the user can switch the LBaaS api version in current situation, A new entry will be added in the configuration file to specify the api version of LBaaS.

When the user chooses to use the LBaaS api v1, the meters supported by LBaaS v1 will be collected. But LBaaS api v1 has been deprecated in the release Mitaka, we should suggest the user should not choose to use LBaaS api v1.

When the user chooses to use the LBaaS api v2, the meters supported by LBaaS v2 will be collected. But some meters will not be supported by LBaaS v2, including ‘network.services.lb.vip’, ‘network.services.lb.vip.create’, ‘network.services.lb.vip.update’. we will add some error messages in the log files to identify these meters can not be supported by LbaaS v2. And The meters are supported both by LBaaS v1 and LBaaS v2, we will switch to use the new neutron client methods to achieve the data of them. In the neutorn client[5], the new LBaaS v2 apis have characters ‘lbaas’.

These change will not change the data model. It just switches to use the LBaaS v2 api in the client.

New meters are introduced and supported by LBaaS v2.

Metric Definitions:

Name Type Unit Origin network.services.lb.loadbalancer g loadbalancer p network.services.lb.listener g listener p network.services.lb.loadbalancer.create d loadbalancer n network.services.lb.loadbalancer.update d loadbalancer n network.services.lb.listener.create d listener n network.services.lb.listener.update d listener n

Note: loadbalancer: existence of a loadbalancer listener : existence of a listener g : gauge d : delta p : pollster n : notification

Metadata:

We extract the data from the response of calling neutron client to constitute the metadata of sample. The fields of metadata we use are fixed in the code. After checking the response of LBaaS v2 apis, most of the fields are still in the response. Only the field ‘status_description’ is not in the response. But the relationship between field ‘status’ and ‘status_description’ are list in the documentation[5], we can use the valute of field ‘status’ to set the value of the field ‘status_description’.

Basd on the above reason, the structure of metadata has not changed. We will not meet the problem when both the data collected by LBaaS v1 and LBaaS v2 are in the same database.

Alternatives

Option one: Add the limitation in the admin guide[3], let the users to know the ceilometer can not support the LBaaS v2. But the LBaaS v2 is more powerful and more safe than v1, more and more users will choose to use LBaaS v2 in the future. And the LBaaS v1 is deprecated in Mitaka release. So this limitation is not satisfied.

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

jizilian

Primary assignee: jizilian

Other contributors:

Ongoing maintainer:

Work Items

a. Add the new entry in the configration file to specify the version of LBaaS b. Use the LBaaS v2 api in ceilometer neutron client when the user decide to use the LBaaS v2 c. Add the new UT test cases

Future lifecycle

Ongoing maintenance will be handled by the Ceilometer team, myself included.

Dependencies

None

Testing

Unit and integration Tests will be added to cover the necessary metrics and validate samples generated.

Documentation Impact

Need to update the doc about LBaaS v2 support.

References

[1]https://specs.openstack.org/openstack/neutron-specs/specs/kilo/lbaas-api-and-objmodel-improvement.html [2]https://blueprints.launchpad.net/neutron/+spec/lbaas-api-and-objmodel-improvement [3]http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html [4]http://developer.openstack.org/api-ref-networking-v2-ext.html [5]https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py

Tempest Plugin

Thu, 01 Oct 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/templest-plugin

As with devstack and grenade, the tempest program is encouraging projects to manage their own tempest tests via code hosted in each project’s code repository. This spec proposes that we do this for those projects living under the ceilometer umbrella.

Problem description

Managing tempest tests within tempest itself in the world of the big tent is problematic: The people who want the tests to exist and the people who review tempest are not congruent. This can lead to delays in getting relevant tests in place. When projects host their own tests as plugins the projects decide what happens when, according to their own requirements.

For ceilometer which is now spread over a few repos and has increasingly complex integration scenarios these problems are multiplied.

Proposed change

Using the tempest plugin documentation and tempest-lib new tempest tests will be created in ceilometer related repos that duplicate existing telemetry-related test functionality from tempest. Initially these will be done in the ceilometer repo itself. There is a sahara example and a manila example that can be used for guidance.

When these have proven themselves the test in tempest will be removed and the project should be in a position to greatly increase the number of tempest tests that are available and run.

Alternatives

We can leave things as they are now but this is contrary to the goals of the tempest project.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

None.

Other deployer impact

If there is improved ceilometer coverage in tempest then operators who use tempest to validate their clouds will have increased confidence in more services in their cloud.

Developer impact

It will be easier for developers to find and improve tempest tests related to ceilometer.

Implementation

Assignee(s)

Primary assignee:: chdent
Other contributors:: you
Ongoing maintainer:: The ceilometer team

Work Items

Evaluate pre-existing work from other projects.
Create a version for ceilometer.
Create experimental gate jobs to test them against commits.
Remove existing tempest tests.
Update gate jobs.

Future lifecycle

The ceilometer team will need to be responsible for the ongoing care and feeding of the in-tree tempest tests.

Dependencies

None.

Testing

See work items above.

Documentation Impact

Tempest documentation will need to be updated to reflect that use of ceilometer will require referencing the plugin.

References

Adds resident memory size meter to libvirt inspector

Thu, 17 Sep 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/memory-resident

Memory statistics include just the actual usage inside the Virtual Machine. The ‘memory.resident’ metric will add the resident memory size. The implementation is done in the libvirt inspector. The value will be fetched using the libvirt API and the ‘rss’ for dommemstats.

Problem description

Resident memory is important to see the amount of memory consumed by the Virtual Machine from the Physical Host. This spec adds the resident memory statistics to the libvirt inspector.

Proposed change

Implements the method ‘inspect_memory_resident’ of LibvirtInspector, fetches the rss value for memory from libvirt API ‘virDomainMemoryStats’. The libvirt API ‘virDomainMemoryStats’.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: vivek-nandavanam

Work Items

Implements the method ‘inspect_memory_resident’ of LibvirtInspector.
Updates ceilometer measurements document.

Future lifecycle

Once this feature enabled, need test and bug fixing in next 2 releases to avoid regression.

Dependencies

libvirt 0.7.5

Testing

Unit tests should be sufficient.

Documentation Impact

The added metrics will need to be documented in https://github.com/openstack/openstack-manuals/blob/master/doc/admin-guide-cloud/source/telemetry-measurements.rst which will get reflected in the cloud admin documentation http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html#openstack-compute

References

Improve Nova instance metering

Thu, 06 Aug 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/improve-nova-instance-metering

Now we have a great load on Nova API because ceilometer-polling with “compute” namespace will polling on Nova API periodically. This proposal aims to reduce the load by abandoning the current way of metrics collection with periodically polling.

NOTE: The ceilometer-polling service with “compute” namespace is previously named as ceilometer-agent-compute.

Problem description

Consider the following example. There is an environment with 500 compute nodes and every compute node deployed with ceilometer-polling with “compute” namespace (this is a real deployment scale of our company). the polling interval is set to 60s. For every polling period, ceilometer-polling will invoke “instance_get_all_by_host()” method (and maybe need querying image and flavor also) to query the instances of this host from Nova API. That means the Nova API will receive 500 requests meanwhile every minutes. It would massively increase the workload on Nova API.

In the metadata caching proposal[1], the instance info from Nova is be cached in ceilometer-polling by using ‘change-since’ query filter of Nova server-list API. That has significantly decreased the amount of items of every Nova query. But that cannot help to reduce the query frequency.

To illustrate this, We assume that we can deploy 100 instances on one host at most. with proposal[1], that may reduce 100 instances to 20 instances returned in every Nova list request, it’s not a significant improvement. But with this proposed change it may will reduce the times of Nova API request from 500 requests/min to 1 requests/min, it would be a significant improvement and that is the goal of this proposal.

Proposed change

This proposal try to solve the problem described above with changes:

The ceilometer-polling agent with “compute” namespace runs on every compute nodes. Currently, the agent need to get the instances list of this host by calling “instance_get_all_by_host” method of novaclient, then traverse this instance list to get metrics of instance(e.g. cpu, disk.read.bytes, network.incoming.bytes) by inspector, which will invoke the interfaces of virt layer to get this metric data. Then ceilometer-polling will assemble the metric data from virt layer and instance info from Nova API as sample objects.

Differently, this change want to add a global cache mechanism to cache the resource metadata(instance info from Nova db). ceilometer-polling will use “instance_get_all” to query all the instance from Nova, and then cache them to the global cache. Also, ceilometer-polling will record the resource caching time to the cache system, and add an option named “interval_to_update_resource” to indicate the max interval of resource cache updating. If the time from last updating is larger than this option value, the resource cache need to be updated again.

The “change-since” will also used as a query filter of “instance_get_all” to massively reduce the amount of every query and only update the changed instance to the resource cache.
In every ceilometer-polling of every compute node, unlike previous, agent will try to query instances list of this host from the global resource cache if the cached resources are not out of date, otherwise, query all the instances from Nova API to update the cached instance resources and get the instances of this host. The rest of the process will be same as before. The above “out of date” means the time from last cached resource updating is larger than the “interval_to_update_resource” option value.

It allows users to enable/disable this global resource cache mechanism, when disable this mechanism, the process is same as before, ceilometer-polling won’t use the global cache but use a local cache implemented by proposal[1].

Alternatives

Another choice of resource metadata updating solution

Another suggested solution for this goal is using instance events by notifications instead of instance metrics by polling to collect instance resource metadata.

Using events to update resource metadata in Ceilometer or Gnocchi will be more efficient because the resource metadata may only need to be update when something about resource happened. but I have some concerns about that approach:

The resource metadata updating will depend on events collection, what if some events didn’t be collected ?
We need to ensure all the resource update actions in other projects of OpenStack have corresponding events published, that need other resources providers to ensure.
Do all the events include enough info for ceilometer? we may need the resource metadata do some filter query. e.g. use query filtered by instance metadata do auto-scaling.

Another choice of where to use global cache

Instead of the global cache directly used by compute pollsters of ceilometer-polling service, another choice is using the global cache in notification-agent service. Instances info will be polled and cached to the global cache, metric data will be polled by ceilometer-polling and sent to notification-agent, in notification-agent, the instances info will be queried from the global cache, and then assemble the instance info and metric data to sample objects, then push them to pipeline.

With this proposal, the resource metadata won’t be taken from ceilometer-polling to notification-agent, only the lightweight metric data will be pushed to notification-agent. In a large scale deployment, this will also significantly reduce the load of message bus.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

Need to deploy an external global cache system, more details please see[2].

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: liusheng
Ongoing maintainer:: liusheng

Work Items

Implement a global resource caching mechanism to cache instance info.
Change the instance metric discover to use the cached resource.

Future lifecycle

None

Dependencies

None

Testing

The current tests will need to modify to adapt these change.
Unit tests should be add to cover these change.

Documentation Impact

Documentation should be updated to add description for this.

References

[1] https://review.openstack.org/#/c/185084/

[2] https://review.openstack.org/#/c/250778/

Events RBAC via Policy

Wed, 29 Jul 2015 00:00:00

OpenStack needs to support granular, customizable Role-Based Access Control (RBAC) for ceilometer events. Policy should be dependent on policy.json rather than simply hard-coded.

https://blueprints.launchpad.net/ceilometer/+spec/events-rbac

Problem description

The only RBAC validation for ceilometer’s /events REST API that is in place today is hard-coded logic to restrict requests to admins. This is not granular enough. In a multi-tenant environment, there must be provision to restrict data from events to the scope of the token given on the request (e.g. an admin from one project should not be able to view the events of another project). There should also be a way to allow non-admins access to some events (e.g. events for their resources). This issue was originally raised as a bug [1].

Proposed change

This blueprint proposes the following changes in the behavior of ceilometer’s /events REST API:

Add the following rules to ceilometer’s default policy.json:

“telemetry:events:index”: “role:admin” “telemetry:events:show”: “role:admin”

Modify the code to uses those checks to determine who is allowed to request a list of events (index) or a specific event (show). This pattern is in line with other OpenStack projects.

Further modify the code to, if/when those checks pass, filter which events will be returned based on the scope of the token. If the token is for an admin user (determined by checking the special context_is_admin rule from policy.json), only return events corresponding to the project to which the supplied X-Auth-Token is scoped. This is also consistent with other projects. If the token is for a non-admin user, further filter the results to only return events corresponding to the user to which the supplied X-Auth-Token is scoped, as well as the project.

The events REST API will be processed only if the user passes a project-scoped token. This is required to filter the events based on the project. If the user passes an unscoped or domain-scoped token, a 403 error will be thrown.

There may be events for which no project/user information is recorded. Most if not all of these should have project/user information, so event-generating code will need to be modified to include that when creating future events. Updating existing events within the database is outside the scope of this spec. Until all events have project/user information, events which lack this data will be returned along with events corresponding to the token’s project, if the token’s user is an admin on that project.

For the first implementation, common name-reserved traits will be used to store project/user information.

Alternatives

We could use policy.json checks to filter results as well as determine whether the request is allowed. This does not fit with the role and purpose of policy.json, and may also have significant performance implications.

Storing project/user information in base attributes rather than traits was considered. This may be better long-term, but there has been some debate on the appropriateness of using base attributes if there will be some events that are not scoped to a project/user. Implementing this using traits first will help us determine whether that is a valid case. A switch to base attributes could come later.

Rather than return both project/user-scoped events and unscoped events in a single response, we could require a separate API call for unscoped events if/when that is what the user desires. E.g. /broadcasts instead of /events. As it is unclear whether we will end up with any events that do not have project/user information, considering this now would be premature.

We could create another policy check to identify whether a non-admin role should be considered an admin for the purpose of the /events API (and thus able query events for the entire project rather than just for themselves). But since it is expected that events will be split out of ceilometer like meters (gnocchi) and alarms (aodh), that would seem to be a short-term solution.

We could support domain-scoped tokens to allow domain admins to query events for the entire domain with a single request. This may be needed in the future, but does not appear to be something that must be done as part of this effort. And there could be benefits to waiting for the keystone reseller spec [2] implementation.

Data model impact

None, as long as we stick using traits.

REST API impact

Whereas before this change an admin of any project saw events for EVERY project in the response to a GET /events request, now GET /events will filter out events belonging to projects other than the one in the scope of the request’s auth token. This closes a security hole that allowed admins of one project to gain information about another project in which they have no role.

Security impact

This will greatly enhance security by default. It will also provide a mechanism for operators to customize policy related to events, so operators taking advantage of this capability will need to consider the potential security impacts of their configuration changes.

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Filtering on project/user may have a slight negative performance impact, though this should be offset by the likely much more substantial performance improvement of returning less data to the caller.

Allowing events that do not have project/user information via the same API queries will mean making two internal calls (one to get events scoped to the project/user and another to get events not scoped to a project/user) and then merging them. This may have a slight negative performance impact.

Other deployer impact

Updating from a previous release will need to include moving to a new version of policy.json including the new checks, else those would resolve to the default which is to allow for anyone.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: edmondsw
Other contributors:: dikonoor

Work Items

Check context_is_admin to determine appropriate response filtering
Check policy.json to determine whether the request is allowed
Add project/user to events that are currently lacking those details

Future lifecycle

This is essentially a large bug fix, not a feature. As such, all members of the Telemetry program will be expected to keep this from breaking again.

Several potential future enhancements are discussed under the Alternatives section. It is expected that those would require a separate spec if someone wants to pick up any of them in the future.

Dependencies

None

Testing

Unit tests should be sufficient.

Documentation Impact

Developer documentation [3] will be updated to add user_id to the list of default traits, and to explain that non-admins will only be able to view events with their user_id, while admins will only be able to view events with their tenant_id plus events not associated with a project (aka tenant).

References

[1] https://bugs.launchpad.net/ceilometer/+bug/1461767 [2] https://github.com/openstack/keystone-specs/blob/master/specs/liberty/reseller.rst [3] http://docs.openstack.org/developer/ceilometer/events.html

Highly Distributed Coordinated Notifications

Tue, 07 Jul 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/disributed-coordinated-notifications

In Kilo, support for coordinated notifications agents was added. This enabled users to deploy multiple notification agents and ensured related messages within a pipeline were funnelled into the same agent to allow for proper aggregation calculations.

Problem description

In the initial implementation, all the data relating to a pipeline was funneled into a single queue per pipeline. While it ensured all corresponding data is sent to the same place, it removed the ability to scale horizontally as the pipeline queues can only have a single consumer listening to it. This means that while multiple agents/handlers could be used to pull data off main OpenStack queue, once the data reached pipeline processing, it was relegated to a single worker.

Data can be handled in parallel even at the pipeline level. For example, when there are no transformers, datapoints do not need to be handled sequentially. Additionally, when transformers are present, datapoints of different resources have no relevance to each other and can be handled in parallel.

Proposed change

To parallelise and scale out processing, we will create multiple copies of each pipeline. When a datapoint arrives, we will bucketise each datapoint by a hashed grouping key. Each transfromer will have a grouping key assigned to it to note any dependency requirements (ie. transformers that work on resource ids). When setting up pipeline, all the keys of transformers in the pipeline will be combined to ensure that related datapoints will be consistently sent to the same pipeline for processing.

The basic workflow is as follows:

* on notification agent startup, create a listener for main queue
* for each pipeline definition, we create x queues and x listeners where x
  corresponds to the number of notification agents registered to group
* when a datapoint is received, agent builds sample.
* after sample is built, we hash fields defined by transformer requirements
  and mod by number of agents.
* using mod value we push datapoint to corresponding pipeline queue.
* same processing steps here on out (listener grabs data -> pipeline -> pub)

This solution CANNOT handle multiple grouping_keys in pipeline. To properly handle multiple grouping_keys in a pipeline we need to requeue after each transform. The logic would become: main queue -> build sample -> pipe1.transform1 queue -> pipe1.transform2 queue -> etc -> publish.

Studying the existing transformers we have:

* Accumulator - this does not really have any grouping requirements, it just
                batches samples.
* Arithmetic - this is grouped by resource_id
* RateOfChange - this is grouped by name+resource_id. but it can more
                 be more generally grouped by just resource_id
* Aggregator - this is grouped by name+resource_id+<custom> but can also be
               more generally grouped by just resource_id

Based on the above, it seems like resource_id is always a valid general grouping key and the transformers may do more granular groupings themselves. Because of this, it seems safe to assume we don’t need to support multiple grouping keys (for now).

Alternatives

Dumb easy fix is to detect whether a pipeline has transformers. If not, it can be consumed by any number of consumers and thus we can assign multiple listeners to those queues. This doesn’t help distribute load of pipelines with transformers but allows for transformers spanning resources.
We implement a shared memory/storage mechanism so any worker can discover the historical context. This is hard. I feel like i would end up recreating Storm/Spark
Magic

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

Nothing from user point of view. Internally, we will have more pipeline queues.

Other end user impact

None. Unless we decide to add option to define number of copies of pipeline queues.

Performance/Scalability Impacts

Positive. It will distribute pipeline processing when running in coordinated mode. Message queues are consistently used with thousands of queues so creating copies of pipeline queues (a relatively small set) should not be an issue.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: chungg
Ongoing maintainer:: chungg

Work Items

add grouping key to each transformer to define grouping of datapoints * will be just resource_id
add support to build pipeline hashing from above grouping keys
add functionality to create pipelines queues per agent and distribution logic.

Future lifecycle

Support different grouping keys in a pipeline.

Dependencies

None

Testing

Test already exists. Just need to validate that we create appropriate amount of copies.

Documentation Impact

None. Maybe dev docs.

References

https://docs.google.com/presentation/d/1QgjDOLRnKDboqP8P1LvV0kR5aQEv_VJsDtMlh6u7tIY/edit?usp=sharing

Track DBaaS(Trove) Notifications

Wed, 17 Jun 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/track-dbaas-notifications

The goal of this blueprint is to capture the notification events emitted by Trove (DBaaS) during create, modify volume, resize, delete operations and periodic exist events.

Problem description

Trove is Database as a Service for OpenStack. It automates the creation, configuration and management of relational or non-relational database without the burden of handling complex administrative tasks.

The trove-taskmanager service dispatches tasks such as provisioning database instances, managing the lifecycle of database instances, and performing operations on the database instance and emits notification event for each such task.

These notifications are CRUD events, that notify the creation, update or modification of a trove instance and its underlying volume, and also have measurement values such as instance size and volume size.

Ceilometer needs to be updated to consume and record these notifications as samples and events (traits).

From Trove perspective they are running instances of a database, not compute instances. Metering and billing will be based on hours that the database has been running, not necessarily how long the hosting instance has been run. This requires a unique set of metering records(samples) to be generated and stored to enable usage tracking and billing of the individual database instances.

If Ceilometer processes and record these notifications, other services will be able to use this for aggregation and billing purpose.

Proposed change

To represent as events, define event definitions and field mappings (trait definitions) for Trove resource type - instance in the /etc/ceilometer/event_definitions.yaml file.

The trait field syntax in trait definitions follow a variant of the JSONPath.

Example of the event definition and field mappings for the domain notifications that need to go in event_definitions.yaml,

event_type: trove.instance.*
traits: &trove_traits
instance_size:
fields: payload.instance_size
resource_id:
fields: payload.instance_id
instance_name:
fields: payload.instance_name
nova_instance_id:
fields: payload.nova_instance_id
created_at:
fields: payload.created_at
launched_at:
fields: payload.launched_at
volume_size:
fields: payload.volume_size
volume_id:
fields: payload.nova.volume_id
modify_at:
fields: payload.modify_at
deleted_at:
fields: payload.deleted_at

To enable event processing in Ceilometer, configuration is needed in ceilometer.conf,

Configure store_events as True under the notification section,

eg - [notification] store_events = True

Specify the event_definitions file name under the event section,

eg - [event] definitions_cfg_file = event_definitions.yaml

Using events provides better querying of metadata and avoids the effort to aggregate samples with a fixed measurement value of 1.

To represent as samples, a notification plugin is required in Ceilometer, to hear notifications at an exchange and topic and transform them into samples.

This is needed to account for the database uptime hours, which is based on the trove.instance.exists event. Specifically, the audit_period_beginning and audit_period_ending for each exists event will be used to compute the uptime hours

A counter type - database can be used to associate the trove samples

Add a new listener class - TroveMetricsNotificationBase that extends plugin.NotificationBase for DBaaS service to transform Trove event data into samples

A sub-class of TroveMetricsNotificationBase that emits samples for the trove.instance.create and trove.instance.exists events and another for the trove.instance.modify* and trove.instance.delete events

The Trove event types have the following pattern,

trove.<resource>.create
trove.<resource>.modify_volume
trove.<resource>.modify_flavor
trove.<resource>.delete
trove.<resource>.exists

, where resource is instance.

Alternatives

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

No new impacts. Pre-existing concerns with capacity at the notification and storage handling layers remain.

Other deployer impact

Enable Trove exchange in ceilometer.conf

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: rjaiswal
Ongoing maintainer:: rjaiswal

Work Items

Event definitions and field mappings for all supported Trove resource types.
Event validation.
Trove Notifications Listener class to process create/update/delete/exists events for trove instance.
Test coverage for the above listener class and sample validation.

Future lifecycle

There could be new types of notifications when new resource types are added or new features are supported like backup/restore, patching, monitoring etc.

When Trove conforms to the PaaS event format, the field mappings for integrated events will need to be refactored to adjust to the new event format.

With the impending move towards declarative metering in Ceilometer, the proposed notification plugin for Trove will become redundant/deprecated and replaced with declarative meter definitions.

Dependencies

DBaaS is construed as a PaaS service by some and might have it own message bus where it is sending notifications. Ceilometer recently was extended to consume notification from multiple message bus - https://review.openstack.org/#/c/77612/

If there are multiple message buses, then ceilometer will need multiple transport endpoints configured in ceilometer configuration. (ceilometer.conf)

Testing

Unit and integration Tests will be added to validate events generated.

Documentation Impact

None.

References

http://docs.openstack.org/developer/ceilometer/events.html

http://docs.openstack.org/trunk/config-reference/content/ch_configuring-openstack-telemetry.html

http://specs.openstack.org/openstack/ceilometer-specs/specs/juno/paas-event-format-for-ceilometer.html

https://wiki.openstack.org/wiki/Trove/trove-notifications

https://github.com/kennknowles/python-jsonpath-rw

PowerVM Compute Inspector

Wed, 17 Jun 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/powervm-inspector

PowerVM is a hypervisor for the POWER processor architecture. Community drivers to support PowerVM in an OpenStack environment for Nova and Neutron (ML2 agent) are in active development. Telemetry is a key component for operators within an OpenStack environment. This blueprint proposes that an inspector for PowerVM be included in Ceilometer.

Problem description

OpenStack drivers delivering support for the PowerVM hypervisor are being developed for Nova and Neutron (an ML2 agent). Operators require more support than Nova and Neutron integration - they need telemetry information. PowerVM provides a variety of performance monitoring interfaces for both virtual machine and system monitoring metrics.

This proposal is for the PowerVM Driver Team to add a PowerVM compute inspector to the Ceilometer project. This will allow operators receive and store Telemetry data for PowerVM in the same fashion as other hypervisors.

Proposed change

We propose that a new python package be created: ceilometer/compute/virt/powervm.

This package will contain a compute inspector (similar to those for hyperv, libvirt, vmware, and xen) that gathers the statistics.

The collection of these statistics will be driven through the pypowervm API. This is an open source project that allows python programs to interact with the PowerVM hypervisor.

The module within pypowervm that will be used is the LPAR (logical partition, equivalent to a virtual machine) statistics package. This will provide the interfaces used to collect the statistics.

The metrics reported for a given instance that will be included are:

CPU Statistics
CPU Utilization
vNIC Statistics
vNIC Utilization
Disk Statistics
Disk Rates
Disk IOPs

Metrics that will not be included are:

Disk Latency
Disk Info

Alternatives

An alternative would be to keep the inspector in a separate StackForge project. This was not seen as viable due to how the inspectors are derived. Ceilometer depends on the inspectors residing in the main ceilometer.compute.virt project.

This approach could be supported if the get_hypervisor_inspector method was extended to support external projects.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

The inspector should be running on the PowerVM host system in order to collect the information. This is in line with the Nova compute driver and the Neutron ML2 agent.

Performance/Scalability Impacts

None.

Other deployer impact

The hypervisor_inspector will have a new option available called ‘powervm’. No database impacts would occur as the standard statistics would be supported.

There will be a Chef blueprint that is being developed and worked with that team to support the deployment of the PowerVM drivers/agents. The changes there would support the use of the Ceilometer inspector for PowerVM.

Developer impact

None. Unit tests would be developed for the new inspector but are contained within the given package.

Implementation

Assignee(s)

Primary assignee:: thorst
Other contributors:: adreznec, efried
Ongoing maintainer:: thorst, efried

Work Items

Deliver the package structure for the powervm inspector.
Develop the CPU Statistics method and unit tests.
Develop the CPU Utilization method and unit tests.
Develop the vNIC Statistics method and unit tests.
Develop the vNIC Utilization method and unit tests.
Develop the Disk Statistics method and unit tests.
Develop the Disk Rates method and unit tests.
Develop the Disk IOPs method and unit tests.

Future lifecycle

The PowerVM Drivers team that resides within IBM will continue maintenance and support of the inspector for future releases. The PowerVM Drivers team is outlined here: https://launchpad.net/~powervm-drivers

The maintenance and support that will be provided is, at a minimum:

Changes in the PowerVM inspector due to base library changes
Ongoing testing and validation of the PowerVM inspector
Bug fixes identified within the PowerVM inspector

Dependencies

pypowervm: https://github.com/pypowervm/pypowervm

Testing

The existing tempest tests will be run against the PowerVM inspector. The tempest tests are hypervisor-agnostic, allowing the existing tempest tests to be run against the PowerVM polling code without changes.

The PowerVM Drivers team will directly run the tempest tests against a PowerVM system (using the Nova PowerVM Driver and Neutron PowerVM ML2 Agent).

Documentation Impact

The configuration reference should include an update for the PowerVM inspector: http://docs.openstack.org/kilo/config-reference/content/ch_configuring-openstack-telemetry.html

References

Python library for PowerVM:

pypowervm: https://github.com/pypowervm/pypowervm

Corresponding OpenStack Projects:

nova-powervm: https://launchpad.net/nova-powervm
neutron-powervm: https://launchpad.net/neutron-powervm

Note: These are StackForge projects while we iterate with the core teams to bring them into the core. Neutron will stay a StackForge project given the decomposition goals of Neutron itself.

PowerVM as a Hypervisor:

Hypervisor Landing Page: http://www-03.ibm.com/systems/power/software/virtualization/
Hypervisor Documentation: http://www.redbooks.ibm.com/abstracts/sg247940.html
Hypervisor Best Practices: http://www.redbooks.ibm.com/abstracts/sg248062.html

Mandatory API Query Limits

Wed, 10 Jun 2015 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/ceilometer/+spec/mandatory-api-limit

Currently, Ceilometer’s api allows users to query for as much or as little data as they desire. They are also allowed to query for everything no matter the size of results.

Problem description

Allowing the ability to have return the entire world of data is not a realistic use case. While it is easier in theory to query for everything, in reality, this is not viable as it puts a lot of strain on CPU and memory to manage all the data required. In addition, these queries take a significant amount of time to calculate and will always time out even if enough resources are available to the system.

As Ceilometer intends to remove unused pagination, another form of query restriction is required.

Proposed change

Mandatory limits must be enforced on queries, whether it be a limit value. If neither value is provided, a default restriction of 100 will be applied. This behaves similarly to other dbs such as ElasticSearch which limit the number of results returned if no restrictions are provided.

Users are still allowed to query the entire set of data if they choose to, but instead of implicitly doing so as it currently functions, users will need to explicitly do so.

These restrictions will apply to both events and meters api.

Alternatives

Nothing, and let users blame Ceilometer for insane queries.

Data model impact

None

REST API impact

All GET queries will require a limit value. If none is given, a default of 100 will be applied.

Security impact

None

Pipeline impact

None

Other end user impact

A limit value is required to get data as expected. If none is provided, the returned results may be truncated. A warning will be logged whenever the default limit is applied.

Performance/Scalability Impacts

This will limit the load created by unintentional large queries.

Other deployer impact

A new option will be defined to allow operators control of what the default return limit is.

Developer impact

Unit tests need to be written to consider this limit.

Implementation

Assignee(s)

Primary assignee:: gordc
Other contributors:: None
Ongoing maintainer:: None

Work Items

Add option to control default limit value
Apply limit check to meter queries and fix UT
Apply limit check to event queries and fix UT
Update docs

Future lifecycle

We can derive a query link which returns next set of data based on last timestamp value of returned set.

Dependencies

None

Testing

Unit test to test default limits are applied.

Documentation Impact

This may be a breaking change to some – Ceilometer will probably be already broken to them. That said, this will need to be publicised.

References

Resource Meta-data Caching

Fri, 22 May 2015 00:00:00

This was discussed at the YVR summit as follow up to issues raised during the Operators feedback session.

Problem description

Ceilometer polls regularly via the compute-agent to get resource meta-data’ for all instances. This polling adds load to Nova, Neutron, and Keystone APIs, which at scale causes a significant performance impact. This meta-data does not change very often, and the load is likewise often unnecessary.

Proposed change

Lets take advantage of the changes-since parameter in the nova api call. We’ll start by creating an in-memory cache to store the resource meta-data on the initial polling and the time of that polling. Then the compute agent will use the changes-since parameter in all subsequent pollings. This will reduce the amount of data queried and returned by the Nova API. The results of this call will update the cache, and the cache will be used to return the data. Also, we should populate the instance flavor and image data in the cache as well.

Alternatives

Do nothing

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Should provide greater overall scalability due to the reduction in API load and reduced traffic flows.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: jasonamyers
Other contributors:: None
Ongoing maintainer:: jasonamyers

Work Items

Implement an in-memory cache in the compute agent.

Change compute-agent to populate an in-memory cache.

Change compute-agent to use the changes-since parameter in the Nova API call .

Change compute-agent to record the polling timestamp in the cache.

Change compute-agent to cache the flavor and image metadata as well.

Future lifecycle

None

Dependencies

None

Testing

We will need to add tests for meta-data cache flag, API changed-since queries, compute-agent cache control, and image/flavor caching.

Documentation Impact

We will need to document the changes to the polling process, the new meta-data cache config option, and create new documentation that describes the caching work-flow.

References

https://etherpad.openstack.org/p/YVR-ops-ceilometer

Adopt Oslo Guru Meditation Reports

Thu, 14 May 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/guru-meditation-report

This spec adopts Oslo Guru Meditation Reports in Ceilometer. The feature will enhance debugging capabilities of all Ceilometer services, by providing an easy and convenient way to collect debug data about current threads and configuration, among other things, to developers, operators, and tech support in production deployments.

Problem description

Currently, Ceilometer doesn’t provide a way to collect state data from active service processes. The only information that is available to deployers, developers, and tech support is what was actually logged by the service. Additional data could be usefully used to debug and solve problems that occur during Ceilometer operation. We could be interested in stack traces of green and real threads, pid/ppid info, package version, configuration as seen by the service, etc.

Oslo Guru Meditation Reports provide an easy way to add support for collecting the live state info from any service. Report generation is triggered by sending a special(USR1) signal to a service. Reports are generated on stderr, and can be piped into system log based on need.

Nova has supported Oslo Guru Meditation Reports.

Proposed change

First, a new oslo-incubator module (reports.*) should be synchronized into Ceilometer tree. Then, each service process needs to initialize the error report system before the process executes launch method in the main function.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

This feature could expose service internals to someone who is able to send the needed signal to a service. That said, we can probably assume that the operator is already authorized to achieve a lot more than just having an access to stack traces and configuration used. Of course, if deployers are afraid of the information leak for some reason, they could also make sure their stderr output is channeled into safe place.

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

The feature does not require any additional resources until it’s triggered by the user and reports are not expected to be generated too often, so it has little impact on performance and scalability.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: zhangtralon

Work Items

sync reports.* module from oslo-incubator
adopt it in all ceilometer services under ceilometer/cmd/
add some developer docs on how to use this feature

Future lifecycle

None

Dependencies

Currently, the reports.* module from oslo-incubator is graduating into oslo.reports in Library. In case it’s graduated into oslo.reports before Ceilometer switches to it, we will not need to synchronize the reports.* module.

Testing

None

Documentation Impact

Documentation should be extended to describe the new feature.

References

[1] oslo-incubator module: http://git.openstack.org/cgit/openstack/oslo-incubator/tree/openstack/common/report

[2] nova guru meditation reports: https://blueprints.launchpad.net/nova/+spec/guru-meditation-report

[3] blog about nova guru reports: https://www.berrange.com/posts/2015/02/19/nova-and-its-use-of-olso-incubator-guru-meditation-reports/

[4] oslo.reports repo: https://github.com/directxman12/oslo.reports

Declarative Notification Handling

Tue, 28 Apr 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/declarative-notifications

The goal of this spec is to propose a more declarative approach to defining notification events.

Problem description

The existing implementation of notification handling has various limitations. The issues this proposal addresses include:

There is no way to selectively listen to specific event topics of interest.
Capturing notifications of interest should not require code changes.
Adding new event exchanges involves same routine code changes. Too much code duplication.
Requires more insight into the code base than needed
more elegant exchange control

This proposal takes a more declarative approach in addressing these concerns.

Proposed change

Firstly, we define a notification event template where the event signature and attributes are set. This is simple yaml file of the following format:

---
resources:
  - name: image
     type: "gauge"
     topics:
          - "image.size"
     volume: payload.size
     unit: B
     counter_name: image.size
     tenant_id: payload.tenant_id
     user_id: payload.user_id
     instance_id: payload.instance_id

This essentially depicts the boiler plate code we add in notifications.py. A base notification handler loads the event definition and does the rudimentary steps of calling the process_notifications and yielding samples. The advantage of this is that in future when new events need to be added, the end user only need to update the yaml.

Also note that we’re in the process of deprecating existence meters. All the current existence meters will be put at the bottom of the template file with a note saying they will be removed in ‘M’ release.

Secondly, we need to elegantly handle exchange control as part of this definition. The exchanges are specified in ceilometer.conf

We already have exchange definitions as part of ceilometer.conf. We could leverage these as it is and assume that if these are defined then we will enable them. We could also add a new option to specify all the exchanges we like to allow with same exchange name as defined. These exchanges will be read by the base notification handler and get_targets logic will loop through the allowed exchanges and sets up the exchanges. If an event name is part of the exchange names, we’ll skip processing these notifications and move on.

Alternatives

We could leverage the existing event_definition.yaml used for conversion. But i’m not sure how free flowing and flexible that would be. I’m open to this idea.

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Pradeep Kilambi <pkilambi@redhat.com>
Other contributors:: Pradeep Kilambi <pkilambi@redhat.com>

Work Items

Work items: * Define Event notification template * Implement generic notification handler to process the template and generate samples * Add support for handling exchange control * Clean up existing handlers * Add and update test coverage

Future lifecycle

None

Dependencies

None

Testing

Add unit test coverage

Documentation Impact

Document new ways of enabling notification events and exchanges via the template file with examples.

References

https://etherpad.openstack.org/p/ceilo-declarative-notifications

Declarative snmp metrics

Tue, 28 Apr 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/declarative-snmp-metrics

This blueprint wants to add a mechanism so we can add new snmp metrics in a declarative way instead of writing new source code for new pollsters.

Problem description

Currently, we use a generic snmp hardware inspector and a python dict which defines the mapping between snmp oid/operations and the metrics value the caller wants(See the following link for the description of that mapping).

https://github.com/openstack/ceilometer/blob/stable/kilo/ceilometer/hardware/inspector/snmp.py#L112

On top of that generic snmp inspector, we still have separate hardware pollsters doing things like setting multiple fields in the Sample object. It’s ok to add a dozen of new metrics. But if want to add hundreds of new snmp metrics, we need to do it in a declarative way, instead of writing new pollster code for every new snmp metrics added.

Proposed change

While keeping the current snmp related hardware.* pollsters intact for backward compatibility, we’ll add a new generic snmp pollster, which can return different types of snmp related metrics based on user request.

This generic snmp pollster will load a definition from a yaml definition file which defines the parameters used to call into the generic snmp inspector, and also other fields in the final Sample object the pollster returns, like name/unit/type. However, the resource_id/user_id/project_id will still be read from the resources returned by the discover just like what we do today.

The yaml definition would be something like the followings:

---
- counter_name: hardware.cpu.load.15min
  unit: process
  type: gauge
  snmp_inspector:
    matching_type:  type_exact,
    oid:  "1.3.6.1.4.1.2021.10.1.3.1",
    type: float
    metadata: { }
    post_op:

- counter_name: hardware.network.incoming.bytes
  unit: B
  type: cumulative
  snmp_inspector:
    matching_type: type_prefix
    metric_oid:  "1.3.6.1.2.1.2.2.1.10"
    metadata: {
            name: {
                     oid: "1.3.6.1.2.1.2.2.1.2",
                     type: "str",
            },
            speed: {
                     oid: "1.3.6.1.2.1.2.2.1.5",
                     type: "lambda x: int(x) / 8",
            },
      }
    post_op: "_post_op_net"

Please see the following link for the detailed explanation of the snmp_inspector dict in the definition:

https://github.com/openstack/ceilometer/blob/stable/kilo/ceilometer/hardware/inspector/snmp.py#L112

When adding new snmp metrics, we only needs to add the new definitions into the yaml definition. No need to writing new code.

Because the new generic snmp pollster could return multiple different types of meters, it would bring the following changes to the current polling agent implementation:

1. Currently, because of the mapping of meter pollster plugin one meter, when deciding whether a pollster is needed for a given pipeline, we use the pollster plugin’s entry point name. This will be changed. We’ll add a new helper function which could return all the meters that pollster could return, and use that helper function in the above process to decide whether a pollster is needed for a given pipeline.

2. When calling the pollster plugin to get the samples, we’ll pass the pipeline definition as a new parameter to get_samples() call. So instead of return all the samples, the pollster can only return what the pipeline source asks for.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

No changes for the admin in how to configure the pipeline yaml file.

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

Need to deploy a new yaml definition file on the system running polling agent.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <lianhao-lu>
Ongoing maintainer:: <lianhao-lu>

Work Items

enhance the pipeline to support new source definition
implement the generic snmp pollster and its yaml definition file

Future lifecycle

To be maintained by <lianhao-lu>

Dependencies

N/A

Testing

The functionality test should be covered in the 3rd party CI of Intel Hradware-Meters CI.

Documentation Impact

Need to update the openstack-manual admin manual about the change to the pipeline definition.

References

N/A

Create functional and integration tests in ceilometer project

Wed, 15 Apr 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-functional-tests

At this moment there are no integration tests for Ceilometer outside of Tempest, which are fully covering all its functionality. On Paris summit it was decided to move part of the tests (and the implementation of new ones) in the code tree of Ceilometer itself. Also tests for different databases should be moved to this module as well (currently they are living in the folder of unit tests and this doesn’t correspond, generally speaking, to their aim). And all unit tests should be moved from ceilometer/tests/ path to ceilometer/tests/unit folder, so that ceilometer/tests tree contains only three folders: unit, functional, integration.

It is why we should develop integration tests and create Jenkins job for them, which will run these tests, located in a separate module within the Ceilometer code tree (tests for different backends will be moved to this module as well). These tests will be designed in DSVM-style. Tests that require communication with real other OpenStack services, will follow Tempest practice to use master branches for these components.

Problem description

Implementation of the plan will take a long time because of many subtasks and interaction with other projects (in particular with OpenStack infra). In this approach we may implement tests which are not assumed in tempest: * grey/white-box tests that are not pure API tests and may assumed some knowledge of the implementation * tests that require acceleration of the polling interval in order to complete within a reasonable time, e.g. changing the cpu_source interval in the pipeline.yaml and restarting the compute agent so as to gather sufficient cpu_util datapoints to drive alarming tests * tests that depend on the timeliness of incoming notifications from other OpenStack services The implementation of this task will greatly increase test coverage for ceilometer project, including the features which weren’t tested before.

Proposed change

Alternatives

Contribute new integration tests to the tempest project. This approach does not satisfy us because OpenStack QA team rejects too specific integration tests for projects, as only projects teams can decide if they are good or not.

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: vrovachev <vrovachev@mirantis.com>
Other contributors:: chdent <chdent@redhat.com>

Work Items

Work items: * Add ceilometer/tests/{integration|functional} testcases that use the python-clients directly. * Move ceilometer/tests testcase to ceilometer/tests/unit. * Move all existing unit tests to ceilometer/tests/unit. * Move existing DB scenario tests to ceilometer/tests/{integration|functional}. * Add new tox target(s)(functional, integration and others if necessary) to ceilometer/tox.ini. * Add {pipeline}-ceilometer-dsvm-functional-{datastore} job template to os-infra/project-config/jenkins/jobs/ceilometer.yaml where datastore in [‘sqlalchemy’, ‘mongodb’, ‘hbase’].

Future lifecycle

None

Dependencies

None

Testing

None

Documentation Impact

None

References

None

Track DNS Service Notifications

Wed, 15 Apr 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/dns-service-notifications

The goal of this blueprint is to capture the notification events emitted by Designate (DNSaaS) during create, update and delete operations.

Problem description

Designate (designate-central) emits notifications in response to user actions by designate-api and other openstack service (nova, neutron) events by designate-sink.

These notifications are CRUD events, that notify the creation, update or delete of a DNS resource, with no specific measurement value.

Ceilometer needs to be updated to consume and record these notifications as events and traits.

If Ceilometer processes and record these notifications, other services will be able to use this for aggregation and billing purpose.

Proposed change

Define event definitions and field mappings(trait definitions) for DNS resource types - domain, record in the /etc/ceilometer/event_definitions.yaml file.

The trait field syntax in trait definitions follow a variant of the JSONPath.

Example of the event definition and field mappings for the domain notifications that need to go in event_definitions.yaml,

event_type: dns.domain.*
traits: &dns_domain_traits
status:
fields: payload.status
retry:
fields: payload.retry
description:
fields: payload.description
expire:
fields: payload.expire
email:
fields: payload.email
ttl:
fields: payload.ttl
action:
fields: payload.action
name:
fields: payload.name
id:
fields: payload.id
created_at:
fields: payload.created_at
updated_at:
fields: payload.updated_at
version:
fields: payload.version
parent_domain_id:
fields: parent_domain_id
serial:
fields: payload.serial

To enable event processing in Ceilometer, configuration is needed in ceilometer.conf,

Configure store_events as True under the notification section,

eg - [notification] store_events = True

Specify the event_definitions file name under the event section,

eg - [event] definitions_cfg_file = event_definitions.yaml

The dns event types have the following pattern,

dns.<resource>.create
dns.<resource>.update
dns.<resource>.delete

, where resource is domain, record, pool.

Alternatives

Transform DNS notifications as samples using notification plugin to listen for notifications at an exchange and topic. This could be useful when the notifications represent something other than CRUD events that contain a value of interest that can be measured.

Using events provides better querying of metadata and avoids the effort to aggregate samples with a fixed measurement value of 1.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

No new impacts. Pre-existing concerns with capacity at the notification and storage handling layers remain.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: rjaiswal
Ongoing maintainer:: rjaiswal

Work Items

Event definitions and field mappings for all supported DNS resource types.
Event validation.

Future lifecycle

In the future Designate will emit the dns.domain.exists event. This will need to be handled when designate emits this notification.

The set of dns resource types - (domain, record, pool) may change going forward. There could be new types of notifications which will need to be supported.

When Designate confirms with the PaaS event format, the field mappings for integrated events will need to be refactored to adjust to the new event format.

Dependencies

DNSaaS is construed as a PaaS service by some and might have it own message bus where it is sending notifications. Ceilometer recently was extended to consume notification from multiple message bus - https://review.openstack.org/#/c/77612/

If there are multiple message buses, then ceilometer will need multiple transport endpoints configured in ceilometer configuration. (ceilometer.conf)

Testing

Unit and integration Tests will be added to validate events generated.

Documentation Impact

None.

References

http://docs.openstack.org/developer/ceilometer/events.html

http://docs.openstack.org/trunk/config-reference/content/ch_configuring-openstack-telemetry.html

http://specs.openstack.org/openstack/ceilometer-specs/specs/juno/paas-event-format-for-ceilometer.html

https://wiki.openstack.org/wiki/Designate

https://github.com/kennknowles/python-jsonpath-rw

Healthcheck Middleware Metering

Wed, 15 Apr 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-healthcheck-middleware

Oslo.middleware is going to provide a healthcheck middleware in order to monitor health of HTTP services. We want to monitor this middleware response code and time in Ceilometer in order to have analytics available.

Problem description

The healthcheck functionnality provided by oslo.middleware will provide information about the health of a particular API service. It’ll be useful to retrieve periodically the state of the service in Ceilometer to have the ability to analyse these data.

Proposed change

Let’s write a pollster that polls the healthcheck middleware to generate samples measuring the response time of the middleware, and incorporating the status code as metadata.

In order to find which endpoints to poll, the pollster will rely on the EndpointDiscovery discover to find them.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: jdanjou
Other contributors:: sileht
Ongoing maintainer:: jdanjou sileht

Work Items

Write a pollster

Future lifecycle

As all pollsters.

Dependencies

None

Testing

We should be able to test using unit testing and testing inside Tempest. Devstack should have support of this healthcheck middlware by default.

Documentation Impact

The new pollster should be documented the same way we do for others.

References

oslo.middleware healthcheck blueprint <https://blueprints.launchpad.net/oslo.middleware/+spec/oslo-middleware-healthcheck>_

Event Alarm Evaluator

Mon, 13 Apr 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/event-alarm-evaluator

This blueprint proposes to add a new alarm evaluator for handling alarms on events passed from other OpenStack services, that provides event-driven alarm evaluation which makes new sequence in Ceilometer for handling alarms on events separated from other types of alarms handled in the existing polling-based Alarm Evaluator, and realizes immediate alarm notification to end users.

Problem description

As an end user, I need to receive alarm notification immediately once Ceilometer captured an event which would make alarm fired, so that I can perform recovery actions promptly to shorten downtime of my service. The typical use case is that an end user set alarm on “compute.instance.update” in order to trigger recovery actions once the instance status has changed to ‘shutdown’ or ‘error’. It should be possible for an end user to receive a notification within 1 second of a fault being observed, as with other health- check mechanisms can do in some cases.

The existing Alarm Evaluator is periodically querying/polling the databases in order to check all alarms independently from other processes. This is good approach for evaluating an alarm on samples stored in a certain period. However, this is not efficient to evaluate an alarm on events which are emitted by other OpenStack servers once in a while.

The periodical evaluation leads delay on sending alarm notification to users. The default period of evaluation cycle is 60 seconds. It is recommended that an operator set longer interval than configured pipeline interval for underlying metrics, and also longer enough to evaluate all defined alarms in certain period while taking into account the number of resources, users and alarms.

Proposed change

The proposal is to add a new event-driven alarm evaluator which receives messages from Notification Agent and finds related Alarms, then evaluates each alarms;

New alarm evaluator receives event notification from Notification Agent by which adding a dedicated notifier to publish event in a new topic. The topic name is ‘alarm.all’.
When new alarm evaluator received event notification, it queries alarm database by Project ID written in the event notification. To reduce heavy load of Ceilometer API, this alarm evaluator would cache alarm definitions which queried by Project ID in a certain period.
Found alarms are evaluated by referring event notification.
Depending on the result of evaluation, those alarms would be fired through Alarm Notifier as the same as existing Alarm Evaluator does.

This proposal also adds new alarm type “event” and “event_rule”. This enables users to create alarms on events. The separation from other alarm types (such as “threshold” type) is intended to show different timing of evaluation and different format of condition, since the new evaluator will check each event notification once it received whereas “threshold” alarm can evaluate average of values in certain period calculated from multiple samples.

The new alarm evaluator handles “event” type alarms, so we have to change existing alarm evaluator to exclude “event” type alarms from evaluation targets.

Project ID of events would be retrieved from traits named ‘project_id’ and ‘tenant_id’ by this alarm evaluator, since there is no common field to hold it. Not all events have project ID; all events of virtual resource like VM instance should have project ID which belongs to, but events of physical resources like host don’t have project ID. Since those raw infra events have to be hidden from end users, those will be treated as labeled with PROJECT_NONE (’’ in API) which can be set in an alarm definition only by admin.

Alternatives

There was similar blueprint proposal “Alarm type based on notification”, but the approach is different. The old proposal was to adding new step (alarm evaluations) in Notification Agent every time it received event from other OpenStack services, whereas this proposal intends to execute alarm evaluation in another component which can minimize impact to existing pipeline processing.

Another approach is enhancement of existing alarm evaluator by adding notification listener. However, there are two issues; 1) this approach could cause stall of periodical evaluations when it receives bulk of notifications, and 2) this could break the alarm partitioning i.e. when alarm evaluator received notification, it might have to evaluate some alarms which are not assign to it.

Caching all alarm definitions can reduce the number of query to API/DB in each period and reduce time of evaluation in other projects, but it may lead longer time in first query and larger footprint of cache since there can be large number of projects whereas alarms in a project could be limited by quota.

Event ID can be used instead of Project ID in alarm query, but it cannot ensure the number of alarms retrieved from DB and difficult to get all alarms when we allow users to use wildcard in “event_type” of an alarm definition, which is proposed in this spec.

Resource ID could be added to Alarm data model as an optional attribute. This would help the new alarm evaluator to filter out non-related alarms while querying alarms, otherwise it have to evaluate all alarms in the project. To focus on creating basic framework of event alarms, we decided not to handle Resource ID in this blueprint.

Data model impact

None

REST API impact

Alarm API will be extended as follows;

Add “event” type into alarm type list
Add “event_rule” to “alarm”
- “event_rule” has “event_type”, which can include “*”, to indicate which type(s) of event to be evaluated on the alarm
- “event_rule” has “query” which is a list of conditions combined by AND as the same as AlarmThresholdRule

Sample data of Notification-type alarm:

{
    "alarm_actions": [
        "http://site:8000/alarm"
    ],
    "alarm_id": null,
    "description": "An event alarm",
    "enabled": true,
    "insufficient_data_actions": [
        "http://site:8000/nodata"
    ],
    "name": "InstanceStatusAlarm",
    "event_rule": {
        "event_type": "compute.instance.update",
        "query" : [
            {
                "field" : "traits.instance_id",
                "type" : "string",
                "value" : "153462d0-a9b8-4b5b-8175-9e4b05e9b856",
                "op" : "eq",
            },
            {
                "field" : "traits.state",
                "type" : "string",
                "value" : "error",
                "op" : "eq",
            },
        ]
    },
    "ok_actions": [],
    "project_id": "c96c887c216949acbdfbd8b494863567",
    "repeat_actions": false,
    "severity": "moderate",
    "state": "ok",
    "state_timestamp": "2015-04-03T17:49:38.406845",
    "timestamp": "2015-04-03T17:49:38.406839",
    "type": "event",
    "user_id": "c96c887c216949acbdfbd8b494863567"
}

Security impact

Since default event notification may include raw infra information, operator/administrator should configure event definitions carefully when end users is allowed to operate this event alarm API and can receive event alarm notification.

Pipeline impact

This change needs to add new notifier into event pipeline in order to pass event to this new alarm evaluator.

Other end user impact

None

Performance/Scalability Impacts

When Ceilometer received a number of events from other OpenStack services in short period, this alarm evaluator can keep working since events are queued in a messaging queue system, but it can cause delay of alarm notification to users and increase the number of read and write access to alarm database.

All event alarms defined in the project will be evaluated every time this evaluator received event. The number of alarms to be evaluated can be reduced by adding new parameter (e.g. Resource ID) on Alarm data model and setting filter while alarm querying.

Other deployer impact

Notification Agent have to be configured to publish event to a new topic ‘alarm.all’ which is dedicated to this event alarm evaluator and different from current messaging to store events (i.e. add ‘notifier://?topic=alarm.all’ in event_pipeline.yaml). Note this configuration should be done when the new event alarm evaluator runs in the deployment, otherwise it may fill up the queue.

New service process (this alarm evaluator) have to be run.

A deployer can run multiple evaluators in order to scale out event alarm evaluation process. All event will be dispatched to all evaluators listening the topic in a round robin fashion.

Developer impact

Developers should be aware that events could be notified to end users and avoid passing raw infra information to end users, while defining events and traits.

All events related to virtual resources should have project ID and user ID properly.

Implementation

Assignee(s)

Primary assignee:: r-mibu
Other contributors:: lianhao-lu edwin-zhai
Ongoing maintainer:: None

Work Items

Add new alarm type “event” as well as AlarmEventRule
Modify existing alarm evaluator to filter out “event” alarms
New event-driven alarm evaluator
Make the new evaluator cache alarm definitions

Future lifecycle

This proposal is key feature to provide information of cloud resources to end users in real-time that enables efficient integration with user-side manager or Orchestrator, whereas currently those information are considered to be consumed by admin side tool or service. Based on this change, we will seek orchestrating scenarios including fault recovery and add useful event definition as well as additional traits.

This feature will or can be enhanced by the followings in the future;

Enabling this evaluator to get delta of alarm definitions in the last period, in order to reduce traffic on updating cache. This requires alarm storage to hold deleted alarms and alarm API amendment.
Adopting similar coordination process as the notification agent has, for efficiency of this evaluator e.g. reducing cache of alarm definitions. Key for partitioning might be ‘project_id’, ‘event_type’ or ‘resource_id’. For this coordination, all topic name will have the same prefix ‘alarm.’, so that evaluator can use topic=’alarm.*’ to listen all messages for event alarm evaluation. This is the reason why we use ‘alarm.all’ in this spec.
Mechanism to update cache of alarm definition promptly; poisoning cache on evaluator in which assigned alarm definition has updated, etc.
Filtering out uninterested events in notification agent by leveraging the mechanism of graceful pipeline update and reflecting alarm definitions configured by end users.

We can refactor function to retrieve project ID from event object after creating common field in event object that requires DB and API changes.

Dependencies

None

Testing

New unit/scenario tests are required for this change.

Documentation Impact

Administrator Guide and Installation Guide in OpenStack Manuals should be updated to describe new alarm type and rule as well as all deployer impacts.
Proposed evaluator will be described in the developer document.

References

OPNFV Doctor project: https://wiki.opnfv.org/doctor
Blueprint “Alarm type based on notification”: https://blueprints.launchpad.net/ceilometer/+spec/alarm-on-notification
Liberty Summit Note: https://etherpad.openstack.org/p/event_alarm

Add hardware.memory.buffer and hardware.memory.cached metrics

Mon, 23 Mar 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/hardware-memory-buffer-and-cache-metrics

Add hardware.memory.buffer and hardware.memory.cached metrics to monitor the memory buffer size and memory cache size of a physical machine through SNMP.

Problem description

Currently Ceilometer only support 4 memory oid of SNMP:: _memory_total_oid = “1.3.6.1.4.1.2021.4.5.0” _memory_avail_real_oid = “1.3.6.1.4.1.2021.4.6.0” _memory_total_swap_oid = “1.3.6.1.4.1.2021.4.3.0” _memory_avail_swap_oid = “1.3.6.1.4.1.2021.4.4.0”

But in practice, memory cache and buffer size are also very useful information to determine the status of a physical machine.

Proposed change

Add two metrics, hardware.memory.buffer and hardware.memory.cached, to monitor the memory buffer size and memory cache size of a physical machine.

To achieve this, we need add two SNMP oid and two hardware pollsters.

Firstly, add two oid in SNMP inspector:

_memory_buffer_oid = “1.3.6.1.4.1.2021.4.14.0” _memory_cached_oid = “1.3.6.1.4.1.2021.4.15.0”

Secondly, add two Hardware Pollsters in hardware.pollsters.memory:

MemoryBufferPollster

MemoryCachedPollster

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: luogangyi

Work Items

Add two oid in SNMP inspector:

_memory_buffer_oid = “1.3.6.1.4.1.2021.4.14.0” _memory_cached_oid = “1.3.6.1.4.1.2021.4.15.0”
Add two Hardware Pollsters in hardware.pollsters.memory:
- MemoryBufferPollster
- MemoryCachedPollster

Future lifecycle

None

Dependencies

None

Testing

Need unit test

Documentation Impact

None

References

[1] oid references http://www.net-snmp.org/docs/mibs/ucdavis.html

Regexp in complex query

Fri, 06 Feb 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/regexp-in-complex-query

Problem description

Currently some instance or network related service have extra identifier in resource_id. Example is disk.device.* metrics. They have resource_id with format - <instance_id>-<disk-id>. We can’t make query for getting all samples from these meters with specified instance_id because disks have different ids. This behavior not easy for end users and adds troubles for someone who wants know more information about cluster work.

Proposed change

To support complex queries described above we should add new regex operator with corresponding “=~” designation.

Implementation depends on backend:

1) MongoDB and DB2. These dbs has $regex query support. It provides regular expression capabilities for pattern matching strings in queries. MongoDB and DB2 use Perl compatible regular expressions (i.e. “PCRE” ) version 8.30 with UTF-8 support. For correct work we map “=~” from filter query to “$regex” in pymongo request.

SQLAlchemy. This backend supports “regexp” MySQL operator, also it’s mapped
HBase supports regexp queries with RegexStringComparator.

Alternatives

Add extra information to resource metadata

Data model impact

None.

REST API impact

Add new simple operator “=~” in complex queries.

Security impact

None.

Pipeline impact

None

Other end user impact

You may make regex request in complex query.

Performance/Scalability Impacts

Will affect performance for large regular expressions, api service may be affected.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ityaptin
Ongoing maintainer:: enovokshonova

Work Items

add “=~” to simple operators list in controller
add regexp operator processing to backends

Future lifecycle

None

Dependencies

None.

Testing

extend storage scenarios testing to include regexp operator

Documentation Impact

update complex query notes to include new operator

References

https://blueprints.launchpad.net/ceilometer/+spec/regexp-in-complex-query

Adds Disk Capacity, Allocation and Usage metrics implementation in Libvirt

Thu, 15 Jan 2015 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/disk-info-meters

Problem description

Currently, Libvirt Inspector does not collect disk capacity, usage and allocation metrics. This feature can be added for Libvirt to monitor disk metrics : capacity, usage and allocation. It will help to manage disk resources for addition of Virtual Machinesusing same physical disk.

Proposed change

Implement the method ‘inspect_disk_info’ in Inspector for Libvirt. This method will return the disk capacity,usage and allocation per device. The values correspond to values monitored by domblkinfo command of Libvirt.

Add CapacityPollster which create sample :

name=’disk.capacity’
type=sample.TYPE_GAUGE
unit=’B’

Add PerDeviceCapacityPollster which create sample :

name=’disk.device.capacity’
type=sample.TYPE_GAUGE
unit=’B’

Add AllocationPollster which create sample :

name=’disk.allocation’
type=sample.TYPE_GAUGE
unit=’B’

Add PerDeviceAllocationPollster which create sample :

name=’disk.device.allocation’
type=sample.TYPE_GAUGE
unit=’B’

Add PhysicalPollster which create sample :

name=’disk.usage’
type=sample.TYPE_GAUGE
unit=’B’

Add PerDevicePhysicalPollster which create sample :

name=’disk.device.usage’
type=sample.TYPE_GAUGE
unit=’B’

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

In pipeline.yaml, if user is not having * in meters_sink then new meters disk.capacity, disk.allocation, disk.usage, disk.device.capacity, disk.device.usage and disk.device.allocation needs to be added

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ahmed-nooras-saba

Work Items

Add _DiskInfoPollsterBase base class for reading values and aggregating for all devices
Add CapacityPollster , AllocationPollster , PhysicalPollster for disk metrics at instance level
Add PerDeviceCapacityPollster , PerDeviceAllocationPollster and PerDevicePhysicalPollster for disk metrics at device level of an instance
Add inspect_disk_info to Libvirt inspectors
Add new metrics definition in measurements.rst

Future lifecycle

This feature will need testing for next two releases.

Dependencies

libvirt 0.8.1 and above.

Testing

Unit tests are required to test all the new pollsters at Libvirt. The new meters should be discoverable when listing ceilometer meters. Example : ceilometer meter-list

Documentation Impact

Added the following metrics in ceilometer/doc/source/measurements.rst and “measurement section” of http://docs.openstack.org/developer/ceilometer/measurements.html needs to be updated.


disk.capacity	g	B	inst ID	n	1	Capacity of disk in B
disk.allocation	g	B	inst ID	n	1	Allocation of disk in B
disk.usage	g	B	inst ID	n	1	Usage of disk in B
disk.device.capacity	g	B	disk ID	n	1	Capacity per device of disk in B
disk.device.allocation	g	B	disk ID	n	1	Allocation per device of disk in B
disk.device.usage	g	B	disk ID	n	1	Usage per device of disk in B

References

http://osdir.com/ml/libvir-list/2010-04/msg01300.html

Add Support to include Alarm level as part of notifications

Thu, 18 Dec 2014 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-alarm-level

The goal of this blueprint is to expose an alarm priority field that can be used to set the alarm importance level.

Problem description

A detailed description of the problem:

Alarms in ceilometer have no way to identify the criticality of an alarm. All we know today is if alarm has been triggered or insufficient data. This might be ok in general.
But from auditing point of view, cloud administrators would like to know if an alarm is triggered and if so how critical is it. A hardware failure would be a critical alarm vs an occasional spike in cpu level could be medium or low. Differentiating this level would be very useful for audit.

Proposed change

Expose a new field called alarm_priority as part of the alarm base object. This will be exposed via the alarm notifications so the alarms can be filtered by priority.

Alternatives

None

Data model impact

Need to update the Alarm model to include a new field called priority

REST API impact

We will probably want to expose requests to get alarms by priority.

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: pkilambi
Ongoing maintainer:: pkilambi

Work Items

The specific work items include:

Update the model layer to include the necessary fields

Update the notifier module to expose the priority field

Update the rpc and service modules under alarm

Add support for alarm level in the python-ceilometerclient

Update unit tests.

Future lifecycle

None

Dependencies

None

Testing

Unit and integration Tests will be added/updated to cover the necessary scenarios for alarms

Documentation Impact

We might need to update the example json in alarm api docs.

References

Initial implementation is here https://review.openstack.org/#/c/142849/

Adds disk latency metrics implementation in the Hyper-V Inspector

Thu, 11 Dec 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/hyper-v-disk-latency-metrics

High latency between I/O requests can be signs of issues. Collecting disk latency metrics can help us detect those issues. Hyper-V Inspector can collect those metrics.

Problem description

Disk latency metrics are important in telemetry and useful when determining instance’s performance. This spec is about these stats and their implementation in the Hyper-V Inspector.

Proposed change

Add DiskLatencyPollster and PerDeviceDiskLatencyPollster pollsters, which create samples having the properties: * name: ‘disk.latency’ * type: ‘gauge’ * unit: ‘ms’

Add the method ‘inspect_disk_latency’ in Inspector and implementing it in the HyperVInspector, fetching disk latency stats data from Hyper-V VMs, located in the Msvm_AggregationMetricValue objects (further referred to as ‘metrics objects’) associated with the VMs. The metrics objects ‘MetricDefinitionId’ must be the equal to the ‘Id’ of Msvm_AggregationMetricDefinition object having the Caption ‘Average Disk Latency’.

Hyper-V disk metrics were introduced in Windows / Hyper-V Server 2012 R2 (kernel version 6.3). They are not supported in the previous versions.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

Users will have to add meter ‘disk.latency’ in the disk_source source in pipeline.yaml By default, the disk usage metrics collection is enabled in Nova, we just need to collect the data from the Hyper-V API.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <cbelu>

Work Items

Adds the DiskLatencyPollster and PerDeviceDiskLatencyPollster pollsters
Adds the method ‘inspector_disk_latency’ in Inspector.
Implements the method ‘inspect_disk_latency’ in HyperVInspector.
Adds related unit tests.
Updates ceilometer measurements document.

Future lifecycle

Once this feature is enabled, it needs tests and bug fixing in the next 2 releases to avoid regression.

Dependencies

Windows / Hyper-V Server 2012 R2 (kernel version 6.3)
wmi 1.4.9+

Testing

Unit tests are needed to test the new pollsters and the implementation on the Hyper-V side.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

[1] http://msdn.microsoft.com/en-us/library/cc768535%28v=bts.10%29.aspx

Adds memory usage metrics implementation in the Hyper-V Inspector

Thu, 11 Dec 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/hyper-v-memory-metrics

Currently, the Hyper-V Inspector does not collect the memory metrics. This feature can be added, since Hyper-V can measure the amount of memory the instances are using. This is especially useful if the instances are configured to use dynamic memory.

Problem description

Instances’ memory usage metrics are very important in telemetry, but it is not currently implemented in the Hyper-V Inspector. This spec adds memory usage statistics to Hyper-V, so the user can get vital data on the instance’s performance.

Proposed change

Implements the method ‘inspect_memory_usage’ of HyperVInspector, fetches the memory stats data from Hyper-V VMs, located in the Msvm_AggregationMetricValue objects (further referred to as ‘metrics objects’) associated with the VMs. The metrics objects ‘MetricDefinitionId’ must be the equal to the ‘Id’ of Msvm_AggregationMetricDefinition object having the Caption ‘Aggregated Average Memory Utilization’.

Hyper-V metrics were introduced in Windows / Hyper-V Server 2012 (kernel version 6.2). They are not supported in the previous versions.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None. By default, the memory metrics collection is enabled in Nova, we just need to collect the data from the Hyper-V API.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <itoader>

Work Items

Implements the method ‘inspect_memory_usage’ in HyperVInspector.
Adds related unit tests.
Updates ceilometer measurements document.

Future lifecycle

Once this feature is enabled, it needs tests and bug fixing in the next 2 releases to avoid regression.

Dependencies

Windows / Hyper-V Server 2012 (kernel version 6.2)
wmi 1.4.9+

Testing

Unit tests are sufficient, since the implementation will only require data fetching from Hyper-V.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

[1] http://msdn.microsoft.com/en-us/library/cc768535%28v=bts.10%29.aspx

More Power and Thermal Data

Thu, 04 Dec 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/power-thermal-data

Add more power and thermal data besides IPMI sensor data and Node Manager basic data. These data from platform hardware are independent of OS/driver, and valuable to show running status of node.

Problem description

We already have IPMI sensor data and Node Manager basic data, but they don’t provide enough info of overall picture for server in data center. Some extra data can be added, like CUPS(Compute Usage Per Second), which indicates CPU/IO/Memory utilization, and Volumetric Airflow, which indicates current amount of air that goes through server. These data plus previous basic power/thermal data, can be used as input for Nova scheduling.

Proposed change

Add capability to get new data in NodeManager class. Add new pollsters to get CUPS and airflow data.

Add following new metrics:

Name

Type

Unit

Origin

hardware.ipmi.airflow

g

CFM

p

hardware.ipmi.cups.core

g

%

p

hardware.ipmi.cups.io

g

%

p

hardware.ipmi.cups.mem

g

%

p

g = gauge, n = notification , p = pollster, CFM = Cubic Feet per Minute

Name	Type	Unit	Origin
hardware.ipmi.airflow	g	CFM	p
hardware.ipmi.cups.core	g	%	p
hardware.ipmi.cups.io	g	%	p
hardware.ipmi.cups.mem	g	%	p

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

These new data should be exposed via the Horizon metering dashboard.

Performance/Scalability Impacts

Fetching some new metrics would not cause obvious perf drop

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: edwin-zhai
Other contributors:: lianhao-lu

Work Items

Add raw IPMI command to get new data in NodeManager class
Implement 2 new pollster: CUPS and airflow
Add unit test coverage
Update related docs

Future lifecycle

Once this feature enabled, need test and bug fixing in next 2 releases to avoid regression

Dependencies

This feature depends on IPMI/NM capable servers

Testing

This feature with previous IPMI sensor data and NM basic data need 3rd party testing system to verify its function. This testing system development is in progress.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

None

Event Pipelines

Wed, 03 Dec 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/notification-pipeline

Problem description

Currently, when notifications are picked up by the notification agent, they are all funnelled into a single endpoint, filtered using event_definition and pushed to the dispatchers, which is generally a database. There is a lack of flexibility to transform, extend, trigger on these events and to publish to this data to multiple consumers.

Proposed change

Like samples, we have a pipeline for events. Like samples, we define sources and sinks. Unlike samples, we don’t have an interval defined in sources.

This will also allow ability to publish different data (ie. audit) to different storages and also allow ability to ignore notifications completely (currently, we capture a shell event for all notifications)

The proposed schema is:

---
sources:
    - name: eventA_source # any unique name for source
      events: # list of event_types, same wildcard technique in samples
          - "*"
      sinks:
          - sink1
          - sink2
sinks:
    - name: sink1 # any unique name for sink
      transformers:
          # one or many transformers that work on an Event.
      triggers:
          # potential for triggering (short term inline actions). i'd
          # envision a trigger that built performance samples. ie time
          # between corresponding start/end events and republish as a sample
          # alternatively, that same work could send an alarm if it didn't
          # meet a certain threshold
      publishers: # we will not support rpc it isn't great for performance
          - notifier://

Publishers here can be a little different. we currently publish events straight to the database avoiding the collector. In this spec, we will continue this offering in addition to the notifier, udp, file options. The collector offers no discernible benefit apart from moving any synchronous task off the notification agent to the collector. As we already requeue items for each pipeline, this synchronous bottleneck exists only on the pipeline which has the synchronous task and will not slow down the other pipelines.

Alternatives

publish to the collector instead and let it persist data.
put this in same pipeline.yaml file as samples.

Data model impact

None.

REST API impact

None currently but it’ll affect pipeline in database work.

Security impact

Same security concerns as current publishers.

Pipeline impact

This is a completely new pipeline so yes there is impact: a new pipeline.yaml

Other end user impact

You need another pipeline.yaml; You need to configure it appropriately.

Performance/Scalability Impacts

This will by default cause no difference. There’s potential for less data because of ability to filter out events. There’s potential for more data because of transformers and multiple publishers. We have notification agent coordination which will split pipeline and data across agents to handle scaling.

Other deployer impact

Adding a configuration option for new pipeline.yaml

Developer impact

Understand how pipelines work if they don’t understand already.

Implementation

Assignee(s)

Primary assignee:: chungg
Ongoing maintainer:: chungg

Work Items

create event_pipeline.yaml
redirect current event endpoint to use this pipeline
add publisher support for events

Future lifecycle

Build transformers. Build triggers. Build publishers.

Dependencies

None.

Testing

extend pipeline testing to include event_pipeline

Documentation Impact

rewrite pipeline notes to include new event_pipeline and it’s differences

References

https://blueprints.launchpad.net/ceilometer/+spec/notification-pipelines

Support Time To Live on Event Database

Mon, 01 Dec 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/event-database-ttl

Problem description

Event database grows over time, after we dump data to larger storage system, the old data in event database should be cleared, but now, there is no such way to do it.

Proposed change

Add time to live feature on event database, just like what we do on metering database. A new option event_time_to_live will be added, such as what we do for metering database.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

User now can clean event database when they run ceilometer-expirer and set event_time_to_live options to value that larger than 0.

Performance/Scalability Impacts

Performance can be improved since event database can keep light.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: aji-zqfan
Other contributors:: Contributors who want to help on databases except MongoDB
Ongoing maintainer:: aji-zqfan

Work Items

Implement it on MongoDB
Implement it on other database back end

Future lifecycle

None

Dependencies

None

Testing

Unit test code will be added along with source code.

Documentation Impact

New option will be added, so OS Configuration Document should be update, and new feature is added, Administrator’s Guide Document should be updated too. But not this spec’s job.

References

None

Http Dispatcher Spec

Mon, 01 Dec 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/http-dispatcher

Ceilometer dispatcher framework allows various dispatchers to be developed and configured to make ceilometer meter destination to be fully configurable. Currently there are two dispatchers available for Ceilometer, these two dispatchers are used to save Ceilometer meters either to database or log like files. In some applications, meters or filtered meters are wanted to be sent to other third party systems especially using http protocol. This spec provides the specification for a Ceilometer dispatcher using http protocol.

Problem description

The usecase is as follows: Enterprise Monitoring system QRadar needs to capture activities happening in OpenStack keystone such as user creation, deletion, log in etc. Keystone posts these activities onto OpenStack message queues, Ceilometer collector ultimately collects these messages and turn them into meters, so these meters will be sent over to QRadar via the http post request, once the meters get posted via http post request to QRadar, QRadar will analyze the meters and take actions accordingly.

Proposed change

A new dispatcher should be added in Ceilometer/dispatcher package, the dispatcher will be named HttpDispatcher by inheriting the dispatcher.Base class and implement the record_metering_data method along some dispatcher specific configuration parameters. The record_metering_data method will wrap meters in JSON format and post the data by using HTTP POST method to a target. The http request content type should be application/json. If the post target is not defined, then an error should be logged. The post request should use a configurable time to determine if the connection should be timed out. The implementation file should be named http.py so that the module naming pattern consistency is kept. If the meter data contains auditing information (CADF), then the auditing message will be sent rather than the meter itself since most of the information in the meter will be duplicate of the auditing information.

Alternatives

develop an agent and read data from database, but the impact will be data get saved in the database, but the goal is to have the meter info saved in the system which integrates with Ceilometer.
develop an agent read data from queue, but there will be a lot more code needed to do that.

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Tong Li (IBM)

Primary assignee:: litong01
Other contributors:: None
Ongoing maintainer:: Tong Li

Work Items

Http dispatcher
Document on how to configure this dispatcher
test cases

Future lifecycle

None

Dependencies

No new dependencies required. It only requires requests, oslo.config which have been the common dependencies of this project.

Testing

Tempest tests will be added in ceilometer/tests/dispatcher/test_http.py. These test cases can be invoked at the gate or locally in a dev environment.

Documentation Impact

file at /doc/source/install/manual.rst will be changed to reflect this new dispatcher along other existing dispatchers. Since this is a new dispatcher, there will not be changes to any existing components, only additional information will be added to the doc on how to enable and configure this dispatcher.

References

None

Coordination of Notification Agents

Fri, 07 Nov 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/notification-coordination

Problem description

Currently, the notification agent can function in active/active HA where each notification agent grabs notifications from the message queue in a round robin type rotation (it checks for message(s), grabs message(s)). The issue that arises is that if we want to implement a pipeline to process events, we cannot guarantee what event each agent worker will get and because of that, we cannot enable transformers which aggregate/collate some relationship across similar events.

This also fixes an existing bug where if multiple notification agents are enabled, the pipeline transformers may not work as expected because it will only act on the samples it sees on current worker.

Proposed change

Similar to how we implemented coordination between the central agents, this bp is to proposed a way to coordinate which event gets passed to which notification agent.

The proposed solution is to implement additional processing steps as follows:

1. Existing listener in notification agent continues as is, each agent will
   listen to the same queue and grab messages as they arrive without any
   coordination.
2. After the agent has converted messages into samples and events, the result
   will be republished onto a ceilometer internal queue. The sample and event
   will be published to (multiple) topics corresponding to known pipeline
   sinks. For example, the pipeline has two sinks, one that processes all
   meters, the other only processes compute meters. In this case, all agents
   will publish to a queue for 'all-meters' sink, and agents with compute
   meters will publish to 'compute-only' sink.
3. An additional listener will be added to each agent which will listen
   to new internal queues. This is where the coordination will happen. Each
   agent will listen to a set of targets created in step 2.

The existing notification agent will exists as is as there isn’t a need for additional queueing to handle single worker scenario.

Alternatives

The notification agent can be coordinated across existing targets ie. each existing projects exchange. This does not allow for cross exchange sinks. ie. we cannot have an aggregation that combines events from different projects. This might be a non-issue for samples (who needs to aggregate and transform a new sample from samples from nova and neutron?). It will probably be an issue for events which we may want to coordinate across exchanges. This would mean we should have samples listener to coordinate across exchanges, and events listener which would do the above.
The notification agent can be coordinated across endpoints. This will cause issues as we need to have duplicate queues that each agent listens to (to ensure agent with endpoint gets the message it needs). It also runs into potential loss or duplicate messages because of duplicate queues.

Data model impact

None.

REST API impact

None.

Security impact

Same as existing security concerns polling agents face – we need to ensure that phantom agents can’t register and create phantom tasks.

Pipeline impact

It should actually work properly when multiple workers deployed. Additional event pipeline will be added in a subsequent bp and patch.

Other end user impact

None.

Performance/Scalability Impacts

This should improve write performance (at least from SQL pov) as we can do batch inserts.

Other deployer impact

Users will need to enable coordination. By default, the notification agent is expected to continue as currently ie. pull messages if they exists.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: chungg
Ongoing maintainer:: chungg

Work Items

add coordination of notification agent
reuse or add tests for coordination

Future lifecycle

First step to adding in event specific pipeline.

Dependencies

tooz
one of the backends for tooz (ZooKeeper, memcached, redis, etc…)

Testing

coordination tests exist. We may just need to add it for notification agent if the existing agent coordination tests are central agent specific.

Documentation Impact

add notes on enabling coordination of notification agents.

References

[1] https://github.com/openstack/ceilometer-specs/blob/master/specs/juno/central-agent-partitioning.rst

Self-disabled Pollster

Fri, 24 Oct 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/self-disabled-pollster

Avoid loading one pollster if required environment is not ready. Remove resources that cause continuous exception in polling time.

Problem description

Currently pollsters are used in following way.

define via setup.cfg
load each pollster in initialization
pollster’s get_samples get called in periodic way to collect metrics

This fixed process is not flexible enough to handle various situation, like following cases:

One pollster depends on specific environment, e.g compute node pollsters need right hypervisor inspector, ipmi pollsters need ipmitool installed. Loading pollster unconditionally produces extra exceptions or dummy samples with minor performance drop. Check in deployment can not resolve all the issues, as environment changes dynamically. The perfect solution is that check in loading time and do not load the pollster if no required environment.
One pollster gets loaded and runs well, then some polling resource becomes unavailable, so the pollster fails to produce samples and probably throws exceptions with performance drop. We need detect such changes, then remove this resource from polling. Also need put warning in log, so operator can track and handle it.

Proposed change

Provides infrastructure to get pollster specific response then take different action.

For case 1, need provide a function in pollster constructor, to raise specific exception if required hardware is missing. When loading extensions, add on_load_failure_callback, which checks and suppresses the exception to avoid loading this extension. Probably need one configuration list to indicate which pollster can be skipped.

For case 2, we need pollsters throw 2 different exceptions in run time:

transient failure: treat like existing ones
permanent failure: Not poll that resources from this pollster any more

To achieve this, we have PollingTask to catch that permanent failure, then block polling for that resources.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

This BP improve performance due to avoid loading unnecessary pollster and remove unavailable resources to eliminate continuous exceptions.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: edwin-zhai
Other contributors:: lianhao-lu
Ongoing maintainer:: edwin-zhai

Work Items

For loading time pollster
- Add environment check function in pollster constructor
- Add new on_load_failure_callback to avoid loading failed extension
For running time pollster
- Modify required pollster to throw permanent failure exception
- Modify PollingTask to skip failure resources

Future lifecycle

Once this feature enabled, need test and bug fixing in next 2 releases to avoid regression

Dependencies

None

Testing

Need unit test

Documentation Impact

None

References

None

Add new notifications types for volumes/snapshots

Thu, 23 Oct 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/add-new-notifications-types-for-volumes-and-snapshots

Now we have only two types of notifications about volumes/snapshots: volume/snapshot existence and their size. This change allows to collect and view notifications of different types. We can get information about which events have occurred: volume/snapshot has been created or deleted or updated(renamed or modified description), volume has been resized or attached/detached. This will allow to process additional events and will improve the overall Ceilometer functionality.

Problem description

Now we have only two types of notifications about volumes/snapshots: volume/snapshot existence and their size. But there is no information about events like volume/snapshot was created or deleted or updated or volume was resized or attached/detached.

Proposed change

This change allows to collect and view notifications of different types - volume/snapshot.create.start, volume/snapshot.create.end, volume/snapshot.delete.start, volume/snapshot.delete.end, volume/snapshot.update.start, volume/snapshot.update.end, volume.resize.start, volume.resize.end, volume.attach.start, volume.attach.end, volume.detach.start, volume.detach.end.

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: enovokshonova <enovokshonova@mirantis.com>

Work Items

Implement appropriate handler classes.

Future lifecycle

None.

Dependencies

None.

Testing

This change needs to be tested by unit tests.

Documentation Impact

We need to add new meter types to http://docs.openstack.org/developer/ceilometer/measurements.html.

References

https://github.com/openstack/cinder/blob/master/bin/cinder-volume-usage-audit https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py https://github.com/openstack/cinder/blob/master/cinder/api/v2/volumes.py https://github.com/openstack/cinder/blob/master/cinder/api/v1/volumes.py https://github.com/openstack/cinder/blob/master/cinder/api/v1/snapshots.py https://github.com/openstack/cinder/blob/master/cinder/api/v2/snapshots.py https://github.com/openstack/cinder/blob/master/cinder/volume/flows/manager/create_volume.py

Adds memory utilization meter to libvirt inspector

Mon, 20 Oct 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/libvirt-memory-utilization-inspector

Memory usage statistics is not implemented in libvirt inspector. We can get memory stats of the instance from libvirt API ‘virDomainMemoryStats’ in order to add memory usage meter to libvirt inspector.

Problem description

Memory usage of instance is very important data in telemetry, but now it is not implemented in libvirt inspector. This spec adds memory usage statistics to libvirt, so the user can get the data on the performance of the instance.

Proposed change

Implements the method ‘inspect_memory_usage’ of LibvirtInspector, fetches the memory stats data from libvirt API ‘virDomainMemoryStats’, used memory is calculated by the available and unused memory. The libvirt API ‘virDomainMemoryStats’ may raise an exception if the method is not supported by libvirt, refer to ‘Dependencies’ section, catches the exception and translates that into an empty data of memory stats.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

User need to prepare suitable balloon driver in image, particularly for Windows guests, most modern Linuxes have it built in. Booting instance will be successful without image balloon driver, just can’t get guest memory usage meter.

Performance/Scalability Impacts

None

Other deployer impact

None. By default, the memory statistical feature is enabled in Nova, refer to [1], we just fetch and collect the data from libvirt API.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <kiwik-chenrui>

Work Items

Implements the method ‘inspect_memory_usage’ of LibvirtInspector.
Adds relate unit tests.
Updates ceilometer measurements document.

Future lifecycle

Once this feature enabled, need test and bug fixing in next 2 releases to avoid regression.

Dependencies

libvirt 1.1.1+
qemu 1.5+
guest driver that supports memory balloon stats

Testing

Unit tests are sufficient since only data fetching need test.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

MagnetoDB Notifications

Mon, 06 Oct 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/support-magnetodb

This spec proposes to add MagnetoDB metering to Ceilometer by catching notifications emitted by MagnetoDB. For example, when a table is created a notification called table.create.start is emitted and when the table creation is finished a notification called table.create.end is emitted.

Problem description

The notifications sent by MagnetoDB needs to be fetched from message bus and then transformed into samples and then stored in the database. To achieve this a notification handler is needed.

Proposed change

A new notification plugin for MagnetoDB to transform the notifications into samples using existing NotificationBase implementation as a model.

Example of a notification and corresponding sample

List of new samples:

Resource ID:

id

Name:

table.create

Type:

gauge

Volume:

1

Unit:

table

Timestamp:

time
Resource ID:

id

Name:

table.delete

Type:

gauge

Volume:

1

Unit:

table

Timestamp:

time
Resource ID:

id

Name:

index.size

Type:

gauge

Volume:

2

Unit:

index

Timestamp:

time

Alternatives

None.

Data model impact

None.

REST API impact

There will be additional valid values in the query parameters but no changes to API endpoints.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

No new impacts. As we are only storing table creation and deletion notification, the impact would be negligible.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: ajayaa
Other contributors:: None.

Work Items

Establish expected data.
Create tests of transformation of notifications to samples.
Create notification plugin to consume notifications.
Create tests of notifications across fake bus.
Create sample query tests.

Future lifecycle

In future new types of notifications are expected from the MagnetoDB. These will need to be handled either by additional notification plugins or (hopefully) generic notification handling. The MagnetoDB team will be responsible for collaborating with the ceilometer team to ensure these are handled smoothly.

Dependencies

None.

Testing

Unittests.

Documentation Impact

The added metrics will need to be documented in the measurements section.

References

None.

Merge Compute and Central agents

Thu, 04 Sep 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/merge-central-and-compute-agents

Central agent was previously extracted from compute agent code to support not only compute metrics to be polled, but also to collect measurements from other services and resources. Actually the main difference between these two agents was in the way they discover and poll the resources. After the discovery was unified for the all pollsters, we actually are having the logic duplication, having two different agents, although they both discover and poll some resources and collect then some metrics.

It looks logical to merge the code of agents back. They even might have exactly the same pipeline being used, but differ only in the console script parameter –polling-namespace to have the opportunity still to separate them in the real deployment. Also that might be possible to define exact list of pollsters to be used via –pollster-list script parameter here.

Problem description

We have logic mismatch with two separated agents, although they might be part of one piece of code, configured via pipeline.

Proposed change

We need to merge carefully code of compute and central agents, using unified mechanism we already have to process discovery and polling processes.

Also we need to save the opportunity of setting up agent to poll only compute or only meters used to be polled by central agent. Basically all-in-one installation will be satisfied with agent polling all available meters, but production installations need this separation. This will be achieved by adding additional per-agent CLI script parameter: –polling-namespace={compute|central|compute,central} (with adding here ipmi when this agent will be added here as well). In this case running of polling agent will look like following:

ceilometer-polling –polling-namespace central compute –logfile /var/log/ceilometer/polling.log

This parameter will be passed to the agent manager class directly, and will define what setup.cfg namespace (ceilometer.poll.central, ceilometer.poll.compute or both) with possible pollsters to load. This will emulate current workflow (and probably become deprecated in next cycles).

Also this change will propose new way of how do pollsters to be used are chosen for polling agent instance - via CLI script parameter –pollster-list=<list of pollster entrypoints or wildcards>. In this case polling agent running command will look like this:

ceilometer-polling –pollster-list image image.size storage.* –logfile /var/log/ceilometer/polling.log

If both of these parameters will be set, we need to use them both (possibly using logical AND operation to find out final list of pollsters to be used).

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Better RAM usage in case of development OpenStack all-in-one installations.

Other deployer impact

It’ll be less services Ceilometer will run on node (if it’ll be all-in-one installation) and we’ll have more unified way of Ceilometer installation.
Downstream impact on how Ceilometer packages will be created and on the deployment tools that use them right now. Currrently we are having two separated packages for central and compute agents in Ceilometer (both for the .rpm and .deb), and these packages have different .service/.upstart definitions for launching the corresponding service. The simplest way to fix this moment will be to add new combined package and make current separated ones to use introduced in this change namespaces like below:
```
s/ExecStart=...ceilometer-agent-compute/
  ExecStart=...ceilometer-agent-polling --pollster-namespace=compute/
```
We need to update puppet-ceilometer module and add there new manifest that will support combined agent.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: dbelova <dbelova@mirantis.com>

Work Items

Merge central agent code into the current compute one (ceilometer-polling). Leave ceilometer-agent-compute and ceilometer-agent-central CLI commands for a deprecation cycle.
Investigate Tempest testing impact and provide new tests if needed.
Provide Devstack with new agents scheme support (for all-in-one and multinode installations).
Investigate Grenade tests aspect.

Future lifecycle

None

Dependencies

None

Testing

This change needs to be tested by merged unit tests and via integration tests (old and possible new ones).

Documentation Impact

We’ll need to rewrite our installation guide and common documentation parts with the information about one agent instead of two.

References

None

Horizontal scaling and work-load partitioning of the Central Agent

Mon, 04 Aug 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/central-agent-partitioning

The central agent’s job is polling resources for information, transforming that information into samples and passing the samples on to the Collector Agent.

This specification proposes an implementation of coordination between multiple Central Agents, which could then dynamically distribute the workload between them, providing scalability and high availability.

Problem description

Currently, each Central Agent retrieves a set of resources and polls all of them. If we have multiple Central Agents running, they all poll the same set of resources, which prevents us from scaling out horizontally.

At the start of each polling interval, each of the pollsters retrieves a list of resources to poll from its Discovery plugin (configured in the pipeline or a default one). This makes the discovery process a great place to implement the coordination and partitioning logic, while the pollsters themselves can remain in blissful ignorance of anything going on.

Proposed change

The basic idea is to use the tooz [1] library for group membership and hashing to assign resources to active Central Agents.

Discovery plugins on different Central Agents that discover the same set of resources would join the same group in tooz.
For each polling interval, after independently discovering the global set of resources that need to be polled by one or other of the central agents, they would get a list of all group members (Discovery plugins in other Central Agents that have the same set of resources).
They would use hashing to determine the set of resources assigned to the Discovery’s Central Agent.

Determining the resources we’re responsible for

We have a list of resources and get a list of active agents from tooz. We then get our assigned resources as follows:

our_key = sorted(agents).index(our_agent_uuid)
our_resources = []

for resource in resources:
    key = hash(resource) mod len(agents)
    if key == our_key:
        our_resources.append(resource)

# or more pythonic
our_resources = [r for r in resources if hash(r) mod len(agents) == our_key]

In essence we hash the resources to <number of Central Agents> of buckets and only poll the resources that fall into our bucket. A good hash function [3] ensures that the resources are evenly distributed to the active Central Agents.

Grouping

The pollster’s Discovery plugin (be it a Compute Discovery, Hardware Discovery, etc.) provides the scope its resources are a part of.

For example, if a Discovery plugin isn’t constrained to a subset of resources, as is the case for most Discovery plugins, then it should simply join the global group of unconstrained Discovery plugins.

If, on the other hand, the resources that the Discovery plugin can discover are constrained, like in the case of Compute Discovery, then the group name should reflect their scope. An example of this would be ‘compute-<hostname>-discovery’. This way only the pollsters that are polling the same host will share their workload between them.

What happens when we start another agent (or stop an existing one)?

tooz allows us to register a callback that is called when a member joins or leaves the group. It keeps track of member liveness using a heartbeat mechanism.

When a member joins or leaves the group, this is what happens to:

The agents already running If they are currently in the middle of polling, they complete their full polling cycle and only then they re-balance their hash buckets.
The agent we just started Joins a group first, but then waits for one polling interval to ensure all the other agents have updated their hash buckets as well, then starts polling.
The agent that is stopped/crashes mid-cycle The resources that the stopped agent hasn’t polled yet will not be polled in this cycle, but they will be polled from the next one on.

Generalizing the implementation for re-use

The need for coordinated assignment of “things” (resources, alarms, …) to agents is not unique to the Central Agent. Currently, the Alarm Evaluator could make use of it as well to have multiple Alarm Evaluators running, each evaluating their share of alarms.

This functionality could be captured in a PartitionCoordinator class, which agents could use like:

partition_coordinator = PartitionCoordinator(group='alarm')
partition_coordinator.start()

every evaluation_interval:
    all_alarms = get_all_alarms()
    my_alarms = partition_coordinator.get_my_subset(all_alarms)
    for a in my_alarms:
        evaluate(a)

Note

The actual change-over of the alarm partitioning coordination to the proposed approach will be tracked in a separate blueprint.

or in the case of the central agent:

partition_coordinator = PartitionCoordinator(group='central_agent')
partition_coordinator.start()

every polling_interval:
    all_resources = discover_resources()
    my_resource = partition_coordinator.get_my_subset(all_resources)
    for r in my_resources:
        poll(r)

Alternatives

Fabio Gianetti’s approach [2].

Fabio’s approach uses source<->agent assignments in the database for figuring out what to poll and a heartbeat in combination with additional agents listening for that heartbeat for failure detection.

In contrast, this proposal uses tooz for failure detection (via heartbeats as well). Additionally, the resource allocation is more dynamic since the resources are assigned to agents evenly at any point in time. It is also more lightweight since we don’t need to keep an explicit resource<->agent mapping in the database, but use hashing instead.
Locking

Another approach would be to use distributed locking provided by tooz. Before a pollster would poll a resource, it’d need to acquire its lock. Pollsters contend for the locks and whoever gets the lock, polls the resource.

The downside of this approach is the overhead of distributed locking. Acquiring a distributed lock incurs a cost (time, network traffic). When using distributed locks for resource contention, this cost is incurred per-resource. Whereas in the approach with group membership, the coordination cost is incurred only when a member joins/leaves the group, the frequency of which is negligible compared to the amount of resources.

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Positive

Other deployer impact

If deployers want to use multiple central agents, they will need to deploy one of the tooz backends (ZooKeeper, memcached, possibly just an AMQP broker soon)

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: nejc-saje
Other contributors:: chdent
Ongoing maintainer:: ceilometer team

Work Items

Future lifecycle

Dependencies

tooz
one of the backends for tooz (ZooKeeper, memcached, possibly just oslo.messaging)

Testing

The implementation should be tested with unit tests.

Documentation Impact

Operator’s manual should explain the process and properties of running multiple Central Agents.

References

Multi meter arithmetic transformer

Tue, 08 Jul 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/arithmetic-transformer

Adding the ability to take more than one meter into account when performing a transformation. An example of this is the memory utilization, where:

memory_util = 100 * memory.usage / memory

Problem description

Pipeline transformers are currently performing calculations only on one or more values of the same meter. Some calculations need to take into account multiple meters, like in the case of memory utilization, where:

memory_util = 100 * memory.usage / memory

Proposed change

We suggest an implementation of a flexible transformer capable of performing arithmetic operations on multiple meters and/or their metadata. To prevent irregular cadence of the produced meter, calculation is limited to meters with the same interval.

The configuration of the new transformer would be:

- name: "arithmetic"
  parameters:
    target:
      name: "memory_util"
      unit: "%"
      type: "gauge"
      expr: "100 * $(memory.usage) / $(memory)"

To demonstrate the use of metadata, here is the implementation of a silly metric that shows average CPU time per core:

- name: "arithmetic"
  parameters:
    target:
      name: "avg_cpu_per_core"
      unit: "ns"
      type: "cumulative"
      expr: "$(cpu) / ($(cpu).resource_metadata.cpu_number or 1)"

Expression evaluation would gracefully handle NaNs and exceptions. In such a case it would not create a new sample but would only log a warning.

Parsing the meter names

Since meter names are not necessarily valid Python identifiers, we must escape them. The expression string is only parsed at the time the pipeline is initialized, so it does not cause any performance overhead.

The escaping is done as follows:

$(cpu) -> cpu
$(cpu_util) -> cpu_util
$(cpu.util) -> _cpu_util_ESCAPED (in order to avoid clashes with legal
  meter names)
$(class) -> _class_ESCAPED (avoid clash with Python keywords)

Also, if the meter name isn’t followed by a dot (‘.’), we default to the volume parameter.

So, the expression:

$(memory.usage) / $(memory)

will be translated to:

_memory_usage_ESCAPED.volume / memory.volume

and eval’d with the following Namespace object

Namespace({
    "_memory_usage_ESCAPED": { ... memory.usage sample dict ... }
    "memory": { ... memory sample dict ... }
})

Gathering the values

Not all the needed samples will arrive at the same time. Multiple approaches are possible here.

1. [CHOSEN APPROACH] We can cache samples of the necessary meters (keyed by resource ID) as they arrive. When the pipeline is flushed, we can check if we have the necessary samples and do the transformation.

pro: no need for specifying TTL for cached samples
con: arithmetic operation limited only to samples that originate
from the same polling task (=> same interval)

2. We cache the samples same as above, but do the transformation as soon as we have the necessary samples (samples that are used in the calculation expression). After the transformation we clear the cache. The user provides the maximum age (TTL) of a sample for it to be considered for calculation. E.g. we get a sample of ‘memory.usage’ and we already have a sample of ‘memory’ in cache, which is less than TTL=10s old. We can perform the calculation.

pro: meters used not limited to the same polling task
pro: largest flexibility of all options
con: need to specify maximum age for samples to be used in calculation
con: might produce samples with irregular cadence, as in the case of
the calculation involving meter A with 60s cadence and meter B with 45s cadence

Similar as #2, but limit only to meters from sources with the same interval.

pro: produced samples can’t have irregular cadence
pro: since all meters involved have the same interval, we can have a
static TTL for samples, no need for user configuration. If we know that all the samples arrive at approximately the same time (because they are on the same interval and therefore in the same polling task), we only have a short period of time in which the samples are valid (say, 5 seconds).
con: The next question is how to set the TTL? With a 600s cadence,
5 seconds is OK, but with 10s cadence it probably isn’t.
con: how to technically enforce this restriction?

4. Same as #2, but provide a configuration option to not clear the cache after calculation.

pro: enables use cases like “we get ‘memory.usage’ every 10s and ‘memory’
every ‘600s’. If we don’t clear the cache, we can calculate ‘memory_util’ every 10s using the cached value of ‘memory’.
con: extra configuration option might be confusing for the user. The
benefit might outweigh this.

Alternatives

require user to assign the meters to “variables”, e.g.:

- name: "arithmetic"
  parameters:
    target:
      name: "memory_util"
      unit: "%"
      type: "gauge"
      a:    "memory.usage"
      b:    "memory"
      expr: "100 * a / b"

Unnecessary since the parsing is only done at pipeline initialization and so does not cause any overhead in processing.

somehow change the pipeline to only send the samples to transformer when it has all the required samples

Our approach is simpler, since other transformers already use caching so it wouldn’t make sense to push this functionality to the pipeline.

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

It impacts the format of the pipeline.yaml as described in section Implementation.

Other end user impact

None

Performance/Scalability Impacts

None. Expression parsing is performed only at pipeline initialization. In regard to caching, other transformers already cache values between intervals. The only difference is that we would store values for more than one meter.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: nejc-saje
Ongoing maintainer:: nejc-saje

Work Items

create the new transformer
write unit tests for the new transformer

Future lifecycle

None

Dependencies

None

Testing

With the current capabilities of Tempest testing I think unit tests should suffice.

Documentation Impact

Documentation should be added for the new type of transformer.

Documentation must make it clear that the meters included in the expression must have the same cadence. In case that the expression contains meters with different cadences or nonexisting meters, the user should expect to have a warning logged for each unsuccessful calculation.

References

None

Dispatcher for Gnocchi Integration

Tue, 08 Jul 2014 00:00:00

Ceilometer is planned to be re-based on a TimeSeries as a Service concept, namely Gnocchi. We’ve reached the state, when these two projects can be loosely integrated. This blueprint is about to describe the options for implementing a dispatcher for realizing the data flow between the monitored infrastructure and the database.

There are several ways for integrating these two projects. This blueprint represents a first stage, proof of concept solution for testing how the new design is performing. It could/should be refactored later to reach a better performance and closer integration.

https://blueprints.launchpad.net/ceilometer/+spec/dispatcher-for-gnocchi-integration

Problem description

Gnocchi has several new terms comparing to Ceilometer, like entity and archive/archiving policy. Ceilometer does not create/define these items now, therefore it is the key point of the integration process, how to synchronize the mechanisms that are needed for storing the samples and the corresponding metadata in the new structure.

Proposed change

This proposal describes the solution, which is about to implement a new dispatcher that can be used in the collector to send data to Gnocchi, instead of one of the currently supported database backends.

For storing time series data, we need to do several preparation steps in the polling/notification handler workflow. The processed notifications and also the polled data have metadata information attached to it. We need to define resources that will hold this additional information.

The resource creation has to happen before storing any datapoints. To make this mechanism effective and easy to implement and extend later the plan is to create stevedore extensions for the different resource types, which provides a pluggable solution for the future.

These resource creation extensions will each handle the creation of a strongly-typed resource of a single type, and be aware of the resource-specific attributes required in the Gnocchi representation. The loaded resource creation extensions will be managed by the dispatcher as a map keyed on meter-name (acting as a proxy for resource type). Resources of an unrecognized type will be modelled in Gnocchi as generic resources.

The resource creation would happen using PUT request, if the given resource does not exist.

The next step after having a resource is to create entity for the given measurement, that is connected to the previously created resource. The entity creation should also happen automatically, without any human interaction. Meter name will be used to name each entity. As a first step, the entity creation will happen in two steps. First to retrieve the resource and check whether the particular entity exists or not and in the latter case create the entity with a default archiving policy.

Later on as a next step, to avoid additional overhead of checking whether the entity exists or not, we try to POST new measurements with using the following request:

POST /v1/resource/UUID/entity/{entity.name}/measures ...

If the request fails with missing entity error, then we create the entity with using a default archiving policy, as it is described below associate the entity with the resource, and then, and post the measurements again. In the next polling cycle or when the next notification is received for the given entity, the POST request will be successful. This option requires a new operation to be supported on the Gnocchi API, that is why it will be a future improvement.

There is also a need for at least one archiving policy to be existed before storing any time series data. We can reuse the currently existing configuration parameters that are used for the pipelines or the samples lifecycle to define archives.

As these archives can be defined based on the lifespan and granularity values the ttl value that specifies how long the samples should be kept in the database can be reused to define the former parameter. The granularity will be configured as one second by default in the first step.

Later on it will be also supported to change these values or adding new archives via the Ceilometer’s REST API, but currently we provide the mechanism the can be built in the current data collection and storage mechanisms.

The data will be stored by calling the REST API of Gnocchi from the collector.

Alternatives

The alternate solution is to remove the collector from the chain and store the collected data directly from the pollsters. This would increase the performance, as we do not add additional steps before storing the data in the database.

This solution needs more effort, therefore it is a proposed way of future improvement of the integration.

Data model impact

New resource types were introduced by Gnocchi, which are handled inside that module, Ceilometer is aware of the available types, for creating new instances, when it’s necessary.

REST API impact

The data retrieval will change after fully integrating Gnocchi into Ceilometer. This will be introduced by the new v3 API, described in a separate blueprint.

Security impact

None

Pipeline impact

None

Other end user impact

The deployment of Gnocchi should be done before using this new solution.

Performance/Scalability Impacts

One of the goals of this blueprint is to make it possible how this new approach behaves comparing with the old one.

Other deployer impact

None

Developer impact

The new way of storing data will not replace the currently existing solution. If someone wants to develop some feature based on the new approach, he/she has to be aware of how Gnocchi works.

Implementation

Assignee(s)

Primary assignee:: ildikov
Other contributors:: jdanjou
Ongoing maintainer:: ildikov, jdanjou, eglynn

Work Items

create extensions and the additional steps for resource creation
finish the implementation of the dispatcher

Future lifecycle

The mid-term plans are to integrate Gnocchi closely into Ceilometer, if the performance and functionality are both fulfills the requirements. This means that this first step may be changed in some points and the final solution will then be maintained by the Ceilometer core team and further contributors.

Dependencies

The current state of Gnocchi supports this integration.

Testing

Unit tests should be provided for the new implementation.

Documentation Impact

The documentation should be updated with the configuration needs for deploying Ceilometer with Gnocchi and also with the information about how to reach the data via the Gnocchi API, until Ceilometer v3 API is not available.

References

Initial dispatcher implementation

IPMI support

Thu, 03 Jul 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ipmi-support

Add IPMI function in ceilometer, so that power/thermal data can be fetched through IPMI capable servers. These data can be exported to other openstack component for other usage, like policy control.

Problem description

The Intelligent Platform Management Interface (IPMI) is a standardized computer system interface used by system administrators for out-of-band management of computer systems and monitoring of their operation. Enabling IPMI in ceilometer allows the cloud operator to monitor the power and thermal behaviors of servers and make reasonable operation or policy according to the real-time power/thermal status of data center.

With IPMI support,the power and thermal information of these servers can be used to improve overall data center efficiency and maximize overall data center usage. Data center managers can maximize the rack density with the confidence that rack power budget will not be exceeded. During a power or thermal emergency, we can limit server power consumption and send alert to administrators via the pre-defined policy.

Currently, Ironic project is doing work to emit notifications on the message bus containing IPMI sensor data, so that Ceilometer is possible to process these notifications for high level usage. But for nodes that are not managed by Ironic, IPMI data is still missing. This spec removes dependency on Ironic by adding IPMI support in Ceilometer.

Proposed change

Add a new IPMI agent, who get the power & thermal data via IPMI protocol in periodic way and publish these data via pipeline. IPMI agent lives in each IPMI capable node, and gets configured in deployment.

In this way, data can be fetched locally without credentials via network. Also agent polling period can be controlled by pipeline file.

Line up metrics with IPMI support in Ironic, so we get same IPMI data regardless of the underlying method (an agent-per-node directly emitting samples, or the ceilometer notification agent consuming notifications emitted by Ironic. If both available, user need deploy one only):

Name

Type

Unit

Origin

hardware.ipmi.fan

g

RPM

p

hardware.ipmi.temperature

g

C

p

hardware.ipmi.current

g

W

p

hardware.ipmi.voltage

g

V

p

g = gauge, n = notification , p = pollster

Name	Type	Unit	Origin
hardware.ipmi.fan	g	RPM	p
hardware.ipmi.temperature	g	C	p
hardware.ipmi.current	g	W	p
hardware.ipmi.voltage	g	V	p

IPMI is generic protocol, each vendor may add specific metric for their own hardware. So there are probably more vendor specific metric in implementation.

Aim of this spec is to provide: 1) a standard metrics that exist for most of vendor 2) infrastructure to enable IPMI, that is, basic working unit to send IPMI commands.

It could include an simple vendor specific implementation based on 2 as example, so other vendor can do their own work easily.

Alternatives

Instead of deploying IPMI agent for each node, Ceilometer itself hosts one agent for a bunch of nodes. It poll IPMI data from each node via network in periodic way. SNMP support in Ceilometer and IPMI in ironic take similar way.

This method avoid deploying agent in each nodes, but it introduce more work loads in Ceilometer itself, thus bad scalability. Furthermore, IPMI via network requires credential management.

Data model impact

None.

REST API impact

None

Security impact

None.

Pipeline impact

None.

Other end user impact

Power/Thermal data should be exposed via the Horizon metering dashboard.

Performance/Scalability Impacts

By default, IPMI agent is not enabled until operator explicitly turn on it in config file for IPMI capable servers. So no performance impacts by default.

Other deployer impact

IPMI config option(default off) need to be added
an IPMI agent need to be added and deployed to every node, along with ceilometer config such as the metering secret
If both Ironic and Ceilometer available for IPMI data, user need deploy one only

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: edwin-zhai
Other contributors:: lianhao-lu shane-wang

Work Items

Implement an IPMI agent for data fetching deployed in each IPMI capable node.
Add unit test coverage
Update related docs

Future lifecycle

Once this feature enabled, need test and bug fixing in next 2 releases to avoid regression

Dependencies

This feature depends on IPMI capable servers

Testing

Unit tests are sufficient since only data fetching/exporting need test.

Documentation Impact

The added metrics will need to be documented in the measurements section.

The new configuration for deployment need to be documented.

IPMI agent shouldn’t be deployed if a node is managed with Ironic, which need to be documented.

References

http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-home.html

Improve the snmp hardware inspector

Wed, 02 Jul 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/snmp-improvement

We could make some improvement to the existing hardware snmp inspector for the following objectives:

Minimize the coding effort in snmp inspector when adding new hardware metrics
Reduce the network traffic between the inspector and the remote snmpd agent

Problem description

Currently, to add new hardware metrics, the developer needs to do the following:

Add new inspect method in ceilometer.hardware.inspector.base.Inspector class.
Add new returned data type definition of the above new inspect method.
Implement the new inspect method in snmp inspetor.
Add new hardware pollsters and register it in setup.cfg.

This could result in quite a lot of coding, considering the tons of metrics available through snmp which could be added.

Besides, since most of the existing snmpd agents support version 2c, we should leverage the snmp GetBulkRequest introduced in version 2c to minimize the network traffic between the snmp inspector and snmpd agent. If the snmpd agent doesn’t support version 2c and above, we should fall back to GetNextRequest.

Proposed change

Because for each hardware metric, it could be mapped to one or more snmp OIDs. Using these OIDs, the snmp inspector could get the sample value as well as the sample’s metadata.

So here we propose the following changes:

In ceilometer.hardware.inspector.base.Inspector:

We’ll replace the current inspect_xxx() methods with a generic inspect method:

def inspect_generic(self, host, identifier, cache):
    """ return an iterator of (value, metadata, extra)
    param host: the target host
    param identifier: the identifier of the metric
    param cache: cache passed from the pollster
    return value: sample value
    return metadata: dict to construct sample's metadata
    return extra: dict of extra information to help pollster
                  to construct the sample
    """

In ceilometer.hardware.inspector.snmp:

Here we define a mapping dict like the following:

MAPPING = {
  identifier: {
    'matching_type': EXACT or PREFIX,
    'metric_oid': oid of the sample value,
    'metadata': {
       metadata_name1: [oid1, value_converter],
       metadata_name2: [oid2, value_converter],
    },
    'post_op': special func to modify the return data,
  },
}

The identifier parameter passed in to the method inspect_generic() serves as the key to the above mapping.

The inspect_generic() method in snmp inspector would do the following things:

1. map = MAPPING[identifier]
   value = None
   metadata = {}
   extra = {}

2. Collect all the oids in the map, including the metric_oid as
   map['metric_oid'] and oids in metadata map['metadata'].

3. Send out GetRequest or GetBulkRequest(based on the matching_type
   respectively) to get those oid values which are not in the cache,
   and save the response as key-value pairs like oid:value in the cache.

4. Based on the matching_type, construct value and metadata using the
   oid:value key-value pairs in the cache. The metadata is constructed
   as a dictionary like:
   {
     metadata_name1: value_converter(select_data(cache, oid1, index)),
     metadata_name2: value_converter(select_data(cache, oid2, index)),
   }

5. call post_op(host, map, value, metadata, extra)
   The post_op is assumed to be implemented by new metric developer. It
   could be used to add additional special metadata(e.g. ip address), or
   it could be used to add information into extra dict to be returned
   to construct the pollster how to build final sample, e.g.
       extra.update('project_id': xy, 'user_id': zw)

Based on the different matching_type, the way to construct the returned (value, metadata) in the above step 5 is a little bit different:

If matching_type is EXACT, we use oid as the exact key into the cache to find the corresponding value, i.e. cache[oid].

If matching_type is PREFIX, all the keys in the cache starting with oid as the prefix matches, and remaining part in the key will be treat like the index. The index could be used to find corresponding metadata value. For example, suppose we have the following mapping:

MAPPING = {
  'disk.size.total': {
    'matching_type': PREFIX,
    'metric_oid': "1.3.6.1.4.1.2021.9.1.6",
    'metadata': {
       'device': ["1.3.6.1.4.1.2021.9.1.3", str],
       'path': ["1.3.6.1.4.1.2021.9.1.2", str],
    },
    'post_op': None,
  },
And in the cache, we have something like the following:
{
  '1.3.6.1.4.1.2021.9.1.6.1': 19222656,
  '1.3.6.1.4.1.2021.9.1.3.1': "/dev/sda2",
  '1.3.6.1.4.1.2021.9.1.2.1': "/"
  '1.3.6.1.4.1.2021.9.1.6.2': 808112,
  '1.3.6.1.4.1.2021.9.1.3.2': "tmpfs",
  '1.3.6.1.4.1.2021.9.1.2.2': "/run",
}
So here we'll return 2 instances of (value, metadata, extra):
 (19222656, {'device': "/dev/sda2", 'path': "/"}, None)
 (808112, {'device': "tmpfs", 'path': "/run"}, None)

In ceilometer.hardware.pollsters:

We need to change accordingly to the new generic inspector interface.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

The existing model is to use GetNextRequest first to get the index and then use that index to GetRequest mutiple OIDs. We replace it with GetBulkRequest. This could significantly reduce the snmp packet number exchanged between the snmp inspector and snmpd agent. Thus we could improve the performance.

Other deployer impact

In order to make snmp inspector working, a snmp agent ‘snmpd’ needs to be running on the machine, listening on the network interface which is accessible to the snmp inspector and the ceilometer agent. However, this is not a new deploy requirement because we already require a snmpd agent deployed.

Developer impact

The interface change in ceilometer.hardware.inspector.base.Inspector might requires other hardware inspector implementations to be changed too. But the only known implementation now is the snmp inspector, so there should be no other significant impact.

Implementation

Assignee(s)

Primary assignee:: <lianhao-lu>
Other contributors:: <None>
Ongoing maintainer:: <lianhao-lu>

Work Items

adding new inspect_generic interface and the snmp implementation
change the pollster to new interface and remove old inspector interface

Future lifecycle

None

Dependencies

https://review.openstack.org/#/c/92370

Testing

None

Documentation Impact

None

References

Juno summit session log

https://etherpad.openstack.org/p/ceilometer-snmp-inspector

pysnmp library

http://pysnmp.sourceforge.net

Gnocchi – Ceilometer API v3

Tue, 01 Jul 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/gnocchi

Problem description

From the beginning of the Ceilometer project, a large part of the goal was to store time series data that were collected. In the early stages of the project, it wasn’t really clear what and how these time series were going to be handled, manipulated and queried, so the data model used by Ceilometer was very flexible. That ended up being really powerful and handy, but the resulting performance has been terrible, to a point where storing a large amount of metrics on several weeks is really hard to achieve without having the data storage backend collapsing.

Having such a flexible data model and query system is great, but in the end users are doing the same request over and over and the use cases that need to be addressed are a subset of that data model. On the other hand, some queries and use cases are not solved by the current data model, either because they are not easy to be expressed or because they are just too damn slow to run.

Lately, during the Icehouse Design Summit in Hong-Kong, developers and users showed interest in having Ceilometer doing metric data aggregation, in order to keep data in a more long running fashion. No work has been done during the Icehouse cycle on that, probably due to the lack of manpower around the idea, even if the idea and motivation was validated by the core team back then.

Considering the amount of data and metrics Ceilometer generates and has to store, a new strategy and a rethinking of the problem was needed, so Gnocchi is a try on that.

Proposed change

Ceilometer is nowadays trying to achieve two different things:

Store metrics, that is a list of (timestamp, value) for a given entity, this entity being anything from the temperature in your datacenter to the CPU usage of a VM.
Store events, that is a list of things that happens in your OpenStack installation: an API request has been received, a VM has been started, an image has been uploaded, a server fell of the roof, whatever

These two things are both very useful for all the use cases Ceilometer tries to achieve. Metrics are useful for monitoring, performance analysis, and alarming, where events are useful to do audit, billing, debugging, etc.

However, while the event collection of Ceilometer is pretty solid and ok (but still needs to be working on), the metrics part suffers terrible design and performance issues.

Having the so-called free form metadata associated with each metric generated by Ceilometer is the most problematic design we have. It stores a lot of redundant information that it is hard to query in an efficient manner. On the other hand, systems like RRD have existed for a while, storing a large amount of (aggregated) metrics without much problem. The metadata associated to these metrics being another issue.

So that left us with 2 different problem to solve: store metrics and store information (the so-called metadata) about resources.

Alternatives

None

Data model impact

This will bring a whole new data model, where metrics and resources are split with snapshots of resource metadata no longer wedded to individual sample datapoints.

It will be possible to migrate data from the current Ceilometer database to this new system by writing and running a script taking charge of that. A future blueprint should cover this topic.

REST API impact

The change of the data model is consequent, so a new API is also needed. It should be a version 3 of the Ceilometer API, or if the project is kept separate a version 1 of it.

The proposed API is:

POST /v1/entity:

-> {"archives": [{"lifespan": 3600, "points": 1000},
                 {"lifespan": "1 year", "interval": 60},
                 {"points": 1000, "interval": 60}]}
<- 201 Created
   Location: /v1/entity/<uuid>

Create an entity storing:
  - 1000 points over an hour
  - a point every minute over a year
  - 1000 points with a point every minute.
The uuid of the entity is returned.

POST /v1/entity/<uuid>/measures:

-> [{"timestamp": "2013-01-01 12:12:23", "value": 42.0},
    {"timestamp": "2013-01-01 12:12:24", "value": 43.1}]
<- 204 No Content

Store measures for an entity

GET /v1/entity/<uuid>/measures:

<- [{"timestamp": "2013-01-01 12:12:23", "value": 42.0},
    {"timestamp": "2013-01-01 12:12:24", "value": 43.1}]

Returns a list of measures from this entity.
Time span can be specified with a query string.

DELETE /v1/entity/<uuid>:
```
<- 204 No Content

Delete an entity.
```

POST /v1/resource/<resource type>:

-> { "id": <uuid>,
     "started_at": "2013-01-01 12:23:12",
     "project_id": "foobar",
     "entities": { "cpu.util": <entity uuid> },
     "user_id": "foobaz"}
<- { "id": <uuid>,
     "started_at": "2013-01-01 12:23:12",
     "project_id": "foobar",
     "entities": { "cpu.util": <entity uuid> },
     "type": <resource type>,
     "user_id": "foobaz"}

Create a resource. The UUID has to be provided by the caller (and is
expected to match the native UUID of the underlying resource) and
various attributes can also be provided.

Entities can be specified with their UUID, or with creation parameters:

-> { "id": <uuid>,
     "started_at": "2013-01-01 12:23:12",
     "project_id": "foobar",
     "entities": { "cpu.util": {"archives": [{"lifespan": 3600, "points": 1000}]} },
     "user_id": "foobaz"}
<- { "id": <uuid>,
     "started_at": "2013-01-01 12:23:12",
     "project_id": "foobar",
     "entities": { "cpu.util": <entity uuid> },
     "user_id": "foobaz"}

GET /v1/resource/<resource type>:

<- [{ "id": <uuid>,
      "started_at": "2013-01-01 12:23:12",
      "project_id": "foobar",
      "type": "generic",
      "entities": { "cpu.util": <entity uuid> },
      "user_id": "foobaz"}]

Return list of resources.

GET /v1/resource/<resource type>/<uuid>:

<- { "id": <uuid>,
     "started_at": "2013-01-01 12:23:12",
     "project_id": "foobar",
     "type": "generic",
     "entities": { "cpu.util": <entity uuid> },
     "user_id": "foobaz"}

Return details about a resource.

DELETE /v1/resource/<resource type>/<uuid>:
```
<- 204 No Content

Delete a resource.
```

PATCH /v1/resource/<resource type>/<uuid>:

-> {"started_at": "2013-01-01 12:23:13"}
<- { "id": <uuid>,
     "started_at": "2013-01-01 12:23:13",
     "type": "generic",
     "entities": { "cpu.util": <entity uuid> },
     "project_id": "foobar",
     "user_id": "foobaz"}

Change value for a mutable attribute. The list of attributes that is
mutable depends on the resource type, but all resource type can change:
* ended_at
* entities

All resources inherits from the generic resource type and can therefore be partially manipulated by using this resource type. Otherwise, resource types with more attributes are provided such as instance to create more complete resources.

All resources types are builtin within Gnocchi in order to be more performant. If a resource type needs to be indexed but is not known to Ceilometer, one can relies on the generic resource type and manage the attributes of the resource is another system, per user discretion.

The resource type known by Gnocchi will be the resource types provided by OpenStack, e.g. instance, port, network, volume, etc.

Security impact

Usual Keystone token-based authN and RBAC-based authZ.

No security mechanism is proposed to access entities. As the entities UUID are dynamically and randomly allocated, one has to know the UUID of that entity to access it. It can therefore be considered as a secret.

Access to the resources can be filtered based on the user_id and project_id fields that are stored and mandatory attributes. That’s the same mechanism currently used in Ceilometer API v2.

Pipeline impact

The publishing mechanism will need to be adapted to that new model, as resources needs to be created before they can be metered. Another blueprint should cover this topic.

Other end user impact

The ceilometerclient will need to be extended to support both the old and new APIs to that also.

Performance/Scalability Impacts

The scalability and performances of this new system should be drastically better than the old one.

Having real benchmarks of this system would also be interesting.

Other deployer impact

None

Developer impact

It’s likely that the API v2 of Ceilometer should be frozen and that no further improvements should be made at this stage.

Implementation

Assignee(s)

Primary assignee:

jdanjou

Other contributors:

sileht
dbelova

Ongoing maintainer:

jdanjou

Work Items

Build the Gnocchi service and API
Adjust Ceilometer data retrieval and publishing as needed to adapt to the new data storage and API
Make things work together

Future lifecycle

This will be a core component of Ceilometer, so everyone is going to take care of that, me included.

Dependencies

Canonical implementation of the storage driver requires Pandas and Swift.
Alternative statistical/storage driver(s) with different dependencies may also be provided in time.

Testing

Unit tests are provided.

Tempest tests should be added to cover the new API. A variant of the v2 tests running with v3 mechanism enabled is a possibility.

Documentation Impact

We must document the new API. There is currently no mechanism to auto-generate the API documentation, though it should be doable and interesting to do so.

References

OSprofiler notification plugin

Wed, 25 Jun 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/osprofiler-plugin

This spec is about integration between osprofiler and ceilometer. Actually osprofiler requires some kind on notification API and collector. Which is actually Ceilometer.

More about OSprofiler: https://github.com/stackforge/osprofiler

Problem description

In OpenStack we have bunch of services inside every project. It’s really hard to understand where and why your request works slow. To resolve this issue we introduce tiny library, that can be easily integrated in all existing service and will allow us to get 1 trace (tree of elements) per API request. It’s simpler to see, then to explain sample: http://pavlovic.me/rally/profiler/

To resolve this task, we have to send from all services to some common collector special notifications. As there is already Ceilometer, that is integrated with everything and as well notification API that perfectly works for us we decide to use it (Ceilometer).

Proposed change

Add new Ceilometer notification plugin, that will collect notifications related to profiler.

Alternatives

Make another collector of notifications (Ceilometer) and integrate it in all projects ;)

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

It depends on how heavy you are using osprofiler. By default even if everything is turned on, osprofiler won’t sent any notifications. profiler will send notifications only in case if person that knows hmac key sends special trace info in special header.

Amount of data that will be send depends on API request. One of worst API call is boot VM. If we trace all rest/rpc/db calls it will be about 150kb of samples. Note with disabled tracing of DB we can reduce multiple times amount of profiling data.

Warnings:

Do not trace every request, especially such like “nova boot VM”.
Keep hmac in safe, if somebody knows it, he can create trace for every request or in another words DDOS Ceilometer.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Boris Pavlovic <boris@pavlovic.me>

Work Items

Add new notification plugin (already done): https://review.openstack.org/#/c/100239/
Turn it on by default in devstack

Future lifecycle

This plugin is essential for OSprofiler, and OSprofiler is essential for OpenStack. So this code will be fully supported.

Dependencies

There is no dependencies.

Testing

For the begging we are going to have only unit tests. Actually code is quite simple, so it doesn’t require functional testing, as well this code won’t be changed a lot (at all).

The second reason why unit test are enough is that this plugin will be heavy used in rally gates. So if something went wrong we will catch it quite fast.

Documentation Impact

Add warning that profiler creates huge load on Ceilometer backend. It means that it shouldn’t be turned on for every request. As well we should say that load can be drastically reduced in case of turned off tracing of DB request.

References

OSprofiler lib: https://github.com/stackforge/osprofiler
oslo.messaging notification api for osprofiler: http://goo.gl/nCK21D
Cinder integration: http://goo.gl/yY42jd
Result: http://pavlovic.me/rally/profiler/

XenAPI support

Wed, 25 Jun 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/xenapi-support

Currently ceilometer can support libvirt, hyperv and vmware hypervisor. And now it is necessary to add xenapi inspector for XenServer/Xen Cloud Platform.

Problem description

The usecase is to use xenapi to inspect the XenServer hypervisor.

Proposed change

Create xenapi directory in ceilometer/compute/virt, and implement xenapi inspector to support all existing meters.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

The deployer can now optionally define connection information to XenServer/Xen Cloud Platform by adding:

[xenapi]
connection_url =
connection_username =
connection_password =

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

qiaowei-ren

Ongoing maintainer:

qiaowei-ren

Work Items

Implement xenapi inspector for XenServer/Xen Cloud Platform.
Test method in unit tests

Future lifecycle

None

Dependencies

None

Testing

Unit Tests will be added to cover the necessary inspector calls.

Documentation Impact

The Measurement docs need to be updated to reflect xenapi support.

References

XenAPI Documentation: http://docs.vmd.citrix.com/XenServer/6.2.0/1.0/en_gb/api/

PaaS Event Format

Mon, 23 Jun 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/paas-event-format-for-ceilometer

The general mechanism for setting up Ceilometer collection for a new service has been to either create a targeted agent or use the existing service APIs to meter these systems. While this approach has worked for the existing set of services, it will run into a several problems as the number of PaaS and SaaS services that need to be metered expand. This blueprint provides a proposal for streamlining the format of integrations so that Ceilometer can consume metering data from any conforming PaaS application without doing any custom integration.

Problem description

There are a number of PaaS services that are currently in development and a growing number of applications running on top of OpenStack infrastructure. Many of these new applications will need to be metered for a variety of reasons: internal systems management, license management, end customer billing. Ceilometer currently supports the collection of metering information for many infrastructure-level services, however it won’t be viable to do custom integrations for the potentially hundreds (and eventually thousands) of services that may be hosted in OpenStack. We want to head off that issue by defining a universal PaaS notification payload so Ceilometer can consume PaaS notifications without significant integration work.

A few use cases come to mind: Database as a Service

Database as a Service (DBaaS), i.e. Trove, has an architecture where a service controller manages special Nova compute instances on behalf of the customer. From their perspective they are running instances of a database, not compute instances. Metering and billing will be based on hours that the database has been running, not necessarily how long the hosting instance has been run. This requires a unique set of metering records to be generated and stored to enable usage tracking and billing of the individual database instances.

DNS as a Service

DNS as a service runs on a similar service controller architecture to DBaaS. In this case, the meters that need to be measured are the existence of a DNS zone and the number of DNS queries that have been served. Once again these are application level meters that do not correlate directly to the host instances that are running. Queries could be served by a variety of hosts and a DNS zone’s existence does not depend on a specific compute instance. Application level metrics must be tracked by the DNS service and reported out so that these systems can be tracked.

Load Balancing as a Service

A load balancer is a logical system that consists of some number of independent backends that host the load balancing software. Meters that must be measured would be things like the existence of a load balancer VIP, its members, and the amount of data that is being transferred through the system. Once again, this does not directly correlate to the underlying infrastructure but must be reported at the application level

Proposed change

We need to standardize the content of the PaaS payload to include a set of data that will include data needed for metering and billing. This essentially constitutes a documentation change and a set of tests to verify key fields.

The following is the proposed schema:

 [
  {
      "Field": "event_type",
      "Type": "enumeration",
      "Description": "for event type records, this describes the actual event that occurred",
      "Compliance": "required for events",
      "Notes": "depends on service, defaults to create, exists, delete"
  },
  {
      "Field": "timestamp",
      "Type": "UTC DateTime",
      "Description": "timestamp of when this event was generated at the resource",
      "Compliance": "required",
      "Notes": "ISO 8859 date YYYY-mm-ddThh:mm:ss"
  },
  {
      "Field": "message_id",
      "Type": "String",
      "Description": "unique identifier for event",
      "Compliance": "required",
      "Notes": ""
  },
  {
      "payload": [
      {
          "Field": "version",
          "Type": "String",
          "Description": "Version of event format",
          "Compliance": "required",
          "Notes": ""
      },
      {
          "Field": "audit_period_beginning",
          "Type": "UTC DateTime",
          "Description": "Represents start time for metrics reported",
          "Compliance": "required",
          "Notes": "Format ISO 8859 date YYYY-mm-ddThh:mm:ss"
      },
      {
          "Field": "audit_period_ending",
          "Type": "UTC DateTime",
          "Description": "Represents end time for metrics reported",
          "Compliance": "required",
          "Notes": "Format ISO 8859 date YYYY-mm-ddThh:mm:ss"
      },
      {
          "Field": "record_type",
          "Type": "enumeration ",
          "Values": {
              "event": "events describe some kind of state change in the service",
              "quantity": "quantity describes a usage metric value"
          },
          "Compliance": "required",
          "Notes": ""
      },
      {
          "Field": "project_id",
          "Type": "UUID",
          "Description": "Keystone project_id identifies the owner of
                          the service instance",
          "Compliance": "required",
          "Notes": ""
      },
      {
          "Field": "user_id",
          "Type": "UUID",
          "Description": "Keystone user_id identifies specific user",
          "Compliance": "optional",
          "Notes": ""
      },
      {
          "Field": "service_id",
          "Type": "UUID",
          "Description": "Keystone service_id uniquely identifies a service",
          "Compliance": "required",
          "Notes": ""
      },
      {
          "Field": "service_type",
          "Type": "String",
          "Description": "Keystone service_type uniquely identifies a service",
          "Compliance": "required",
          "Notes": ""
      },
      {
          "Field": "instance_id",
          "Type": "UUID",
          "Description": "uniquely identifies an instance of the service",
          "Compliance": "required",
          "Notes": "assuming instance level reporting"
      },
      {
          "Field": "display_name",
          "Type": "String",
          "Description": "text description of service",
          "Compliance": "optional",
          "Notes": "used if customer names instances"
      },
      {
          "Field": "instance_type_id",
          "Type": "enumeration",
          "Description": "used to describe variations of a service",
          "Compliance": "required",
          "Notes": "needed if variations of service have different prices or
                    need to be broken out separately"
      },
      {
          "Field": "instance_type",
          "Type": "String",
          "Description": "text description of service variations",
          "Compliance": "optional",
          "Notes": ""
      },
      {
          "Field": "availability_zone",
          "Type": "String",
          "Description": "where the service is deployed",
          "Compliance": "optional",
          "Notes": "required if service is deployed at an AZ level"
      },
      {
          "Field": "region",
          "Type": "String",
          "Description": "data center that the service is deployed in",
          "Compliance": "optional",
          "Notes": "required if service is billed at a regional level"
      },
      {
          "Field": "state",
          "Type": "enumeration",
          "Description": "status of the service at the time of record generation",
          "Compliance": "optional",
          "Notes": "required for existence events"
      },
      {
          "Field": "state_description",
          "Type": "String",
          "Description": "text description of state of service",
          "Compliance": "",
          "Notes": ""
      },
      {
          "Field": "license_code",
          "Type": "enumeration",
          "Description": "value that describes a specific license model",
          "Compliance": "optional",
          "Notes": "this field is TBD depending on dev_pay design work"
      },
          {
              "metrics": [
                  {
                      "Field": "metric_name",
                      "Type": "String",
                      "Description": "unique name for the metric that is represented
                       in this record",
                      "Compliance": "required",
                      "Notes": ""
                  },
                  {
                      "Field": "metric_type",
                      "Type": "enumeration",
                      "Description": "gauge, cumulative, delta",
                      "Compliance": "required",
                      "Notes": "describes the behavior of the metric, from ceilometer"
                  },
                  {
                      "Field": "metric_value",
                      "Type": "Float",
                      "Description": "value of metric for quantity type records",
                      "Compliance": "required for quantities",
                      "Notes": ""
                  },
                  {
                      "Field": "metric_units",
                      "Type": "enumeration",
                      "Description": "describes the units for the quantity",
                      "Compliance": "optional",
                      "Notes": ""
                  }
              ]
          }
      ]
  }
]

Note: Required means that it must be present and described as in the specification. Optional means it can be present or not, but if present it must be described as in the specifications.

Alternatives

The alternative approach to this standardization is to allow PaaS service providers to determine the content of the notifications and the associated plugins, requiring missing data to be addressed post implementation through patching.

Data model impact

What new data objects and/or database schema changes is this going to require?

This format should fit within the existing schema.

What database migrations will accompany this change, treating the SQLAlchemy and NoSQL cases separately.

No DB migrations will be required.

Will this add to the on-the-fly data massaging cruft that we’ve accreted over time?

The purpose of this change is to avoid the explosion of cruft that could potentially be generated as PaaS services implement variations on the notification payload.

REST API impact

The API should transparently handle the PaaS data, there should be no API impact.

Security impact

As this BP only specifies a standard format for PaaS samples (and one that closely resembles the current sample content) current content of Ceilometer samples) there is no security impact.

Pipeline impact

The implementation of the PaaS collection agent is documented in a separate BP; this standardization requires no pipeline changes in itself.

Other end user impact

None.

Performance/Scalability Impacts

None.

Other deployer impact

There should be a positive impact to deployers from implementing this standard: unifying the PaaS content up-front will reduce the likelihood of missing data required for downstream processing.

Developer impact

There is the increase in up-front effort to define, for each PaaS service being implemented, a set of data that fulfills the standard. There is also an implication that there may be evolution of this standard, and therefore evolution of the documentation/tests.

Implementation

Assignee(s)

This is tied to the work documented in the PaaS Event Collection blueprint and will be implemented in parallel with that. That said, there’s certainly benefit in opening this up to those who have an interest in PaaS events, especially if there are data elements that we haven’t considered here.

Primary assignee:: nealph (phil.neal@hp.com)
Other contributors:: fabiog (fabio.gianetti@hp.com)
Ongoing maintainer:: nealph (phil.neal@hp.com)

It’s likely that the standard will expand: the intent here is to define and clearly document the core elements required. For that reason, I expect contributions to this documentation will be done by others beside myself.

Work Items

1. Identify impacts to existing services (if any) and submit appropriate requests (via Launchpad)

Create documentation changes

Future lifecycle

I expect there will be pressure to expand the specification in future release cycles.

Dependencies

None

Testing

There is some implication for testing, since we want to check for existence of the fields defined in the specification.

Documentation Impact

Documentation for this change is the core deliverable. While it won’t require a lot of documentation, the readability and descriptiveness of the documentation is critical. We expect to leverage much of the content at https://wiki.openstack.org/wiki/Ceilometer/blueprints/StandardPaaSEventFormat as well as add updated event payload examples.

References

The full blueprint for this change is available at: https://wiki.openstack.org/wiki/Ceilometer/blueprints/StandardPaaSEventFormat

The related blueprint outlining changes to the PaaS event collection mechanism (that benefits from on this standard being in place) is available at: https://wiki.openstack.org/wiki/Ceilometer/blueprints/PaaSEventUsageCollection

Enable event feature on MongoDB

Fri, 13 Jun 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/mongodb-events-feature

Events feature was implemented on SQLAlchemy and HBase backends. And now it is necessary to extend it on MongoDB and DB2.

Problem description

The usecase is to store events in database (MongoDB or DB2) and to get events (event’s types, traits, trait’s types) from this database.

The main idea is to implement event feature in way like it was done on other drivers (SQLAlchemy and HBase).

Usecases:

to get all events that were created in a specified period of time
to get events by event’s type
to get events by event’s message id
to get events by trait’s description, type, and value
to get all event’s types that were stored
to get all trait types belonging to specific event type
to get all traits that could be found in specific event’s type, and optionally with specific trait type.

Proposed change

Add new methods in ceilometer.storage.pymongo_base.py for storing and getting events from database. Basing on events implementation that have been already done we should create next methods:

record_event – method for storing collected events into MongoDB and DB2
get_events – method for getting events from MongoDB and DB2 with specified filters: - period of event’s generation time - event’s type - event’s message id - trait’s description, type, and value (here should be taken into account possible operations on values, for ex: <lt, le, eq, ne, ge, gt>)
get_event_types – method for getting all event’s types that were stored in database.
get_trait_types – method returns all trait types for all stored event types or only for one event’s type if it is specified.
get_traits – method returns all traits for certain event type, or traits for specified trait’s type if type is defined.

Alternatives

None

Data model impact

We will need to add new collection into MongoDB and DB2 for events

Collection:

events
- _id: uuid of event (Event.message_id)
- event_type: description of event’s type
- timestamp: time stamp of event generation
- traits: [array of {trait_name: string, trait_type: integer,
  trait_value: data}]

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

idegtiarov

Ongoing maintainer:

idegtiarov

Work Items

Create new method for event feature implementation on MongoDB and DB2
Test method in unit tests

Future lifecycle

None

Dependencies

None

Testing

This code will be tested in unit tests for event feature

Documentation Impact

None

References

None

Metering Network Services - LoadBalancer as a Service

Thu, 05 Jun 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-meter-lbaas

Problem description

Ceilometer currently has no support for metering Network Services. Cloud providers/Operators have the need to monitor and meter various aspects of the Network Services. This spec deals with metering Load Balancer as a service(LBaaS).

Proposed change

The measurements needed for metering LBaas are categorized into two, provider and service level. Following are the measurements targeted to be included in Ceilometer:

Provider level metrics:
- Type of LB ( HAproxy, VPx etc..)
- Features (SSL, Non-SSL, stickiness etc..)
Service level metrics:
- LB pool
  
  Status
  
  Number of Connections - Use statistics API call from LBaaS side
  
  Bandwidth - Use statistics API call from LBaaS side
- Members
  
  Status
  
  Bandwidth - Use statistics API call from LBaaS side
- Health monitors
  
  Status of the health probe
Metric Definitions:

Name

Type

Unit

Origin

network.services.lb.type

g

type

p

network.services.lb.pool

g

pool

p

network.services.lb.vip

g

vip

p

network.services.lb.member

g

member

p

network.services.lb.health_monitor

g

monitor

p

network.services.lb.total.connections

g

connection

p

network.services.lb.active.connections

g

connection

p

network.services.lb.incoming.bytes

c

B

p

network.services.lb.outgoing.bytes

c

B

p

g = gauge, c = cumulative, p = pollster

Name	Type	Unit	Origin
network.services.lb.type	g	type	p
network.services.lb.pool	g	pool	p
network.services.lb.vip	g	vip	p
network.services.lb.member	g	member	p
network.services.lb.health_monitor	g	monitor	p
network.services.lb.total.connections	g	connection	p
network.services.lb.active.connections	g	connection	p
network.services.lb.incoming.bytes	c	B	p
network.services.lb.outgoing.bytes	c	B	p

Status is captured in an enum-style value in the sample volume, as opposed to the resource metadata, for each pool, vip, members and monitor.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

The end user should be able to interact via the existing API and CLI.

It would be good to expose these metrics on horizon dashboard, but that is outside the scope of this spec.

Performance/Scalability Impacts

This change should not have any major impact on performance/Scalability.

Other deployer impact

None

Developer impact

This feature should have minimal impact on developers for ongoing maintenance

Implementation

Assignee(s)

Primary assignee:

pkilambi

Other contributors:

None

Ongoing maintainer:

pkilambi

Work Items

Add neutron client APIs to query LBaas calls in neutron_client.py
Add new Pollsters and notification handlers
Add Unit/Integration test coverage
Update measurement docs

Future lifecycle

New measurements around LBaaS and other network services will be part of the network pollsters and notifications. So ongoing maintenance will be handled by the Ceilometer team, myself included.

Dependencies

None

Testing

Unit and integration Tests will be added to cover the necessary neutron_client calls, pollsters and notifications.

Documentation Impact

The Measurement docs need to be updated to reflect the new meters captured from LBaaS API and notifications.

References

https://etherpad.openstack.org/p/juno-summit-metering-network-services

Enable event feature on HBase

Fri, 30 May 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/hbase-events-feature

Currently events feature is implemented only on SQLAlchemy backend, so choosing another database events are not supported.

Problem description

The usecase is to store events in HBase and to get events (event’s types, traits, trait’s types) from this database.

The main idea is to implement event feature in way like it was done on SQLAlchemy driver.

Usecases:

to get all events that were created in a specified period of time
to get events by event’s type
to get events by event’s message id
to get events by trait’s description, type, and value
to get all event’s types that were stored
to get all trait types belonging to specific event type
to get all traits that could be found in specific event’s type, and optionally with specific trait type.

Proposed change

Add new methods in ceilometer.storage.impl_hbase.py for storing and getting events from database. Basing on events implementation for SQL database we should create next methods:

record_event – method for storing collected events into HBase
get_events – method for getting events from HBase with specified filters: - period of event’s generation time - event’s type - event’s message id - trait’s description, type, and value (here should be taken into account possible operations on values, for ex: <lt, le, eq, ne, ge, gt>)
get_event_types – method for getting all event’s types that were stored in database.
get_trait_types – method returns all trait types for all stored event types or only for one event’s type if it is specified.
get_traits – method returns all traits for certain event type, or traits for specified trait’s type if type is defined.

Alternatives

None

Data model impact

We will need to add new collection into HBase for events

Collection:

events
- row_key: timestamp of event’s generation + uuid of event
  in format: “%s+%s” % (ts, Event.message_id) If we store events with such row_key they are sorted by timestamp in the database.
-Column Families:

f: contains the following qualifiers:
-event_type: description of event’s type -timestamp: time stamp of event generation -all traits for this event in format “%s+%s” % (trait_name, trait_type)

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

idegtiarov

Ongoing maintainer:

idegtiarov

Work Items

Create new method for event feature implementation on HBase
Test method in unit tests

Future lifecycle

None

Dependencies

None

Testing

This code will be tested in unit tests for event feature

Documentation Impact

None

References

None

Dedicated database for the alarm definition and history of Ceilometer

Thu, 29 May 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/dedicated-alarm-database

Currently if we use MongoDB for storing metering data, we are forced to use MongoDB for storing events and alarms. Because the API service can use only one database connection object.

Also metering and alarm are two completely different things, but the db code are currently in the same place.

The blueprint proposes to allow to have a different database for alarms definition and alarms history.

Problem description

The usecase is I want to store my metering data into MongoDB and my alarm definition and history into MySQL

Proposed change

The idea is to move all alarms db stuffs from ceilometer/storage to ceilometer/alarm/storage

ceilometer.storage.get_connection_from_conf get a new argument ‘purpose’, that can be set to metering or alarm.

This will allow to use a different namespace to load a driver that depends of the purpose.

New entry points will be added to have a different set of driver for alarm and metering.

Ceilometer API will use two connection objects, one for metering and one for alarm depending on the used API endpoint requested.

Alternatives

None

Data model impact

None

In case of SQLAlchemy, migration scripts will continue to be stored in a common place to ensure easy migration. And if a dedicated database is used, we accepted having unused and empty tables.

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

The deployer can now optionally define a dedicated database for the alarming of ceilometer by adding:

[database]
connection = mongodb://localhost/ceilometer
alarm_connection = mysql://localhost/ceilometer

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: sileht
Ongoing maintainer:: sileht

Work Items

Splits the alarm db API from metering db API
Remove unuseful drivers for alarm API

Future lifecycle

None

Dependencies

None

Testing

The is code refactoring with a new optional configuration option. This configuration will be tested in new unit tests.

Documentation Impact

None

References

None

Metering Network Services - Firewall as a Service

Tue, 27 May 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-meter-fwaas

Problem description

Ceilometer currently has no support for metering Firewall as a Service. Cloud providers/Operators have the need to monitor and meter various aspects of the Network Services. This spec deals with metering Firewall as a Service(FWaaS).

Proposed change

The measurements needed for metering FWaas are categorized into two, provider and service level. Following are the measurements targeted to be included in Ceilometer:

Provider level metrics:
- Type of Firewall (iptables, CSR virtual firewall etc..) - depends on flavor framework in neutron(See Dependencies)
Service level metrics:
- Firewall Rule/Policy existence
- Number of Connections - Needs changes to Neutron FWaaS(See Dependencies)
- Bandwidth - Needs changes to neutron FWaaS(See Dependencies)
Metric Definitions:

Name	Type	Unit	Origin
network.services.fw	g	firewall	p
network.services.fw.policy	g	policy	p
network.services.fw.type	g	firewall	p
network.services.fw.total.connections	g	connections	p
network.services.fw.active.connections	g	connections	p
network.services.fw.incoming.bytes	c	B	p
network.services.fw.outgoing.bytes	c	B	p

g = gauge, c = cumulative, p = pollster

Status is captured in an enum-style value in the sample volume, as opposed to the resource metadata, for each firewall.

The resources associated with these metrics are captured as part of resource discovery. Neutron exposes apis to capture this data which are invoke via pollsters from the ceilometer side. The notifications on neutron services side are a bit slim. As we add these notification messages to the neutron side, we will enhance the ceilometer side to capture these events through notification handlers.

For reference implemenation on neutron side, there will be an api call to retrieve stats such as connections and bandwidth.The backend implementation will be based on iptables. Iptables provides us a way to gather average hit counts to provide these stats. Other vendor based implementation can do the same.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

New sources need to be included in pipeline.yaml for each group of pollsters that share a discovery extension. An example below:

sources:
  - name: fw_source
    interval: 600
    meters:
      - "network.services.fw"
    discovery:
      - "firewall"
    sinks:
      - network_services_sink

Similarly, we will have sources for firewall policy, hit counts and bandwidth.

Other end user impact

The end user should be able to interact via the existing API and CLI.

It would be good to expose these metrics on horizon dashboard, but that is outside the scope of this spec.

Performance/Scalability Impacts

This change should not have any major impact on performance/Scalability.

Other deployer impact

None

Developer impact

This feature should have minimal impact on developers for ongoing maintenance

Implementation

Assignee(s)

Primary assignee:

pkilambi

Other contributors:

None

Ongoing maintainer:

pkilambi

Work Items

Add neutron client APIs to query FWaaS calls in neutron_client.py
Add new Pollsters and notification handlers
Add Unit/Integration test coverage
Update measurement docs

Future lifecycle

New measurements around FWaaS and other network services will be part of the network pollsters and notifications. So ongoing maintenance will be handled by the Ceilometer team, myself included.

Dependencies

Flavor Framework in Neutron to determine the type of firewall. - https://blueprints.launchpad.net/neutron/+spec/neutron-flavor-framework
Need statistics calls for hit counts on neutron FWaaS side to support connections and bandwidth.

Testing

Unit and integration Tests will be added to cover the necessary neutron_client calls, pollsters and notifications.

Documentation Impact

The Measurement docs need to be updated to reflect the new meters captured from FWaaS API and notifications.

References

https://etherpad.openstack.org/p/juno-summit-metering-network-services

Metering Network Services - VPN as a Service

Tue, 27 May 2014 00:00:00

https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-meter-vpnaas

Problem description

Ceilometer currently has no support for metering VPN as a Service. Cloud providers/Operators have the need to monitor and meter various aspects of the Network Services. This spec deals with metering VPN as a service(VPNaaS).

Proposed change

The measurements needed for metering VPNaas are categorized into two, provider and service level. Following are the measurements targeted to be included in Ceilometer:

Provider level metrics:
- Type of VPN (Openswan, Cisco etc..) - depends on flavor framework in neutron(See Dependencies)
Service level metrics:
- VPN service Status
- Number of Connections
- Bandwidth - Needs changes to neutron VPNaaS(See Dependencies)
Metric Definitions:

Name	Type	Unit	Origin
network.services.vpn	g	vpn	p
network.services.vpn.type	g	vpn	p
network.services.vpn.connections	g	connections/s	p
network.services.vpn.incoming.bytes	c	B	p
network.services.vpn.outgoing.bytes	c	B	p

g = gauge, c = cumulative, p = pollster

Status is captured in an enum-style value in the sample volume, as opposed to the resource metadata, for each vpn.

For reference implemenation on neutron side, there will be an api call to retrieve stats such as connections and bandwidth.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

New sources need to be included in pipeline.yaml for each group of pollsters that share a discovery extension. An example below:

sources:
  - name: fw_source
    interval: 600
    meters:
      - "network.services.vpn"
    discovery:
      - "vpn"
    sinks:
      - network_services_sink

Similarly, we will have sources for connections and bandwidth.

Other end user impact

The end user should be able to interact via the existing API and CLI.

It would be good to expose these metrics on horizon dashboard, but that is outside the scope of this spec.

Performance/Scalability Impacts

This change should not have any major impact on performance/Scalability.

Other deployer impact

None

Developer impact

This feature should have minimal impact on developers for ongoing maintenance

Implementation

Assignee(s)

Primary assignee:

pkilambi

Other contributors:

None

Ongoing maintainer:

pkilambi

Work Items

Add neutron client APIs to query VPNaaS calls in neutron_client.py
Add new Pollsters and notification handlers
Add Unit/Integration test coverage
Update measurement docs

Future lifecycle

New measurements around VPNaaS and other network services will be part of the network pollsters and notifications. So ongoing maintenance will be handled by the Ceilometer team, myself included.

Dependencies

Flavor Framework in Neutron to determine the type of firewall. - https://blueprints.launchpad.net/neutron/+spec/neutron-flavor-framework
Need statistics calls for hit counts on neutron VPNaaS side to support connections and bandwidth.

Testing

Unit and integration Tests will be added to cover the necessary neutron_client calls, pollsters and notifications.

Documentation Impact

The Measurement docs need to be updated to reflect the new meters captured from VPNaaS API and notifications.

References

https://etherpad.openstack.org/p/juno-summit-metering-network-services

Ceilometer Python Client Keystone V3 Upgrade

Fri, 23 May 2014 00:00:00

Keystone declared V2 deprecated in K cycle with the intent of maintaining V2 for another two cycles after that. All the services clients are going to add support to V3 in order to smooth this transition.

Problem description

Keystone V3 API has introduced the following major changes:

Authentication: Custom auth method can be developed as plug-ins, this enables support for: OAuth 1.0 and SAML based federation
Authorization: in V2 API there are only two levels “admin” or none. V3 API enables control of who can call each method if the policy file is correctly specified.
Richer set of APIs. Vendor specific extensions are not supported directly anymore and there is a set of optional extensions supported by Keystone.

Proposed change

Since Ceilometer Client imports keystoneclient, the main changes to client.py are to selectively import the correct keystoneclient library version and pass a number of new options. In shell.py the new options are added to the parser.

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

There will be new commands to be used with the ceilometer client to scope the request to Domain/Project and support different auth methods. Alarms are using the ceilometer client and there could be a potential impact.

Performance/Scalability Impacts

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: fabgia
Other contributors:: robsparker
Ongoing maintainer:: fabgia

Work Items

Add new parameters to the client and cli.
Call the V3 Keystone client.
Validate the authorization and authentication via python-keystoneclient.

Future lifecycle

None.

Dependencies

Keystone client: python-keystoneclient.

Testing

Unit tests will be added to the Ceilometer client to support the new parameter submission as well as validation of the authorization and authentication with Keystone client.

Documentation Impact

Documentation changes specific to new parameters added to the client.