Monasca Specs

Yoga Project Priorities

Mon, 14 Mar 2022 00:00:00

List of priorities the Monasca drivers team is prioritizing in Yoga.

The owners listed are responsible for tracking the status of that work and helping get that work done. They are not the only contributors to this work, and not necessarily doing most of the coding!

The implementation progress on these priorities and other identified important tasks is tracked in this board.

Essential Priorities

Title	Owners
Update Docker Images	chaconpiza
Update https://github.com/monasca/monasca-docker	chaconpiza
Bring compatibility with Python3 zed unit tests	chaconpiza

High Priorities

Title	Owners
Thresholding engine in cluster mode	chaconpiza
Add Time and Times in Monasca UI
Define Prometheus based architecture
Application Credentials
Middleware upgrades ELK 7.3.0 -> OpenDistro

Optional Priorities

Title	Owners

Selenium Tests for Monasca-UI
OpenStack Client Integration

Details

Application Credentials

Keystone appliction credentials offer the mechanism to allow applications to authenticate to Keystone. The ability to specify access rules for application credentials has been implemented in the Train cycle.

The goal of this story is to add application credentials support in monasca-agent. This will prevent the security risk of revealing OpenStack user’s password when installing the agent on the tenants environment. The access rules of these application credentials should be limited to posting measurements. monasca-setup should be extended to automatically generate such credentials and save them in configuration file if needed.

Similar task should be implemented in monasca-grafana-datasource.

Stories:

Xena Project Priorities

Wed, 08 Sep 2021 00:00:00

List of priorities the Monasca drivers team is prioritizing in Xena.

The owners listed are responsible for tracking the status of that work and helping get that work done. They are not the only contributors to this work, and not necessarily doing most of the coding!

The implementation progress on these priorities and other identified important tasks is tracked in this board.

Essential Priorities

Title	Owners
Migrate CI/CD from Travis-CI to Github actions	chaconpiza
Update Docker Images	chaconpiza
Update https://github.com/monasca/monasca-docker	chaconpiza

High Priorities

Title	Owners
Thresholding engine in cluster mode	chaconpiza
Add Time and Times in Monasca UI
Define Prometheus based architecture
Application Credentials
Middleware upgrades ELK 7.3.0 -> OpenDistro

Optional Priorities

Title	Owners

Selenium Tests for Monasca-UI
OpenStack Client Integration

Details

Application Credentials

Similar task should be implemented in monasca-grafana-datasource.

Stories:

Victoria Project Priorities

Tue, 30 Jun 2020 00:00:00

List of priorities the Monasca drivers team is prioritizing in Victoria.

The owners listed are responsible for tracking the status of that work and helping get that work done. They are not the only contributors to this work, and not necessarily doing most of the coding!

The implementation progress on these priorities and other identified important tasks is tracked in this board.

Essential Priorities

Title	Owners
Middleware upgrades Grafana 4.0 -> 7.0.1	dougszu
Kafka/InfluxDB Sharding	witek
Merge events-api into monasca-api	adriancz

High Priorities

Title	Owners
New thresholding engine	chaconpiza
Application Credentials
Define Prometheus based architecture
Selenium Tests for Monasca-UI
Middleware upgrades ELK 7.3.0 -> 7.7.0	dougszu
Middleware upgrades Apache Kafka 2.0.1 -> 2.5.0
Middleware upgrades InfluxDB 1.7.6 -> 1.8.0

Optional Priorities

Title	Owners
Extend Monasca API
Add Time and Times in Monasca UI
Monasca agents RPM Packaging
OpenStack Client Integration
Refresh Monasca transform engine	dougsz

Details

New thresholding engine

Faust library has been evaluated and the prototype of the thresholding engine based on this library has been implemented. The goal of this effort is to implement the new thresholding engine for Monasca to replace Apache Storm Java application.

Kafka/InfluxDB Sharding

Story: https://storyboard.openstack.org/#!/story/2005620

Application Credentials

Similar task should be implemented in monasca-grafana-datasource.

Stories:

Middleware upgrade

Story: https://storyboard.openstack.org/#!/story/2006768

Ussuri Project Priorities

Thu, 24 Oct 2019 00:00:00

List of priorities the Monasca drivers team is prioritizing in Ussuri.

The owners listed are responsible for tracking the status of that work and helping get that work done. They are not the only contributors to this work, and not necessarily doing most of the coding!

The implementation progress on these priorities and other identified important tasks is tracked in this board.

Essential Priorities

Title	Owners
New thresholding engine	chaconpiza
Monasca Events Agent	witek
IPv6 support	witek

High Priorities

Title	Owners
InfluxDB HA Setup	dougsz
Query Logs API	dougsz
Application Credentials	dougsz
New InfluxDB query capabilities	dougsz
Middleware upgrade	witek

Optional Priorities

Title	Owners
Sharding model for InfluxDB
Refresh Monasca transform engine	joadavis

Details

New thresholding engine

Monasca Events Agent

The goal is to implement Monasca Events Listener which will publish Openstack notifications and events from third party applications to Monasca Events API. Specification listing existing requirements and proposed implementation has been written up in the past.

IPv6 Support

It is the community wide goal to support IPv6-Only Deployments.

InfluxDB HA Setup

Story: https://storyboard.openstack.org/#!/story/2005620

Query Logs API

Add support for querying ElasticSearch via the Monasca Log API to support tenant scoped access to logs.

Application Credentials

Similar task should be implemented in monasca-grafana-datasource.

Stories:

New InfluxDB Query Capabilities

The goal is to extend the Monasca API to query measurements using aggregation functions available in InfluxDB, like e.g. DERIVATIVE(). Another goal is to investigate the new Flux QL to allow basic arithmetic operations between different measurements, e.g. (disk_used / disk_total).

Middleware upgrade

Story: https://storyboard.openstack.org/#!/story/2006768

Train Project Priorities

Tue, 21 May 2019 00:00:00

List of priorities the Monasca drivers team is prioritizing in Train.

The owners listed are responsible for tracking the status of that work and helping get that work done. They are not the only contributors to this work, and not necessarily doing most of the coding!

The implementation progress on these priorities and other identified important tasks is tracked in this board.

Essential Priorities

Title	Owners
Kafka client upgrade	witek
Merge Monasca APIs	adriancz
Middleware upgrade	dougsz
Thresholding engine replacement (tech prev.)
PDF generation for documentation

High Priorities

Title	Owners
Application credentials (Grafana)	dougsz
Application credentials (agent)
Documentation refresh	joadavis
Java Persister deprecation	joadavis

Optional Priorities

Title	Owners
Monasca Events Agent
New query language
OpenStack CLI	sc
Reuse Prometheus dashboards
Vitrage integration	chaconpiza

Backlog

Title	Owners
Sharding model for InfluxDB	dougsz
OpenStack Helm
OpenStack Ansible	sc
Senlin integration
Gnocchi support

Details

Kafka client upgrade

Currently, in all Python Monasca components, the copy of kafka-python library in version 0.9.5 (released on Feb 16, 2016) is used. Sticking with the old frozen client version is also unacceptable in terms of security. The goal is to upgrade the Apache Kafka client to confluent-kafka-python. This will dramatically improve the performance and reliability.

Story: https://storyboard.openstack.org/#!/story/2003705

Merge Monasca APIs

The goal is to merge all Monasca APIs into a single unified API to reduce maintenance overhead, make it easier for developers to add new features and improve the user experience.

Story: https://storyboard.openstack.org/#!/story/2003881

Middleware upgrade

We want to change the general approach and try to use the newest (stable) versions of software available. The beginning of the cycle is the good time point to upgrade components such as e.g.: Apache Kafka, InfluxDB, Apache Storm, ELK.

Story: https://storyboard.openstack.org/#!/story/2005624

Thresholding engine replacement (tech prev.)

The goal of this task is to provide the technical preview of the new component replacing the current thresholding engine.

Story: https://storyboard.openstack.org/#!/story/2005598

PDF generation for documentation

This is the community wide goal.

https://governance.openstack.org/tc/goals/train/pdf-doc-generation.html

Application credentials (Grafana)

Keystone appliction credentials offer the mechanism to allow applications to authenticate to Keystone. The ability to specify access rules for application credentials is being developed and will be released in the Train cycle.

The goal of this story is to add application credentials support in monasca-grafana-datasource. The access rules should be limited to only reading the measurements from Monasca. It will allow storing these credentials directly in the datasource without the security risk of revealing the OpenStack user’s password. It will also decouple the datasource from Grafana’s authentication.

Story: https://storyboard.openstack.org/#!/story/2005623

Application credentials (agent)

Documentation refresh

Story: https://storyboard.openstack.org/#!/story/2005625

Java Persister deprecation

Story: https://storyboard.openstack.org/#!/story/2005628

Monasca Events Agent

The goal is to extend Monasca Ceilometer project and add a new events publisher which will publish Openstack notifications (or events) to Monasca Events API.

Story: https://storyboard.openstack.org/#!/story/2003023

Example Spec - The title of your feature request

Tue, 23 Apr 2019 00:00:00

Include the URL of your story:

https://storyboard.openstack.org

Introduction paragraph – why are we doing anything? A single paragraph of prose that operators can understand. The title and this first paragraph should be used as the subject line and body of the commit message respectively.

Some notes about the monasca-spec and stories process:

Not all stories need a spec. For more information see https://docs.openstack.org/monasca-api/latest/contributor/index.html
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Monasca REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/monasca-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this feature request addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

At this point, if you would like to just get feedback on if the problem and proposed change fit in monasca, you can stop here and post this for review to get preliminary feedback. If so please say: Posting to get preliminary feedback on the scope of this spec.

Alternatives

What other ways could we do this thing? Why aren’t we using those? This doesn’t have to be a full literature review, but it should demonstrate that thought has been put into why the proposed solution is an appropriate one.

Data model impact

Changes which require modifications to the data model often have a wider impact on the system. The community often has strong opinions on how the data model should be evolved, from both a functional and performance perspective. It is therefore important to capture and gain agreement as early as possible on any proposed changes to the data model.

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Note that the schema should be defined as restrictively as possible. Parameters which are required should be marked as such and only under exceptional circumstances should additional parameters which are not defined in the schema be permitted.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

For more detailed guidance, please see the OpenStack Security Guidelines as a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These guidelines are a work in progress and are designed to help you identify security best practices. For further information, feel free to reach out to the OpenStack Security Group at openstack-security@lists.openstack.org.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-monascaclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure Monasca that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on Monasca.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a feature where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

Dependencies

Include specific references to specs and/or blueprints in monasca, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Monasca (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Please discuss the important scenarios needed to test here, as well as specific edge cases we should be ensuring work correctly. For each scenario please specify if this requires specialized hardware, a full openstack environment, or can be simulated inside the Monasca tree.

Please discuss how the change will be tested. We especially want to know what tempest tests will be added. It is assumed that unit test coverage will be added so that doesn’t need to be mentioned explicitly, but discussion of why you think unit tests are sufficient and we don’t need to add more tempest tests would need to be included.

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

Which audiences are affected most by this change, and which documentation titles on docs.openstack.org should be updated because of this change? Don’t repeat details discussed above, but reference them here in the context of documentation for multiple audiences. For example, the Operations Guide targets cloud operators, and the End User Guide would need to be updated if the change offers a new feature available through the CLI or dashboard. If a config option changes or is deprecated, note here that the documentation needs to be updated to reflect this specification’s change.

References

Please add any useful references here. You are not required to have any reference. Moreover, this specification should still make sense when your references are unavailable. Examples of what you could include are:

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Queens	Introduced

Example Spec - The title of your feature request

Tue, 23 Apr 2019 00:00:00

Include the URL of your story:

https://storyboard.openstack.org

Some notes about the monasca-spec and stories process:

Not all stories need a spec. For more information see https://docs.openstack.org/monasca-api/latest/contributor/index.html
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Monasca REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/monasca-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this feature request addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-monascaclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure Monasca that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on Monasca.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a feature where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in monasca, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Monasca (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Queens	Introduced

Monasca services in Docker

Tue, 23 Apr 2019 00:00:00

The main task of this story is to move building and publishing Docker images of all Monasca components from https://github.com/monasca/monasca-docker to their respective OpenStack repositories.

Problem description

At the moment Docker images for Monasca components are built from Dockerfiles provided in monasca-docker repository. The process of building and publishing the images has to be explicitly triggered and is described here. Due to the separation of image definitions and the actual upstream code they can easily diverge.

Use Cases

Have supported and standardized option to deploy Monasca services with Docker.

From the development point of view very little will change.

A developer is writing code and putting it into Gerrit, then automatic process test if Docker image will be built properly.

The images can be used in integration tests running in OpenStack CI.

End User should see no change.

Deployer has one more supported way of deploying Monasca.

Proposed change

Every repository with each component would have additional docker folder that would contain all necessary files for building Docker image.

Example files:

Dockerfile
README
Starting script that will template needed configuration files, wait for other needed components to be available (like Keystone) and start service on Docker image run.
Templates for configuration files, bootstrap scripts for configuring database schemas etc..

The process should be started after commit land in git repository (so all tests passing). The goal is a fully automatic process without the need for human intervention. Every image should have an easy and standardized way of getting information from what version of Monasca component it was built (definitely needed for master and latest tags). Every Dockerfile will be in a separate repository so there is a need for standardization of these files so that they look mostly the same. There are linters for Dockerfiles, one need to be chosen and added to the testing process (will help with standardization of the Dockerfiles, check Kolla jobs).

Images will be build with Python 3.5. If any Monasca service does not support Python 3, we will wait for support before creating container for it.

Images will be published to https://hub.docker.com/u/monasca/

Each image should have README file in RST format with information about running specific component.

We will be supporting Docker actual stable version and 2 previous versions.

All stable branches will have specific versions of Docker that they are build with and this versions will be frozen at the moment of branching them from master. Master branch will be build with 3 newest Docker versions.

All images will be based on custom monasca-base image that is using official Python on Alpine Linux to conserve disc space. We will be using ideas for minimal configuration from other base image examples (like based on Ubuntu https://github.com/phusion/baseimage-docker or based on Alpine https://github.com/blacklabelops/baseimages).

All images should be created in a way that will try to make them as small as possible without compromising on functionality. Possible ways of making them smaller could be found in articles like the following

https://blog.codeship.com/alpine-based-docker-images-make-difference-real-world-apps/

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

All images standardize on Jinja2 for templating configuration files with the following tool:

https://github.com/Aisbergg/python-templer - Python3 command-line tool for templating in shell-scripts, leveraging the Jinja2 library

Allows to use environment variables

YAML data sources supported

LGPL license but we are not linking against it, only using CLI

Alternatives

As an example of the CI configuration, we could use Kolla process of creating Docker images: https://github.com/openstack/kolla https://hub.docker.com/u/kolla/ It is though much too complex for what we need, so should be used just as a reference.

For templating:

https://github.com/wrouesnel/p2cli - command line tool for rendering pongo2 (Django-syntax like, similar to Jinja2) templates (Go binary, 4x smaller size if the image does not already have Python)

Allows to use environment variables

YAML, JSON data sources supported

https://github.com/jwilder/dockerize - Utility to simplify running applications in docker containers (Go binary)

Generate application configuration files at container startup time from templates (Go text/template) and container environment variables

Tail multiple log files to stdout and/or stderr

Wait for other services to be available using TCP, HTTP(S), unix before starting the main process.

Data model impact

None

REST API impact

None

Security impact

All services will be closed in Docker container.

Other end user impact

None

Performance Impact

Because of additional Docker layer services could run slower.

Other deployer impact

Deployment should be easier as deployer would need to create configuration and services with all dependencies will be enclosed in Docker images.

Developer impact

Features adding new dependencies or changing the way Monasca components are installed would have to be reflected in Docker image definitions.

Implementation

Assignee(s)

Primary assignee:: dobroslaw-zybort <dobroslaw.zybort@ts.fujitsu.com>

Work Items

What is needed:

Building test images for every change in Gerrit.
- Automated process of building images on every git commit and pushing them to the Docker Hub.
- Automated process of building images on every tag and pushing them to the Docker Hub.
- Automated process of building images on OpenStack releases and pushing them to the Docker Hub (could be the second step).
Adding labels with source code base information of the image (e.g. build-date, commit-sha1).
Running integration tests on each code commit to verify the Docker binary. Single node deployment setup with Docker Compose.
- Tempest tests.
- Smoke tests.

Five types of tags on Docker Hub:

On every new release/tag.
latest pointing to the last tag.
master pointing to the last git commit - useful for testing last. changes/fixes before release.
Tags for the last commit on stable OpenStack versions (queens, rocky etc.) - useful for testing last changes/fixes before release.

Optional:

Build all images with Python 3.6 and test stability.
Run multi node deployment integration tests with Kubernetes Helm.

Needs to remember (to make maintenance easier):

Using proper init process.
Cleaning after the installation process.
Same name of starting script in all repositories.
Use same config templating mechanism in all repositories.
All components should be downloaded from Git repository and have a possibility to change branch for building Docker image (useful for testing changes proposed on Gerrit).

Non objectives of this story (could be next steps):

Automated process of building images on specific past commits and pushing them to the Docker Hub.
Migrate Devstack to Docker images of Monasca.

Dependencies

What should be moved to where:

Not in scope:

https://github.com/monasca/monasca-docker/monasca-alarms - This image contains a container that can be used to create Monasca Notifications and Alarm Definitions.
https://github.com/monasca/monasca-docker/monasca-log-agent - Logstash output monasca_log_api plugin.
https://github.com/monasca/monasca-docker/monasca-log-metrics - Contains Logstash configuration to transform logs into metrics based on log’s severity.
https://github.com/monasca/monasca-docker/monasca-log-persister - Contains Logstash configuration to save logs inside log-db (i.e. ElasticSearch).
https://github.com/monasca/monasca-docker/monasca-log-transformer - Image contains Logstash configuration to detect log’s severity.

Testing

At this moment CI for https://github.com/monasca/monasca-docker run two types of tests on each change:

tempest-tests https://github.com/monasca/monasca-docker/tree/master/tempest-tests
smoke-tests https://github.com/monasca/monasca-docker/tree/master/smoke-tests
- https://github.com/monasca/smoke-test

Both of them, at the moment, are running on metrics part of the Monasca stack.

Tests should consider if tested service started and is behaving correctly in built and running Docker container. We need to decide if we want to run all tests on the whole Monasca stack on every change or if we should create some smaller tests for each separate service.

Documentation Impact

Basic installation instructions should be added here [1] and published to https://docs.openstack.org

[1] https://opendev.org/openstack/monasca-api/src/branch/master/doc/source/install

Now high level documentation is stored in: https://github.com/monasca/monasca-docker/tree/master/docs

Separate images also have README.md files that give lower level information.

Documentation should contain all necessary information how to configure and run all services.

References

History

Optional section intended to be used each time the spec is updated to describe the new design, API or any database schema updated. Useful to let reader understand what’s happened at the time.

Revisions
Release Name	Description
Rocky	Introduced

Monasca Events Publishing - from Ceilometer

Tue, 23 Apr 2019 00:00:00

https://storyboard.openstack.org/#!/story/2003023

Monasca Events API [1] was developed to store Openstack Notification data in Elasticsearch. There is still a need to collect and publish Openstack Notifications to Monasca Events API. Monasca Ceilometer project[2] currently publishes ceilometer samples[3] to Monasca API. We are proposing to extend Monasca Ceilometer project and add a new events publisher which will publish Openstack notifications (or events)[3] to Monasca Events API.

UPDATE: This spec is being superceded by the ../../stein/approved/monasca-events-listener.rst spec, but is kept here for reference.

Problem description

All Openstack services generate a lot of notifications or events which contain large amounts of operational and state information about the service and its resources. This notification data is not currently available in Monasca.

Ceilometer data processing pipeline[3] provides an extensible mechanism of publishing samples and events using a custom publisher. Ceilometer samples represent a quantity that can be measured (for e.g. the size of a volume) and events represent an occurrence of an event and do not have any associated quantity (e.g. volume was created).

Monasca Ceilometer project currently provides a samples publisher. Monasca Ceilometer samples publisher converts Ceilometer samples to Monasca Metrics format which are then published to Monasca API. There is no corresponding events publisher to Monasca yet.

By adding an event publisher to Monasca Ceilometer project we could take advantage of Ceilometer’s event publishing mechanism to publish events to Monasca Events API.

Ceilometer consists of different data collection components - namely Polling Agent, Notification Agent and Compute Agent. (Please see [7] for System Architecture diagram) Ceilometer also has a data storage and retrieval component, which would be Monasca in our case.

Samples publisher and new proposed events publisher run within Ceilometer’s Notification Agent component and are part of Notifcation Agent’s data processing pipeline. Monasca Ceilometer presumes the need to install and deploy Ceilometer Notification Agent component (doesn’t need Polling Agent or Compute Agent deployed) on all the control nodes. Ceilometer Notification Agent is Highly Available (HA) and can run on multiple nodes. We will have to evaluate its performance in terms of scaling for events, but we haven’t run into performance/scale problems with current samples publisher.

Use Cases

Openstack notification data would be stored in Elasticsearch via the Monasca Events API

Example sequence from Nova notification to Monasca API #. Nova completes the creation of a VM #. Nova generates a Notification message to oslo.messaging #. Ceilometer Notification agent receives the Notification message #. Ceilometer translates the Notification to a Monasca API format according to the configuration #. Ceilometer Event Publisher publishes formatted Notification to Monasca Events API #. Monasca Events API receives and validates formatted Notification #. Monasca Events stores event Notification in configured Elasticsearch instance

Proposed change

Openstack Notifications consist of envelope and payload fields

Example Openstack Notification data format:

{
    "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
    "_context_domain": null,
    "_context_instance_lock_checked": false,
    "_context_is_admin": true,
    "_context_project_domain": null,
    "_context_project_id": "a4f77",
    "_context_project_name": "admin",
    "_context_quota_class": null,
    "_context_read_deleted": "no",
    "_context_read_only": false,
    "_context_remote_address": "192.168.245.4",
    "_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
    "_context_resource_uuid": null,
    "_context_roles": [
        "monasca-user",
        "admin",
        "KeystoneAdmin"
    ],
    "_context_service_catalog": [
        {
            "endpoints": [
                {
                    "adminURL": "http://192.168.245.8:8776/v2/a4f77",
                    "internalURL": "http://192.168.245.8:8776/v2/a4f77",
                    "publicURL": "http://192.168.245.9:8776/v2/a4f77",
                    "region": "region1"
                }
            ],
            "name": "cinderv2",
            "type": "volumev2"
        },
        {
            "endpoints": [
                {
                    "adminURL": "http://192.168.245.8:8776/v1/a4f77",
                    "internalURL": "http://192.168.245.8:8776/v1/a4f77",
                    "publicURL": "http://192.168.245.9:8776/v1/a4f77",
                    "region": "region1"
                }
            ],
            "name": "cinder",
            "type": "volume"
        }
    ],
    "_context_show_deleted": false,
    "_context_tenant": "a4f77",
    "_context_timestamp": "2015-09-18T20:54:23.468522",
    "_context_user": "be396488c7034811a200a3cb1d103a28",
    "_context_user_domain": null,
    "_context_user_id": "be396488c7034811a200a3cb1d103a28",
    "_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -",
    "_context_user_name": "admin",
    "_unique_id": "ff9699d587bf4283a3c367ab88be1541",
    "event_type": "compute.instance.create.start",
    "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
    "payload": {
        "access_ip_v4": null,
        "access_ip_v6": null,
        "architecture": null,
        "availability_zone": null,
        "cell_name": "",
        "created_at": "2015-09-18 20:55:25+00:00",
        "deleted_at": "",
        "disk_gb": 1,
        "display_name": "testeee",
        "ephemeral_gb": 0,
        "host": null,
        "hostname": "testeee",
        "image_meta": {
            "base_image_ref": "df0c8",
            "container_format": "bare",
            "disk_format": "qcow2",
            "min_disk": "1",
            "min_ram": "0"
        },
        "image_name": "glanceaaa3",
        "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
        "instance_flavor_id": "1",
        "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
        "instance_type": "m1.tiny",
        "instance_type_id": 4,
        "kernel_id": "",
        "launched_at": "",
        "memory_mb": 512,
        "metadata": {},
        "node": null,
        "os_type": null,
        "progress": "",
        "ramdisk_id": "",
        "reservation_id": "r-1ghilddw",
        "root_gb": 1,
        "state": "building",
        "state_description": "",
        "tenant_id": "a4f77",
        "terminated_at": "",
        "user_id": "be396488c7034811a200a3cb1d103a28",
        "vcpus": 1
    },
    "priority": "INFO",
    "publisher_id": "compute.ccp-compute0001-mgmt",
    "timestamp": "2015-09-18 20:55:37.639023"
}

All the fields with the prefix of ‘_context” are the envelope fields, the other interesting fields are
1. ‘message_id’ - notification identifier
2. ‘payload’ - contains most of the relevant and useful information in JSON format
3. ‘priority’ - notification priority
4. ‘publisher_id’ - notification publisher
5. ‘timestamp’ - notification timestamp

Ceilometer event publishing framework converts the Openstack notifications to events format[4]. Event publishing framework also has the ability to extract only some of the ‘payload’ data into a flat set of key-value pairs called ‘traits’ and publish the normalized ‘event’ with ‘traits’ extracted from the payload using a custom publisher.

Extraction of certain fields into traits from the payload is driven by configuration file, but by default “publisher_id”, ‘request_id’, ‘tenant_id’, ‘user_id’ and ‘project_id’ fields are always extracted and added as ‘traits’.

The event can also have an optional field called ‘raw’ which has original notification, provided ‘store_raw’ option is set in ceilometer.conf

Questions/TODO:

Q1: Does the store_raw field only apply to events, or to all notifications processed by Ceilometer?
We will have to find it out if it has any adverse impact on sample publisher. Though in the case of samples, monasca sample publisher definitely does not submit raw payload, so it must be getting dropped.

Example Ceilometer Event data format:

{
  "event_type": "compute.instance.create.start",
  "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
  "generated": "2015-09-18 20:55:37.639023",
  "traits": {
     "publisher_id": "compute.ccp-compute0001-mgmt",
     "request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
     "tenant_id": "a4f77",
     "project_id": "a4f77",
     "user_id": "be396488c7034811a200a3cb1d103a28"
   },
   "raw": {  "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
             "_context_domain": null,
             ...
             ...
             "event_type": "compute.instance.create.start",
             "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
             "payload": {
                 "access_ip_v4": null,
                 "access_ip_v6": null,
                 "architecture": null,
                 "availability_zone": null,
                 "cell_name": "",
                 "created_at": "2015-09-18 20:55:25+00:00",
                 "deleted_at": "",
                 "disk_gb": 1,
                 "display_name": "testeee",
                 "ephemeral_gb": 0,
                 "host": null,
                 "hostname": "testeee",
                 "image_meta": {
                     "base_image_ref": "df0c8",
                     "container_format": "bare",
                     "disk_format": "qcow2",
                     "min_disk": "1",
                     "min_ram": "0"
                 },
                "image_name": "glanceaaa3",
                "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
                "instance_flavor_id": "1",
                "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
                "instance_type": "m1.tiny",
                "instance_type_id": 4,
                "kernel_id": "",
                "launched_at": "",
                "memory_mb": 512,
                "metadata": {},
                "node": null,
                "os_type": null,
                "progress": "",
                "ramdisk_id": "",
                "reservation_id": "r-1ghilddw",
                "root_gb": 1,
                "state": "building",
                "state_description": "",
                "tenant_id": "a4f77",
                "terminated_at": "",
                "user_id": "be396488c7034811a200a3cb1d103a28",
                "vcpus": 1
                }
          }
}

Key-Value pairs that can be extracted from ‘payload’ in form of traits can be defined in events definitions file.

For example the following events definitions yaml specifies that for all events which have a prefix of “compute.instance.*” then add “user_id”, “instance_id”, and “instance_type_id” as traits, after extracting values from “payload.user_id”, “payload.instance_id”, and “payload.instance_type_id” respectively.
— - event_type: compute.instance.*

traits: &instance_traits

user_id:
fields: payload.user_id

instance_id:
fields: payload.instance_id

instance_type_id:
type: int fields: payload.instance_type_id

We are for now proposing not to use this feature, of defining traits for each event extracting since we have the ability to store entire payload, via Monasca Events API.

We can certainly look at enabling this feature in the future if we run into trouble storing entire JSON “payload” in Elasticsearch. This is certainly a nifty way to trim the amount of data that will be stored.
The proposed new Custom Monasca Ceilometer event publisher will run within Ceilometer’s Notification Agent component. It will leverage Ceilometer’s data processing pipeline[3] which converts notifications to Ceilometer’s event format. At the end of its processing, Monasca Ceilometer event publisher will convert Ceilometer Event data into Monasca Event format[6] and publish the monasca event to Monasca Events API.

Monasca Events API allows a field called ‘payload’ which can be in an arbitrary nested JSON format. Monasca-Ceilometer event publisher will extract JSON field called ‘payload’ from ‘raw’ (JSON path notation: ‘raw.payload’), publish the payload from the original notification to Monasca Events API.

Example Monasca Event Format:

events: [
{
  dimensions": {
        "service": "compute.ccp-compute0001-mgmt",
        "topic": "notification.sample",
        "hostname": "nova-compute:compute
  },
  event: {

          "event_type": "compute.instance.create.start",

          "payload": {
               "access_ip_v4": null,
               "access_ip_v6": null,
               "architecture": null,
               "availability_zone": null,
               "cell_name": "",
               "created_at": "2015-09-18 20:55:25+00:00",
               "deleted_at": "",
               "disk_gb": 1,
               "display_name": "testeee",
               "ephemeral_gb": 0,
               "host": null,
               "hostname": "testeee",
               "image_meta": {
                   "base_image_ref": "df0c8",
                   "container_format": "bare",
                   "disk_format": "qcow2",
                   "min_disk": "1",
                   "min_ram": "0"
               },
              "image_name": "glanceaaa3",
              "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
              "instance_flavor_id": "1",
              "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
              "instance_type": "m1.tiny",
              "instance_type_id": 4,
              "kernel_id": "",
              "launched_at": "",
              "memory_mb": 512,
              "metadata": {},
              "node": null,
              "os_type": null,
              "progress": "",
              "ramdisk_id": "",
              "reservation_id": "r-1ghilddw",
              "root_gb": 1,
              "state": "building",
              "state_description": "",
              "tenant_id": "a4f77",
              "terminated_at": "",
              "user_id": "be396488c7034811a200a3cb1d103a28",
              "vcpus": 1
              }
         },
    publisher_id: "compute.ccp-compute0001-mgmt",
    priority: "INFO"
 }
]

If no traits are specified in events pipeline yaml configuration file for an event Ceilometer’s data processing pipeline will add the following default traits:
- service: (All notifications should have this) notification’s publisher
- tenant_id
- request_id
- project_id
- user_id
Note: “service” is not the service that produced the event as in say “compute”, “glance”, “cinder” but rather notification RabbitMQ publisher that produced the event e.g. “compute.ccp-compute0001-mgmt” so is not very useful.
Ceilometer event data is converted to Monasca event data format before being published to Monasca Event API. Following fields in Monasca Event data are not available in current Ceilometer Event data format:
- “service”
- “dimensions.topic”
- “event.priority”
We are proposing removing these fields from Monasca Event format (will be done as a separate spec/implementation process) for the following reasons:

“service”: Currently Openstack notifications do not specify a service, that generated the notification in a consistent way. It might be possible to create an external mapping file which maps event name to a service but its hard to maintain such mapping over a period of time.

“dimensions.topic”: This field is not available in the source Openstack notification

“event.priority”: This field is not currently available in Ceilometer Event format. It is available in the source Openstack notification. Note: If we think this field can be useful we can propose adding it to the Ceilometer Event format.
Following new fields will be added to Monasca Event data as dimensions:
- “dimensions.publisher_id”: Identifier for the publisher that generated the event. Maps to “traits.publisher_id” in Ceilometer event data.
- “dimensions.user_id”: Identifier for user that generated the event. Maps to “traits.user_id” in Ceilometer event data.
- “dimensions.project_id”: Identifier of the project that generated the event. Maps to “traits.project_id” or “traits.tenant_id” in Ceilometer event data.
hostname is available in the event payload, but its location might differ from event to event. We can use Ceilometer’s event definitions config to always add a trait called “hostname” to all events. e.g. for compute.instance.* will have a trait called “hostname”, which grabs data from “payload.hostname”
— - event_type: compute.instance.*

traits: &instance_traits

user_id:
fields: payload.hostname
The proposed new Monasca Ceilometer event publisher will have the ability to submit event data in a batch and at a configurable frequency (similar to current samples publisher). The event data will be published if the items in the current batch reach their maximum size (config setting) or if certain time interval has elapsed since the last publish (config setting). This will make sure that the batch does not get huge at the same time there is no significant delay in publishing of the events to Monasca Events API.
Monasca Ceilometer event publisher will use service credentials from ceilometer configuration file (in “[monasca]” section) to get keystone token.

Example “[monasca]” section in ceilometer config file .. code-block:: text

[monasca] service_auth_url = https://localhost:5000/v3 service_password = secretpassword service_username = ceilometer service_interface = internalURL service_auth_type = password # service_project_id may also be used service_project_name = admin service_domain_name = Default service_region_name = RegionOne

The publisher will then make a POST request to Monasca Events /v1.0/events REST api[8] to publish events to Monasca Events API. The URL for the instance of Monasca Events API will be configured in the Ceilometer ‘events-pipeline.yaml’ file. This has the added advantage of allowing different events to be published differently (see Ceilometer pipeline documentation [10]).
“tenant_id” and “user_id” that the notification relates to are available in “payload” section of the notification, and these notifications are generated by each service itself.

There is no additional “Openstack-operator-agent” like component or functionality required to fetch that data from the service and publish to monasca event api on behalf of the original tenant. Ceilometer publishing pipeline simply extracts these “tenant_id” and “user_id” fields from the “payload” and makes those fields available as “tenant_id” and “user_id” traits, which would then be mapped to “dimensions.project_id” and “dimensions.user_id” fields in monasca events format.

In other words, original “tenant_id” and “user_id” values are available in the payload of the notification, and will make its way to “dimensions.tenant_id” and “dimensions.user_id” in Monasca Event.

Questions/TODO: * Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being done for metrics[9] ? Would original user_id and tenant_id in “dimensions.user_id” and “dimensions.tenant_id” fields in dimensions serve this purpose? * Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role “admin” could access data for all tenants, whereas a user with “ceilometer-admin” role could access only data for a particular tenant. Can we implement something like this for monasca-events api when querying for data?
Monasca Ceilometer event publisher will also retry submitting a batch, in case Monasca Events API is temporarily unavailable or down. The retry frequency, the number of retries and the number of items that can be in the retry batch will also be set via configuration.

Alternative Solutions

Standalone monasca event agent which reads Openstack notifications published to RabbitMQ (on “notification” topic) and publishes them to Monasca Events API. Pro: * No dependency on Telemetry project. * May be simple to develop if leverage the oslo.messaging functionality. * Ceilometer has deprecated the events functionality in the Stein release. [13] Con: * Another agent to convince users to install on their systems. * Reinventing work already done in the Ceilometer agent. The OpenStack community already uses Ceilometer and contributes updates when something fails. This alternate solution will be detailed in a separate spec, as it is likely the long term solution Monasca will need.
Openstack Panko [5] is a event storage and REST API for Ceilometer. Pro: * An ‘official’ subproject within Telemetry, so there is some community recognition. Con: * Its primary storage is in a relational database which has problems with scale. * It is not maintained actively and not ready for production. [11] * It will be deprecated eventually. [12]

Data model impact

None

REST API impact

We are proposing to tweak the Monasca Event data format by removing and adding following fields as mentioned in “Proposed change” section above.

Remove fields (JSON path notation): “service”, “dimensions.topic”, “dimensions.hostname” and “event.priority”

Add fields (JSON path notation): “dimensions.publisher_id”, “dimensions.user_id” and “dimensions.project_id”

This change will have an impact on Monasca Events API.

Security impact

The proposed Monasca Ceilometer events publisher will collect and publish Openstack event (notification) data to Monasca API. Openstack notification data does not have any sensitive data like ‘tokens’. Notifications do contain ‘user_id’ and ‘project_id’ fields but do not contain any Personally Identifiable Information (PII) for the user or the project.

Other end user impact

None.

Performance Impact

The number of notifications(events) generated by different services will depend on the capacity of the cloud along with the number of resources being created by the users.

For example, if there was a large number of compute VM’s being created or destroyed it could lead to a surge in number of notifications (events) that would have to be published. Optimum configuration options related to say event batch size and event batch interval would have to be documented, to reduce any adverse affect on performance.
Monasca Ceilometer publisher runs within Ceilometer Notification Agent component and invoked as a last step in its data processing pipeline. It is an additional component that will have to to be deployed on all the controller nodes. We will have to evaluate the performance impact of Ceilometer Notification Agent when publishing events to Monasca Events API.

Other deployer impact

The proposed new Monasca-Ceilometer events publisher will introduce few new configuration options like * events api endpoint * events batch interval * events batch size * events retry interval
Monasca Ceilometer Events publisher will have to to be added to Ceilometer’s “[ceilometer.event.publisher]” section entry_points.txt

For example:

[ceilometer.event.publisher] monasca = ceilometer.publisher.monclient:MonascaEventsPublisher
As part of developing new Monasca Ceilometer Events publisher devstack plugin would be updated to add the above configuration changes.

Developer impact

The proposed change to Monasca Event Format will have an impact on existing Monasca Event API, since Monasca Event Format will have to be tweaked. (See REST API Impact section above)

Implementation

Assignee(s)

Primary assignee:: joadavis, aagate
Other contributors:: <launchpad-id or None>

Work Items

Implement new Monasca Ceilometer Events publisher.
Implement monasca-ceilometer devstack plugin changes to deploy new events publisher.
Implement unit tests for Events publisher.
Implement change to Monasca Event format in Monasca Events API.

Dependencies

Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654
Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer

#. Ceilometer Data processing and pipelines: https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html

Testing

New Monasca Ceilometer Event publisher unit tests will be added, which can test publishing with various config options events batch size, events batch interval, handling retry when Monasca Event API is not available.
Adding tempest tests for Monasca Ceilometer events publisher could be looked at as part of separate effort.

Please note that current Monasca Ceilometer samples publisher does not have tempest tests either so having tempest tests for both events and samples publisher could be considered in the future.

Documentation Impact

New Monasca Events Publisher config options will be documented
- events api endpoint
- events batch interval
- events batch size
- events retry interval
Recommended values for each of the config options will also be documented based on the size of the cloud and resources for Cloud Operators.

References

[1] Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654

[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer

[3] Ceilometer Data processing and pipelines: https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html

[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html

[5] Openstack Panko: https://github.com/openstack/panko

[6] Monasca Event Format: https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json

[7] Ceilometer System Architecture Diagram: https://docs.openstack.org/ceilometer/ocata/architecture.html

[8] Monasca Events POST v1.0 API: https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc

[9] Cross-Tenant Metric Submission: https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission

[10] Ceilometer pipeline yaml documentation: https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html

[11] No future for Panko or Aodh: https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/

[12] Ceilometer Events deprecated means Panko also deprecated: http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html

[13] Ceilometer Events marked as deprecated in Stein: https://review.opendev.org/#/c/603336/

Example Spec - The title of your feature request

Tue, 23 Apr 2019 00:00:00

Include the URL of your story:

https://storyboard.openstack.org

Some notes about the monasca-spec and stories process:

Not all stories need a spec. For more information see https://docs.openstack.org/monasca-api/latest/contributor/index.html
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Monasca REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/monasca-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this feature request addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-monascaclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure Monasca that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on Monasca.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a feature where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in monasca, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Monasca (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Rocky	Introduced

Monasca Events Listener

Tue, 23 Apr 2019 00:00:00

https://storyboard.openstack.org/#!/story/2003023

Monasca Events API 1 was developed to store event data in Elasticsearch. A new application could use the Monasca Events API to post an event directly for processing and storage. However, a general collection service is needed to capture the existing OpenStack Notifications already generated by OpenStack Services and pass them to Monasca for storage and processing. This specification proposes creating a new Monasca Events Listener to capture events and pass them to Monasca services.

Problem description

All Openstack services generate a lot of notifications or events which contain large amounts of operational and state information about the service and its resources. Services such as Nova 14 already publish to a RabbitMQ topic. This notification data is not currently available in Monasca.

Ceilometer data processing pipeline 3 provides an extensible mechanism of publishing samples and events using a custom publisher. Ceilometer samples represent a quantity that can be measured (e.g. the size of a volume) and events represent an occurrence of an event and do not have any associated quantity (e.g. volume was created). The Telemetry project also has the Panko 5 service for indexing and storing these events.

A previous spec was created to specify an enhancement to Ceilometer to allow collected events to be published to the Monasca Events API. However, with the recent deprecation of Ceilometer’s Event functionality 13, this is no longer a long-term option.

This spec is for creating a new Monasca service which would listen for OpenStack Event messages (often called notifications) and process them through Monasca by consuming the RabbitMQ message and producing a post to the Monasca Events API with that event.

This service could use Ceilometer, Vitrage, Watcher, or another service as an example of how to listen to notifications from OpenStack services such as Nova 14.

It is being proposed to make the Monasca Events Listener service a part of the Monasca Agent code base and reuse code, including the existing monasca_setup script and config.

Use Cases

Openstack notification data would be stored in Elasticsearch via the Monasca services

Example sequence:
1. Nova completes the creation of a VM
2. Nova generates a Notification message to oslo.messaging
3. oslo.messaging posts the message to RabbitMQ
4. Monasca Event Listener receives the Notification message from RabbitMQ
5. Monasca Event Listener validates and translates the Notification to a Monasca Events API format according to the configuration
6. Monasca Event Listener publishes formatted Notification to Monasca Events API
7. Monasca Events API receives and validates formatted Notification
8. Monasca Events API stores event Notification in configured Elasticsearch instance

Proposed change

Monasca Event Listener will be a new service which can be run in a Highly Available configuration by running an instance of Monasca Event Listener on each Controller node in a cloud. Each node will listen to OpenStack notifications in RabbitMQ and convert notifications to a post to the Monasca Events API. The Monasca Events API then passes the notification on as Kafka messages on the ‘monevents’ topic, where the Monasca Persister will receive and store them. By the nature of RabbitMQ clients, the load will be distributed between the Monasca Events Listener instances (only one listener will process each RabbitMQ message).
Monasca Event Listener will filter messages from OpenStack services based on specification of event_types to be collected. This will reduce ‘noise’ and focus event collection on those events that are deemed valuable. This filtering specification can be from a configuration file, or optionally could be controlled through a new API implemented as part of the Monasca Events API service.

OpenStack Notifications consist of envelope and payload fields. See 15 and 16 for examples.

Example OpenStack Notification data format:

{
    "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
    "_context_domain": null,
    "_context_instance_lock_checked": false,
    "_context_is_admin": true,
    "_context_project_domain": null,
    "_context_project_id": "a4f77",
    "_context_project_name": "admin",
    "_context_quota_class": null,
    "_context_read_deleted": "no",
    "_context_read_only": false,
    "_context_remote_address": "192.168.245.4",
    "_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
    "_context_resource_uuid": null,
    "_context_roles": [
        "monasca-user",
        "admin",
        "KeystoneAdmin"
    ],
    "_context_service_catalog": [
        {
            "endpoints": [
                {
                    "adminURL": "http://192.168.245.8:8776/v2/a4f77",
                    "internalURL": "http://192.168.245.8:8776/v2/a4f77",
                    "publicURL": "http://192.168.245.9:8776/v2/a4f77",
                    "region": "region1"
                }
            ],
            "name": "cinderv2",
            "type": "volumev2"
        },
        {
            "endpoints": [
                {
                    "adminURL": "http://192.168.245.8:8776/v1/a4f77",
                    "internalURL": "http://192.168.245.8:8776/v1/a4f77",
                    "publicURL": "http://192.168.245.9:8776/v1/a4f77",
                    "region": "region1"
                }
            ],
            "name": "cinder",
            "type": "volume"
        }
    ],
    "_context_show_deleted": false,
    "_context_tenant": "a4f77",
    "_context_timestamp": "2015-09-18T20:54:23.468522",
    "_context_user": "be396488c7034811a200a3cb1d103a28",
    "_context_user_domain": null,
    "_context_user_id": "be396488c7034811a200a3cb1d103a28",
    "_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -",
    "_context_user_name": "admin",
    "_unique_id": "ff9699d587bf4283a3c367ab88be1541",
    "event_type": "compute.instance.create.start",
    "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
    "payload": {
        "access_ip_v4": null,
        "access_ip_v6": null,
        "architecture": null,
        "availability_zone": null,
        "cell_name": "",
        "created_at": "2015-09-18 20:55:25+00:00",
        "deleted_at": "",
        "disk_gb": 1,
        "display_name": "testeee",
        "ephemeral_gb": 0,
        "host": null,
        "hostname": "testeee",
        "image_meta": {
            "base_image_ref": "df0c8",
            "container_format": "bare",
            "disk_format": "qcow2",
            "min_disk": "1",
            "min_ram": "0"
        },
        "image_name": "glanceaaa3",
        "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
        "instance_flavor_id": "1",
        "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
        "instance_type": "m1.tiny",
        "instance_type_id": 4,
        "kernel_id": "",
        "launched_at": "",
        "memory_mb": 512,
        "metadata": {},
        "node": null,
        "os_type": null,
        "progress": "",
        "ramdisk_id": "",
        "reservation_id": "r-1ghilddw",
        "root_gb": 1,
        "state": "building",
        "state_description": "",
        "tenant_id": "a4f77",
        "terminated_at": "",
        "user_id": "be396488c7034811a200a3cb1d103a28",
        "vcpus": 1
    },
    "priority": "INFO",
    "publisher_id": "compute.ccp-compute0001-mgmt",
    "timestamp": "2015-09-18 20:55:37.639023"
}

All the fields with the prefix of ‘_context’ are the envelope fields, the other interesting fields are
1. ‘message_id’ - notification identifier
2. ‘payload’ - contains most of the relevant and useful information in JSON format
3. ‘priority’ - notification priority
4. ‘publisher_id’ - notification publisher
5. ‘timestamp’ - notification timestamp
Monasca Event Listener converts the OpenStack notifications to Monasca events format. This format will be suitable for Kafka messaging and will match the expected data fields of the Monasca Persister. This conversion and validation should be common between the Monasca Event Listener and Monasca Event API.
The Kafka client connection will handle communication issues such as reconnections and resending as needed.

Monasca Events API allows a field called ‘payload’ which can be in an arbitrary nested JSON format.

Example Monasca Event Format:

events: [
{
  dimensions": {
        "service": "compute.ccp-compute0001-mgmt",
        "topic": "notification.sample",
        "hostname": "nova-compute:compute
  },
  event: {

          "event_type": "compute.instance.create.start",

          "payload": {
               "access_ip_v4": null,
               "access_ip_v6": null,
               "architecture": null,
               "availability_zone": null,
               "cell_name": "",
               "created_at": "2015-09-18 20:55:25+00:00",
               "deleted_at": "",
               "disk_gb": 1,
               "display_name": "testeee",
               "ephemeral_gb": 0,
               "host": null,
               "hostname": "testeee",
               "image_meta": {
                   "base_image_ref": "df0c8",
                   "container_format": "bare",
                   "disk_format": "qcow2",
                   "min_disk": "1",
                   "min_ram": "0"
               },
              "image_name": "glanceaaa3",
              "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
              "instance_flavor_id": "1",
              "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
              "instance_type": "m1.tiny",
              "instance_type_id": 4,
              "kernel_id": "",
              "launched_at": "",
              "memory_mb": 512,
              "metadata": {},
              "node": null,
              "os_type": null,
              "progress": "",
              "ramdisk_id": "",
              "reservation_id": "r-1ghilddw",
              "root_gb": 1,
              "state": "building",
              "state_description": "",
              "tenant_id": "a4f77",
              "terminated_at": "",
              "user_id": "be396488c7034811a200a3cb1d103a28",
              "vcpus": 1
              }
         },
    publisher_id: "compute.ccp-compute0001-mgmt",
    priority: "INFO"
 }
]

Following fields in Monasca Event data may not be available in the OpenStack notification data format:
- “service”
- “dimensions.topic”
- “event.priority”
We are proposing removing these fields from Monasca Event format (will be done as a separate spec/implementation process) for the following reasons:

“service”: Currently OpenStack notifications do not specify a service, that generated the notification in a consistent way. It might be possible to create an external mapping file which maps event name to a service but its hard to maintain such mapping over a period of time.

“dimensions.topic”: This field is not available in the source OpenStack notification. However, the Monasca Event Listener may be able to save the RabbitMQ topic that the notification was collected from. In that case, this field should be used.

“event.priority”: This field is not currently available in Ceilometer Event format. It is available in the source OpenStack notification. Note: If we think this field can be useful we can propose adding it to the Monasca Event Listener format.
Following new fields will be added to Monasca Event data as dimensions:
- “dimensions.publisher_id”: Identifier for the publisher that generated the event.
- “dimensions.user_id”: Identifier for user that generated the event.
- “dimensions.project_id”: Identifier of the project that generated the event.
‘hostname’ is available in the event payload, but its location might differ from event to event.
The proposed new Monasca Event Listener will have the ability to submit event data in a batch and at a configurable frequency (similar to current samples publisher). The event data will be published if the items in the current batch reach their maximum size (config setting) or if certain time interval has elapsed since the last publish (config setting). This will make sure that the batch does not get huge at the same time there is no significant delay in publishing of the events to Monasca Events API.
Monasca Event Listener will have a configuration file to configure connection information for both RabbitMQ and Monasca Events API.
The “tenant_id” and “user_id” that the notification relates to are available in “payload” section of the notification, and these notifications are generated by each service itself.

There is no additional “OpenStack-operator-agent” like component or functionality required to fetch that data from the service and publish to monasca event api on behalf of the original tenant. (Ceilometer publishing pipeline simply extracts these “tenant_id” and “user_id” fields from the “payload” and makes those fields available as “tenant_id” and “user_id” traits, which would then be mapped to “dimensions.project_id” and “dimensions.user_id” fields in monasca events format.)

In other words, original “tenant_id” and “user_id” values are available in the payload of the notification, and will make its way to “dimensions.tenant_id” and “dimensions.user_id” in Monasca Event.

Questions/TODO:
- Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being done for metrics 9 ? Would original user_id and tenant_id in “dimensions.user_id” and “dimensions.tenant_id” fields in dimensions serve this purpose?
  - A: Monasca Events Listener can start with sending all events to a single “admin” tenant and if required in the future some other process could copy select metrics back to tenant projects.
- Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role “admin” could access data for all tenants, whereas a user with “ceilometer-admin” role could access only data for a particular tenant. Can we implement something like this for monasca-events api when querying for data?
  - A: In Monasca API every request is scoped to the project, so there is no equivalent of Ceilometer’s “admin” role to query data for all projects. So placing all events in to an “admin” project may be the best approach.
- Q: How should services which generate notifications but do not include a tenant_id be handled? For example Keystone 16. How does Ceilometer handle such events?
  - A: If all events are in an “admin” project then admin metrics like shared ceph cluster load or provider network load can be copied back to tenants so they may understand how infrastructure is affecting their workload.
- Note: Configuration of Elasticsearch cluster is out of scope for this spec. If needed could assign a separate Elasticsearch cluster to the events API to avoid overloads.

Alternative Solutions

Reuse the Ceilometer functionality to collect and publish events to the Monasca Events API. While this may be less work initially, Ceilometer has deprecated the Events functionality as of Stein. 13
An alternate Events Listener was proposed that would listen for RabbitMQ events then publish them directly to the ‘monevents’ topic in Kafka. Discussion on this can be seen in the git history for this document and in IRC logs 18.

Pro:
- A much simpler approach, more efficient that HTTP hop through another service.
- No need for batching in service, as RabbitMQ and Kafka clients would handle fast throughput and short network interruptions.
Con:
- Nova Cells v2 each have their own RabbitMQ instances. While most deployments would likely have a centralized RabbitMQ, it is not required in the documented architecture.
- Regions may also cause separation of RabbitMQ instances that need to be monitored.
- While it might be possible to have a service/agent in each Cell publish back to a centralized Kafka directly, our authentication and networking for Kafka was not designed to support that.
OpenStack Panko 5 is a event storage and REST API for Ceilometer and could be used instead of a Monasca solution.

Pro:
- An ‘official’ subproject within Telemetry, so there is some community recognition.
Con:
- Its primary storage is in a relational database which has problems with scale.
- It is not maintained actively and not ready for production. 11
- It will be deprecated eventually. 12

Data model impact

None

REST API impact

We are proposing to tweak the Monasca Event data format by removing and adding following fields as mentioned in “Proposed change” section above.

Remove fields or make them optional (JSON path notation): “service”, “dimensions.topic”, “dimensions.hostname” and “event.priority”

Add fields (JSON path notation): “dimensions.publisher_id”, “dimensions.user_id” and “dimensions.project_id”

This change will have an impact on Monasca Events API.

Security impact

The proposed Monasca Event Listener will collect and publish OpenStack event (notification) data to Monasca Events API. OpenStack notification data does not have any sensitive data like ‘tokens’. Notifications do contain ‘user_id’ and ‘project_id’ fields but do not contain any Personally Identifiable Information (PII) for the user or the project.

Other end user impact

None.

Performance Impact

The number of notifications(events) generated by different services will depend on the capacity of the cloud along with the number of resources being created by the users.

For example, if there was a large number of compute VM’s being created or destroyed it could lead to a surge in number of notifications (events) that would have to be published. Optimum configuration options related to say event batch size and event batch interval would have to be documented, to reduce any adverse affect on performance.
The proposed Monasca Event Listener is a new service, so performance is unknown. However, the Monasca API has been shown to have a high performance throughput.
If any part of the Monasca notification pipeline goes down, notifications could back-up in RabbitMQ and bring down the cluster. The risk of this could be mitigated by using a separate RabbitMQ instance for notifications.

Other deployer impact

The proposed Monasca Event Listener will introduce a few new configuration options like
- RabbitMQ connection information
- Monasca Events API endpoint URL
- events batch interval
- events batch size
- events retry interval
- Keystone credentials for obtaining a token
- Conversion options for OpenStack notifications to the Monasca Event format. This may be stored in separate pipeline configuration files, similar to how transform specs are configured in Monasca Transform.
As part of developing new Monasca Event Listener devstack plugin would be updated to add the above configuration changes.

Developer impact

The proposed change to Monasca Event Format will have an impact on existing Monasca Event API, since Monasca Event Format will have to be tweaked. (See REST API Impact section above)

Implementation

Assignee(s)

Primary assignee:: joadavis, aagate
Other contributors:: <launchpad-id or None>

Work Items

Implement new Monasca Event Listener.
- Connection to RabbitMQ for OpenStack Notifications
- Add filtering of notifications for configured event_types
  - Specification in configuration file
  - (Optional) Creation of a new API to configure event_type subscriptions
- Validation of OpenStack Notification data and format
- Conversion of data format to meet Monasca Events requirements
- Publishing to Monasca Events API
- Configuration of conversion specifications per-event type
Implement monasca devstack plugin changes to deploy new Events Listener service.
Implement unit tests for Monasca Event Listener.

Dependencies

None

Testing

New Monasca Event Listener unit tests will be added, which can test publishing with various config options events batch size, events batch interval, handling retry when Monasca Event API is not available.
Adding tempest tests for Monasca Event Listener could be looked at as part of separate effort.

Documentation Impact

New Monasca Event Listener config options will be documented
Recommended values for each of the config options will also be documented based on the size of the cloud and resources for Cloud Operators.

References

1: Monasca Events API 1.0: https://opendev.org/openstack/monasca-events-api/

[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer

3: Ceilometer Data processing and pipelines: https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html

[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html

5(1,2): Openstack Panko: https://github.com/openstack/panko

[6] Monasca Event Format: https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json

[7] Ceilometer System Architecture Diagram: https://docs.openstack.org/ceilometer/ocata/architecture.html

[8] Monasca Events POST v1.0 API: https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc

9: Cross-Tenant Metric Submission: https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission

[10] Ceilometer pipeline yaml documentation: https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html

11: No future for Panko or Aodh: https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/
12: Ceilometer Events deprecated means Panko also deprecated: http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html
13(1,2): Ceilometer Events marked as deprecated in Stein: https://review.opendev.org/#/c/603336/
14(1,2): Nova notification version update lists services effected (see “Deprecating legacy notifications”): https://etherpad.openstack.org/p/nova-ptg-stein
15: Nova notification reference: https://docs.openstack.org/nova/latest/reference/notifications.html#existing-versioned-notifications
16(1,2): Keystone notification reference: https://docs.openstack.org/keystone/latest/advanced-topics/event_notifications.html#example-notification-project-create

[17] Monasca Events API publisher to Kafka: https://github.com/openstack/monasca-events-api/blob/master/monasca_events_api/app/common/events_publisher.py

18: Monasca IRC meeting Dec 15, 2018: http://eavesdrop.openstack.org/meetings/monasca/2018/monasca.2018-12-12-15.00.log.html

Upgrade Apache Kafka client

Tue, 23 Apr 2019 00:00:00

Include the URL of your story:

https://storyboard.openstack.org/#!/story/2003705

Currently in all Python Monasca components the copy of kafka-python library in version 0.9.5 (released on Feb 16, 2016) is used 1. This specification describes the process of upgrading the Apache Kafka client to confluent-kafka-python 2. This will improve the performance and reliability. Sticking with the old frozen client version is also unacceptable in terms of security.

Problem description

The use of KeyedProducer and SimpleConsumer in kafka-python library has been deprecated as of version 1.0.0 3. Further use of this code poses a security risk. Additionally, profiling of monasca-persister has shown that most of the time is spent during the consumption of Kafka messages 7. Thus, there is a big potential on improving overall Monasca performance by upgrading the used Kafka client.

Proposed change

The wiki page hosted by Apache Software Foundation lists available Python clients 4. There are currently three actively maintained and supported clients: confluent-kafka-python, kafka-python and pykafka. Several benchmarks have shown 5, 6 that the client maintained by Confluent is both the fastest and most complete.

There is significant performance improvement when using asynchronous producer (~50x). Sending messages asynchronously will require more care to avoid duplicating the persisted data but performance gain justifies that.

confluent-kafka-python is also the only client which offers support for Apache Avro serialization which reduces the size of messages and thus additionally speeds up communication.

The proposed change includes using:

confluent-kafka-python library
in asynchronous mode

Code changes will affect following components:

monasca-common
monasca-{log,event}-api
monasca-persister
monasca-notification
monasca-transform

Java components (monasca-thresh and monasca-persister) are out of scope of this specification. Client upgrading in these components should be handled separately.

This client has an external dependency on librdkafka, a finely tuned C client.

Alternatives

pykafka
new version of kafka-python
use synchronous mode

Data model impact

No data model impact.

REST API impact

No REST API impact.

Security impact

This change will improve the security because of removing the deprecated and unmaintained code.

Other end user impact

No end user impact.

Performance Impact

This change should dramatically improve the performance of the complete solution. In particular performance of monasca-persister and monasca-api is expected to improve.

Other deployer impact

New libraries should be packaged and deployed:

confluent-kafka-python
librdkafka

Developer impact

confluent-kafka-python has to be used instead of kafka-python in all affected components.

Implementation

Assignee(s)

Primary assignee:: witek
Other contributors:: <>

Work Items

remove code using pykafka
remove pykafka from requirements and lower-constraints
add confluent-kafka-python to global-requirements
implement common routines in monasca-common
use new code in:

monasca-{log,events}-api

monasca-persister

monasca-notification

monasca-transform

delete old deprecated code

Dependencies

New packages have to be build for:

confluent-kafka-python
librdkafka

Testing

We should test the implementation using existing integration tests (tempest). Additionally we should test the scenario when the producer fails to receive response from Kafka for some of the messages in the bulk. It should be avoided that duplicate entries are created in the database.

The implantation should be followed by executing following tests on the complete stack:

stress
endurance
performance

Documentation Impact

No documentation impact.

References

1: https://github.com/dpkp/kafka-python/releases/tag/v0.9.5
2: https://github.com/confluentinc/confluent-kafka-python
3: https://github.com/dpkp/kafka-python/blob/master/docs/changelog.rst#100-feb-15-2016
4: https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-Python
5: https://github.com/monasca/monasca-perf/blob/master/kafka_python_client_perf/monascaInvestigationKafkaPythonAPIs.md
6: http://activisiongamescience.github.io/2016/06/15/Kafka-Client-Benchmarking/
7: https://opendev.org/openstack/monasca-persister/commit/a7112fd30bd545dd850e0e267dcceb9ea27551ad

History

Revisions
Release Name	Description
Stein	Introduced

Example Spec - The title of your feature request

Tue, 23 Apr 2019 00:00:00

Include the URL of your story:

https://storyboard.openstack.org

Some notes about the monasca-spec and stories process:

Not all stories need a spec. For more information see https://docs.openstack.org/monasca-api/latest/contributor/index.html
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Monasca REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/monasca-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this feature request addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-monascaclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure Monasca that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on Monasca.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a feature where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in monasca, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Monasca (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Stein	Introduced

Merge Monasca APIs

Thu, 04 Oct 2018 00:00:00

https://storyboard.openstack.org/#!/story/2003881

Monasca currently has 3 APIs; one for metrics, one for logs and one for events. Historically, this has created additional overhead for developers upgrading or adding new features to the APIs, for system administrators configuring the APIs, and for packagers generating Monasca distributions. The purpose of this spec is to consider whether the Monasca APIs should be merged into a single, unified API.

Problem description

The Monasca APIs are written in Python and share a lot of functionality, some, but not all of which has been factored out to the Monasca common library. Here lies the first problem:

There exists significant technical debt.

There is a significant amount of common functionality that has not been factored out to Monasca common. For example, the Monasca API repo contains code for validating metrics and so does the Monasca common library. The same is true for query authorisation, where validation code lives in both repos. This technical debt could of course be addressed, but how did it come about?

Updating the APIs to support common features incurs significant developer and reviewer overhead compared to updating a single API.

The APIs are all written slightly differently. Adding support for Oslo Policy or uWSGI to one API is not the same as adding it to another API. Furthermore, changes frequently have to be made across multiple repos. Whilst Zuul is well suited to coordinating this task, it requires careful synchronisation of versions by developers and meticulous attention from reviewers. If one modifies code which is common to all three APIs, they must systematically verify that in each case the changes work as intended. Of course, automated testing can provide relief here, but it doesn’t make the burden disappear entirely.

Historically it has been difficult to maintain a standard experience across multiple APIs.

At times, APIs have diverged when in an ideal world they should not have. For example, when adding support for a common feature, work has traditionally been focused on one API to keep the task simple, with the view to adding support for the other APIs at a later date. If works stops, for example due to a release, or due to a developer moving on, the APIs can be left in a diverged state for a significant amount of time. Of course, one solution is to block merging of a common feature until the work is complete in all three APIs, and all common code is added to the Monasca common library. In practice, this has been prevented by the number of man hours available. For example, a contributor may only be interested in metrics. They may not be able to justify spending time working on the other APIs.

Packaging, deploying, and configuring 3 APIs is more complex than deploying a single, unified API.

This is an obvious point, but it can be made worse by 3). For example, historically, it has not always been possible to run all APIs in the same way. Configuration of common functionality such as Kafka has not always been uniform. There is extra build system configuration, additional keystone endpoints that need to be registered and monitoring the availability of 3 APIs is more difficult than one.

Use Cases

As a developer I would save time by implementing a feature common to all APIs in a single repo, rather than across four repos.

As a reviewer I will save time by not having to think about how a change may impact multiple repos.

As a deployer I will save time by having two fewer services to deploy and configure.

As a security analyst I can focus my efforts reviewing a single API, rather than three.

As a user of Monasca I would like a consistent experience, including usability and documentation.

As a packager I would like to have a single Monasca API, rather than three to save time configuring and maintaining my build system.

Proposed change

Merge all APIs into a single unified API. Merge all common API code from the Monasca common repo into the unified API repository. Specifically, it is proposed to merge all relevant code into the Monasca API.

Alternatives

Refactor all APIs so that the code is standard. Prevent merging of common features until the work is complete across all APIs, and all common code has been factored out into the Monasca common library. From historical experience this will be difficult without additional developers.
Don’t do anything and carry on working around the technical debt. In the long term this is likely to make it more difficult to add new features, and require more time for maintenance.

Data model impact

None

REST API impact

Aside from the fact that a single service will implement the combined schema from all APIs, the calls should not change. We should be careful when merging, for example dimension validation code, that we do not break things which were accepted in one API, but not another. An ideal result would be that we use the same code for parsing common fields such as dimensions for all API calls.

Security impact

If a single API was previously exposed publicly, deploying the unified API will increase the surface area for attack. However, in general, it will be easier to review changes to make security improvements due to the reduced developer and reviewer overhead.

During the deprecation period of the Events and Log API, security fixes made to the unified API may need to be backported.

Other end user impact

All services talking to the Monasca Events and Log APIs will need to be directed at the unified API. For services which use Keystone to lookup the Monasca endpoints this should occur automatically. Other services such as the Fluentd Monasca output plugin will need to be reconfigured manually. A grace period where the Monasca Events and Log APIs are still supported is one possibility to make this transition easier.

Performance Impact

No direct impact.

In general, it will be less effort to profile and optimise a single API than it would be to do the same for all three APIs.

In clustered deployments with specialised nodes (for example a dedicated logging node, hosting the Monasca Log API) the unified API can be deployed as a direct replacement, and the additional functionality can simply be ignored.

Other deployer impact

For Monasca users supporting legacy releases, any security or bug fixes made to the unified API may need to be backported to the individual APIs.

All actively maintained deployment solutions will need to be updated to deploy the unified API. For example, Monasca-Docker, DevStack, Kolla, OpenStack Ansible, and Helm.

In the case of DevStack we should merge the three existing plugins into one. The resulting plugin should have options like log_pipeline_enabled and metrics_pipeline_enabled to support enabling those pipelines separately. This is useful, for example, when DevStack is used in OpenStack CI to allow testing changes localised to specific areas more efficiently.

Developer impact

The motiviation behind this change is to reduce the burden placed on developers and reviewers when making improvements to the Monasca APIs. It is hoped that this will lead to an increase in developer productivity.

Implementation

Assignee(s)

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Review test coverage of APIs, and add coverage for any missing areas.
Review test coverage of Monasca common and add coverage for any missing areas.
Merge Monasca common Python code into the Monasca API. Common functionality should include:
- authorisation
- validation
- OpenStack policy
- WSGI deployment
Implement Log API schema in the Monasca API and port tests.
Implement Events API schema and port tests.
Merge DevStack plugins into a single plugin and add support for enabling pipelines individually.
Deprecate Monasca Log and Event APIs.
Merge and update documentation

Dependencies

No additional dependencies are added. The dependency on Monasca common can be removed.

Testing

The Monasca API and Log API Tempest test plugins have already been merged into one plugin. Any Tempest tests which exist for the Events API should also be merged into the unified plugin. The Tempest plugin will need to be updated to use the unified API.

Unit tests from the Log API and Events API repo will need to be ported to the Monasca API (unified) repo. Some of these tests may be redundant.

Documentation Impact

Three sets of documentation will be reduced to one. Whilst it will take some effort to merge the documentation, it should hopefully be more consistent.

References

None

History

None

Python Persister Performance Metrics Collection

Thu, 27 Sep 2018 00:00:00

Story board: https://storyboard.openstack.org/#!/story/2001576

This defines the list of measurements for the metric upsert processing time and throughput in Python Persister and provides a rest api to retrieve those measurements.

Problem description

The Java Persister, built on top of the DropWizard framework, provides a list of internal performance related metrics, e.g., the total number of metric messages that have been processed since the last service start up, the average, min and max metric processing time etc. The Python Persister, on the other hand, lacks such instrumentation. This presents a challenge to the operator who wants to monitor, triage, and tune the Persister performance and to the Persister performance testing tool that was introduced in Queens release. The Cassandra Python Persister plugin depends on this feature for performance tuning.

Use Cases

Use case 1: The developer instruments the defined performance metrics.

There are two approaches towards the internal performance metrics. The first approach is in memory metering similar to the Java implementation. The data collection starts when the Persister service starts up and is not persisted through service restart. The second approach is to treat such measurement exactly the same as the “normal” metrics Monasca collects. The advantage is that such metrics will be persisted and rest apis are already available to retrieve the metrics. The list of Persister metrics includes:
1. Total number of metrics upsert request received and completed on a given Persister service instance in the given period of time
2. Total number of metrics upsert request received and completed on a process or thread in a given period of time (P2)
3. The average, min, max metric request processing time in a given period of time for a given Persister service instance and process/thread.
Use case 2: Retrieves persister performance metrics through rest api.

The performance metrics can be retrieved using the list metrics api in the Monasca API service.

Proposed change

Monasca Persister
- Python Persister integrates with monasca-statsd to send count and timer metrics
- Persister conf to add properties for statsd
Persister performance benchmark tool adds support to retrieve the metrics from Monasca rest api source in addition to the DropWizard admin api.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Other end user impact

None

Performance Impact

TBD, The statsd call to update counter and timer is expected to have small performance impact.

Other deployer impact

No change in deployment of the services.

Developer impact

None.

Implementation

Assignee(s)

Contributors are welcome!

Primary assignee:

Other contributors:

Work Items

Monasca Persister
- Python Persister integrates with monasca-statsd to send count and timer metrics
- Persister conf to add properties for statsd
Persister performance benchmark tool adds support to retrieve the metrics from Monasca rest api source in addition to the DropWizard admin api.

Dependencies

None

Testing

Set up a system, use JQuery to automate storing many metrics, check results. The tools to accomplish this testing can be found in monasca-persister/perf/

Documentation Impact

The existing README.md in monasca-persister/perf describes the needed steps. Some minor changes may need to be made to stay current.

References

https://github.com/openstack/monasca-persister/tree/master/perf

History

Revisions
Rocky	Introduced
Stein	Revised with testing notes

Stein Project Priorities

Mon, 24 Sep 2018 00:00:00

List of priorities the Monasca drivers team is prioritizing in Stein.

The owners listed are responsible for tracking the status of that work and helping get that work done. They are not the only contributors to this work, and not necessarily doing most of the coding!

The implementation progress on these priorities and other identified important tasks is tracked in this board.

Essential Priorities

Title	Owners
Kafka client upgrade	witek
Monasca Events Agent	joadavis, aagate
Merge Monasca APIs	dougsz
Add query endpoint for logs/events	dougsz
Run under Python 3 by default	adriancz, Dobroslaw
Pre upgrade checks	joadavis

High Priorities

Title	Owners
Auto-scaling with Heat	witek
Metrics retention policy	joadavis
Documentation refresh
Deployment in OpenStack Helm	srwilkers
Integration with Watcher	yushiro

Details

Kafka client upgrade

Merge Monasca APIs

The goal is to merge all Monasca APIs into a single unified API to reduce maintenance overhead, make it easier for developers to add new features and improve the user experience.

Monasca Events Agent

The goal is to extend Monasca Ceilometer project and add a new events publisher which will publish Openstack notifications (or events) to Monasca Events API.

Add query endpoint for logs/events

Add support for querying ElasticSearch via the Monasca API to support tenant scoped access to logs and events. This should include accessing the logs via Grafana.

Run under Python 3 by default

As OpenStack Technical Committee agreed in the Python2 Deprecation Timeline resolution, the next phase of our adoption of Python 3 is to begin running all jobs using Python 3 by default and only using Python 2 to test operating under Python 2 (via unit, functional, or integration tests). This goal describes the activities needed to move us to this python 3 first state.

Pre upgrade checks

The goal is to provide an upgrade check command which would perform any upgrade validation that can be automated.

Metrics retention policy

The goal is to add a new API for managing the mapping of metrics to TTL values.

Rocky Project Priorities

Wed, 13 Jun 2018 00:00:00

List of priorities the Monasca drivers team is prioritizing in Rocky.

The owners listed are responsible for tracking the status of that work and helping get that work done. They are not the only contributors to this work, and not necessarily doing most of the coding!

Essential Priorities

Title	Owners
Kafka upgrade	witek
Alembic migrations	jgr
Metrics retention policy	jgu
Monasca Transformer refresh	aagate, joadavis
Run API under WSGi	witek
Support Python 3.5	witek, sc
Enable mutable configuration
Policy in Code	amofakhar

High Priorities

Title	Owners
Templating webhook notifications	dougsz
Service Domain for Self Service Agent Users	jgr
Add Monasca publisher to Ceilometer	joadavis, aagate
Alarm grouping, silencing, inhibition	witek
Documentation refresh

Optional Priorities

Title	Owners
New agent plugins for OpenStack
Cross-project integrations
Monasca Query Language
Create Docker images from OpenStack repos
Kolla deployment	dougsz
Query logs pipeline	dougsz
New monasca-thresh
Monasca-persister performance improvements	sgrasley, jgu

Details

Kafka upgrade

The goal is to upgrade all Monasca components to use Apache Kafka 1.0.x. Currently used embedded forked version of kafka-python client should be replaced with pykafka (or alternatively confluent-kafka-python). The integration should be preceded by extensive performance and endurance testing.

Alembic migrations

The goal is to provide a consistent and easy to use way to maintain SQL schema changes. The implementation should allow schema initialization and migration from one version to another. Alembic is a lightweight database migration tool which optimally fulfills our requirements.

Run API under WSGi

This is a community-wide release goal for Pike. The goal is to:

Provide WSGI application script files.
Switch devstack jobs to deploy Monasca APIs under uwsgi with Apache acting as a front end proxy.

Support Python 3.5

This is a community-wide release goal for Pike. The goal is to support, test and running with Python 3.5.

Policy in Code

The goal is to register and document default policies for the APIs in code.

Add Monasca publisher to Ceilometer

Monasca-Ceilometer (aka. Ceilosca) code currently exists in its own project. This is for historical reasons. With changes in Ceilometer and the Telemetry project, it may be possible to have the Monasca publisher from monsasca-ceilometer merged into the Ceilometer repository. This could reduce future workload in maintenance.

Templating webhook notifications

Improve the quality of notifications generated from alerts. We want notifications to be informative, concise and flexible.

Kolla deployment

Add support for deploying Monasca in Docker containers using the OpenStack Kolla project. This change will support deploying Monasca in a high availability configuration. Blueprints exist for containers and the Ansible roles to deploy them.

Query logs pipeline

Add support for querying ElasticSearch via the Monasca Log API to support tenant scoped access to logs. This should include accessing the logs via Grafana.

Introduce Database Migrations

Mon, 12 Mar 2018 00:00:00

https://storyboard.openstack.org/#!/story/2001654

Most OpenStack services use SQLAlchemy Migrate or Alembic to provide migration scripts that (a) update the service’s database schema if it changes and (b) migrate data as required. Currently, Monasca does not follow this pattern. Instead, it provides simple SQL scripts for PostgreSQL or MySQL databases that are only useful for initializing a database schema. For any subsequent changes required by updated code operators are on their own, i.e. they need to apply these manually to keep Monasca working. This spec proposes introducing (a) database migrations and (b) a separate tool that analyses an existing database’s schema state and adds the necessary metadata to update it through migrations in the future.

Problem description

In a nutshell, the database schema used for holding various alarm, notification, and metric metadata is not updateable. This is due to the source of truth for the schema being a simple SQL script that is only useful for schema initialization on a blank database.

The group most affected by this is operators, who are faced with several unattractive choices when a code change requires a database schema change:

They can drop their whole Monasca database and recreate it from scratch with the updated SQL script. That way they will lose all of their alarm definitions along with notification settings, their complete alarm history, and some metrics metadata.
They can manually update the database and migrate data (in the case of columns/tables getting renamed). This is error prone and carries a risk of data loss.
They can decide to live without that code change. If the change breaks on an outdated database they may even have to create and maintain code patches for that.

Besides operators, the lack of database migration infrastructure is a significant obstacle to Monasca developers as well: it turns any data model change into a breaking change due to the schema update that requires. Thus the current state of affairs prevents new features that require database changes or renders them extremely unpopular at the very least.

Use Cases

Operators will be able to update Monasca without risking breakage due to database changes.
Developers can modify the data model without breaking existing installations.

Proposed change

The proposal aims to improve the current state of affairs by adding Alembic based command line tooling for the following operations:

A legacy database migration command line tool. This command line tool will detect which revision of the database schema SQL script was used based on the tables and columns currently in the database. It will then initialize the database’s migration meta data to allow for regular database migration from that point forward.
A database migration command line tool with multiple base revisions to cover the following scenarios:
1. An uninitialized database
2. A schema state resulting from any of the existing SQL script
  revisions.

Both (1) and (2) may be implemented as part of the same tool.

A database schema based on third party SQL scripts (which may be found in various configuration management tools used for deploying Monasca) is not supported.

Alternatives

This change is long overdue and best practice throughout OpenStack. That being said, it would be possible to considerably reduce its scope by omitting the heuristics for detecting an existing legacy database schema. That way operators would be forced to drop the database and recreate it using migrations. If possible we should not impose this on operators.

It would also be possible to use plain SQLAlchemy Migrate rather than Alembic to implement migrations, but OpenStack appears to have standardized on Alembic by now (see https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy for a discussion of why that happened).

Data model impact

There will be no impact on the data model itself. There will only be changes to how the data model is synchronized to the database’s run time schema.

REST API impact

There is no REST API impact.

Security impact

The only slightly security related aspect of this change is providing the database migration command line tools with access credentials for the data base. The accepted best practice for this is running it as root on the machine where the API service (in our case monasca-api) is also running and retrieve these credentials from the API service’s configuration.

Other end user impact

N/A

Performance Impact

N/A

Other deployer impact

Configuration management that uses the legacy SQL scripts must be changed to use the new migration tools. Among other things, the Monasca devstack plugin must be modified to use them. Users updating to a Monasca version with this feature may need to run a special migration to get versioning metadata added to their database schema.

Developer impact

Any developer making data model changes after this feature is implemented will have to write a migration for updating the database schema accordingly.

Implementation

Assignee(s)

Primary assignee:: <jgr-launchpad>

Tasks

Add command line tool(s) for regular migration (i.e. migration of a versioned database) and transition of an unversioned database to a versioned one.
Create migration chains from:
- Empty database
- All revisions of the schema file in the repository.

Dependencies

N/A

Testing

Any meaningful testing can only take place in a full Monasca deployment with an actual database to test against. In particular, testing attention should focus on testing this with all revisions of the legacy SQL script in devstack/files/schema/mon_mysql.sql. This testing can take place at an early stage during Devstack setup as follows:

Each SQL script revision is checked out from the git repository, applied to the database and converted to a “migratable” database by running the conversion operation on the database. After each iteration, the database is dropped to prepare for the next revision.
As a final step, the initial migration for a blank database is performed and Devstack proceeds normally.

Since PostgreSQL is deprecated in OpenStack it will be sufficient to implement testing with the MySQL flavoured SQL script.

Documentation Impact

This feature needs to be documented in operator/deployer documentation to ensure operators use migrations rather than legacy SQL scripts.

References

Etherpad from Rocky PTG where this was discussed: https://etherpad.openstack.org/p/monasca-ptg-rocky

Metric Retention Policy

Thu, 22 Feb 2018 00:00:00

Story board: https://storyboard.openstack.org/#!/story/2001576

Metric retention policy must be in place to avoid disk being filled up. Retention period should be adjustable for different types of metrics, e.g., monitoring vs. metering or aggregate vs. raw meters.

Problem description

In a cloud of 200 compute hosts, there can be up to one billion metrics generated daily. The time series database disks will be filled up in months if not weeks if old metric data is not purged regularly. The retention requirement can be different based on the type of the metrics and the usage model. For example, the customer may want to preserve the metering metrics for months or years, while s/he has no interest in more than a week old monitoring metrics. Some customers’ billing system may pull the metering data on a daily base which could eliminate the need of longer retention of metering metrics. Monasca needs to support metric retention policy that can be tailored per metric or metric type.

Use Cases

Use case 1: Installer sets a default TTL value in configuration. At installation time, a default TTL (time to live) value is specified in the configuration for monasca-api and is used as the default retention policy.

The default retention policy is applied if a metric doesn’t match another retention policy. This default retention is generally a shorter period of time and may be used for the common monitoring metrics.
Use case 2: Installer loads a set of metric to TTL mappings (retention policies), which is stored in the Monasca API data store (mysql database). These mappings may be provided in a JSON structure. This is intended to be useful for bootstrap or restore from backup.
Use case 3: Monasca API receives new metric (regardless of source). Metric is mapped to a dictionary to determine TTL (or default value used if no match). TTL is passed with metric value on to the Persister for storage in TSDB.

Note that the use cases for monasca-agent to post metrics are unchanged, just the processing at Monasca API then the API to Persister message.]

The Monasca Persister then stores the metric and specifies the TTL to the TSDB configured (i.e. InfluxDB or Cassandra).
Use case 4: Operator uses Monasca CLI to specify (or modify) a TTL value for a metric match string. Match string could be specific, such as “cpu.user_perc” or a wildcard string, such as “image.*”. CLI posts request to Monasca TTL API, where it is validated then stored in database.
Use case 5: Operator uses Monasca CLI to GET the dictionary of metric:TTL mappings. This can be used to export the list for backup or verification.
Use case 6 (optional): Operator uses Monasca UI to accomplish use case 4 or 5

Proposed change

Monasca API: Add a new API for managing the mapping of metrics to TTL values. See the REST API impact section below.

Add storage for the mapping in the MySQL database. This is to allow all instances of Monasca API to share the configuration dynamically. TBD - Create a schema for storing the metric:TTL dictionary.

A policy precedence needs to be defined. It is possible that more than one retention policy may apply to a given meter, so a clear precedence needs to be defined to determine which TTL value to apply. TBD - a few concrete examples.
Monasca Persister: Persister reads the default retention policy setting from the service configuration file in the influxDbConfiguration and cassandraDbConfiguration section.
```
# Retention policy may be left blank to indicate default policy.
# Unit is days
retentionPolicy: 7
```
It may be convenient to allow specifying a unit with the policy value. For example “7d” for 7 days or “3m” for 3 months.

It will retrieve the TTL property in the incoming metric message. If not set, the TTL value from the default retention policy will used instead.

It is expected with the addition of this Metrics Retention feature that the default retentionPolicy value would be set to a low value, and that metrics that are to be kept longer would be called out specifically through the Retention API and appropriate values set.

The TTL is set in the parameterized database query when persisting the metrics into the time series database, including both Cassandra and InfluxDB. TBD - exact call structures for each TSDB.

Note that this does mean that each storage back end would need to have code customized in the persister to support passing the TTL value. This may also be possible for ElasticSearch, though that is not part of this initial spec.
Monasca CLI (optional): A new CLI feature could be created to simplify getting the list of TTL mappings or posting an update to a TTL mapping. This would need Keystone authentication, and would use the existing ‘monasca’ CLI authentication.
Monasca UI (optional): A new feature could be added to the Monasca UI that would allow a Cloud Operator to view and edit the list of TTL mappings. Bonus points for allowing the UI to have sample metrics and simulate the mapping on the page.

Alternatives

The original proposal was to have monasca-transform, monasca-ceilometer, and monasca-agent each keep a TTL default setting and have a property to allow specifying a TTL per metric. This would have also required a change to the Monasca API to add an optional TTL to the metric POST listener.

While this would have been simpler to implement in the Monasca API, the additional work to change all the services that originate metrics made this alternative not as appealing.

Another alternative would be to implement a new Monasca Retention API as outlined, but not include dimensions for the metrics. This would allow a much simpler data structure of key:value pairs, with the key being the unique match string and the value the standardized TTL value. While the implementation would be much simpler, it is felt that the additional power of having match dimensions would be beneficial.

Data model impact

The Monasca API data model will need to be extended to store the metric to TTL mappings (retention policies). TBD - schema

REST API impact

A new metric retention API endpoint would be added to Monasca API.

URL: /v2.0/metrics-retention

Method: GET

A GET request will return the current list of metric retention policies. Examples:

Empty list (default retention used for all metrics)
[]

Simple list
[
  {
    match: "cpu.user_perc",
    dimensions: {"host": "node1"},
    retentionPolicy: "7d"
  },
  {
    match: "cpu.stolen_perc",
    dimensions: {},
    retentionPolicy: "7d"
  }
]

Method: PUT

The PUT method is used for all create/update/delete methods on the metric retention policy list. Any list of metrics PUT to the API will be merged with the existing list. Single entries will also be supported.

JSON structure for PUT/GET to Retention API:

{
  match: "cpu.user_perc",
  dimensions: {},
  retentionPolicy: "7d"
}

TBD: do we support adding a character for time unit? Will it be confusing to PUT “1d” and GET back “86400”?

Special case: to delete a retention policy, give a retentionPolicy value of None and it will be removed from the list.

{
  match: "cpu.user_time",
  dimensions: {},
  retentionPolicy: None
}

Additionally, a list of retention policy items may be PUT, with the format matching the response from GET. Each item in the list will be compared to existing metric policies (match string and dimensions). If an exact match is found, the retentionPolicy value will be replaced. Otherwise, the new item is added to the list. (This is intended to make bootstrap or restore from backup easier)

The communication from Monasca API to Persister would have the TTL value added as a parameter.

NOTE: Care should be taken in defining the REST API path, as Gnocchi uses “/metric”, which may be confusing to some users.

Security impact

None. Security measures already in place for the Monasca API would remain.

Other end user impact

None for most users, as access to the Monasca Metrics API is restricted to Cloud Operators. A Cloud Operator would have a new responsibility to configure retention for the metrics.

A future discussion could be had about whether a tenant user should be granted the ability to set their own retention policies, but generally the Cloud Operator is responsible for ensuring there are sufficient resources to meet the retention requirements.

Performance Impact

This feature has no direct impact on the write throughput. However, it allows the user to enable shorter retention period for monitoring metrics which can potentially improve the read performance for the queries that involves search, grouping and filtering when there are less metrics in the table. This improves the storage footprint.

Depending on how complex the metric retention match string gets there could be some performance impact. TBD

Other deployer impact

No change in deployment of the services. The service could be deployed with simply a default TTL value in configuration. If the operator desires, at install time a complete list of TTL values could be loaded as part of the installation process once the Monasca API is running.

For planning, the user now has the option to specify a shorter retention period for monitoring metrics or even per metric or metric category. The disk size should be calculated based upon the retention policy accordingly.

Developer impact

Monasca agent plugin developers should be aware of the new TTL property now available to them. It is an optional property that is only needed if a different TTL value than the default retention policy in the Persister service is needed.

Implementation

Assignee(s)

Contributors are welcome!

Primary assignee:

Other contributors:

Work Items

Add new metrics-retention API endpoint to Monasca API
Add code to match all incoming metrics to the Monasca API with the appropriate retention policy (or default)
Add TTL in seconds as a parameter to the request from Monasca API to Persister
Create a CLI
- PUT of updated retention policy(ies)
- GET of the list
Determine correct precedence for retention policies that overlap, and clearly document with examples.

Dependencies

Dependent on retention policy support in the TSDB storage. Both Cassandra and InfluxDB support specifying a retention policy.

Testing

Unit testing

Unit tests in the Monasca API should be written for the scenarios of defining a TTL for each metric.

Metric received, no matching retention policy found, default policy used
Metric received, one exact matching metric retention policy found, matching policy parameter passed to Persister call
Metric received, more than one matching policy, correct precedent determined and appropriate policy parameter passed to Persister call

Monasca Persister will also need unit tests to verify the passed-in value is passed on to the TSDB retention method call, and to handle the case of a missing TTL parameter. We may decide that the TTL parameter is optional then a global default TTL value should be used.

Functional testing

Functional testing is more involved, as one way to test would be to trigger some metrics, have them stored in the TSDB, then wait for the TTL value to expire and verify the metric is removed correctly. More thought and definition is needed to define what is appropriate and possible (i.e. to not retest features of the TSDB).

Documentation Impact

Operators who use Monasca would need documentation to describe the format of the new API and recommended usage. This may include guidelines on how to set a low default and to choose which metrics should be kept longer. The default TTL value as set in a config file should also be documented.

References

Links
- Stein PTG discussion - https://etherpad.openstack.org/p/monasca-ptg-stein
Glossary
- TTL - short for Time to Live, a setting in TSDB that defines when an item (in this case a metric) will be cleaned out.
- TSDB - Time Series Database, such as InfluxDB or Cassandra.

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Queens	Introduced

Queens Project Priorities

Wed, 03 Jan 2018 00:00:00

List of priorities the Monasca drivers team is prioritizing in Queens.

The owners listed are responsible for tracking the status of that work and helping get that work done. They are not the only contributors to this work, and not necessary doing most of the coding!

Essential Priorities

Title	Owners
Cassandra support	jgu
Add Monasca publisher to Ceilometer	joadavis, aagate
Fix Keystone authentication for Grafana	rhochmuth, witek, Dobroslaw
Alarm grouping, silencing, inhibition	Andrea Adams, rhochmuth
Run API under WSGi (Community Goal)	kornicameister, witek
Support Python 3.5 (Community Goal)	witek, sc
Split Tempest Plugins (Community Goal)	chandankumar

High Priorities

Title	Owners
Service Domain for Self Service Agent Users	jgr
Replace python-kafka with pykafka
Metrics retention policy
Persisting Events	witek
Monasca Query Language
Policy in Code (Community Goal)	jgr

Optional Priorities

Title	Owners
3-nodes cluster with Docker Compose	witek
Add message attributes to Log API	koji

Details

Add Monasca publisher to Ceilometer

Monasca-Ceilometer (aka. Ceilosca) code currently exists in its own project. This is for historical reasons. With changes in Ceilometer and the Telemetry project, it may be possible to have the Monasca publisher from monsasca-ceilometer merged in to the Ceilometer repository. This could reduce future workload in maintenance.

Fix Keystone authentication for Grafana

The current implementation of Keystone authentication for Grafana is maintained in the forked repository. Due to upstream changes in Grafana major refactoring is required to rebase the fork with newest Grafana code.

The goal is to contribute Keystone authentication (or generic pluggable authentication mechanism) to Grafana upstream. If not possible, the current fork should be refactored to allow its further maintenance.

Run API under WSGi (Community Goal)

This is a community-wide release goal for Pike. The goal is to support, and test, running WSGI.

Support Python 3.5 (Community Goal)

This is a community-wide release goal for Pike. The goal is to support, and test, running with python 3.5.

Split Tempest Plugins (Community Goal)

This goal is to make sure we always use a separate python project for monasca-api, monasca-log-api and monasca-events-api tempest plugins.

Policy in Code (Community Goal)

The goal is to register and document default policies for the APIs in code.

Persisting Events

The goal is to provide the pipeline for persisting OpenStack notifications and/or events from external systems to the database, e.g. Elasticsearch.

3-nodes cluster with Docker Compose

The goal is to provide an easy and simple way of deploying Monasca in a static 3-nodes cluster with Docker containers without using cluster management layer like Kubernetes or Docker Swarm.

Service Domain for Self Service Agent Users

Fri, 27 Oct 2017 00:00:00

https://storyboard.openstack.org/#!/story/2001214

In order to send metrics and logs to Monasca the Monasca agent currently needs a Keystone user with a special Keystone role, usually monasca-agent. This is fine for infrastructure monitoring of an OpenStack cloud where the person interested in monitoring can usually create user accounts, too, and where these accounts’ credentials are stored on the cloud’s compute nodes and controllers which should be well protected against breaches. Storing such credentials on public facing instances to be monitored by Monasca is a problem because these instances (a) tend to be more exposed and (b) because an OpenStack user creating instances usually cannot create special purpose user accounts. This spec proposes a solution to these two problems.

Problem description

Currently there are two ways to submit metrics and logs for a given OpenStack project:

Cross tenant submission: this requires a user with a special role that can submit metrics or logs on behalf of any arbitrary project. This role is currently used by compute nodes to submit libvirt metrics for the projects their running instances reside in. These roles are controlled by the security/delegate_authorized_roles setting for monasca-api and the roles_middleware/delegate_roles setting for monasca-log-api.
Metric submission by a user in the project: this is the normal way. Any user with the designated Monasca agent roles (monasca-agent by default) in a project can submit metrics and logs for this project. These roles are controlled by the security/agent_authorized_roles setting for monasca-api and the roles_middleware/agent_roles setting for monasca-log-api.

Both options are bad from a security point of view. (1) is worse, because it will allow submission of arbitrary bogus log entries/metric events for arbitray projects. Because of this high abuse potential, and because it is currently only implemented in monasca-agent’s libvirt monitoring plugin it is unlikely to be employed for instance monitoring, though. (2) comes with a slightly lower but still problematic security risk:

If a user wants to monitor their instances, they need to pass the Keystone credentials of an OpenStack user with the monasca-agent role into their instances. Like any OpenStack user with any role in a project, this user can access arbitrary OpenStack APIs and create/delete resources at will. While it would be possible to add global deny rules for the monasca-agent role to every other OpenStack service’s policy.json, this is unlikely to be implemented in practice. Consequently, the compromise of an instance and its monasca-agent complication will usually leave an attacker with unrestricted out-of-band access to its creating user’s Keystone project. This allows it to typically allows it to, for example

Create, view and delete Nova instances
Create, view and delete Neutron networks
Create, view and delete Cinder volumes
Create, view and delete Glance images

In addition to this security problem there is a usability issue as well. A regular OpenStack user can neither create Keystone users, nor can they assign the monasca-agent role to these users. Consequently any user requiring Monasca monitoring for their instance needs to ask somebody with admin privileges to create a user and assign the monasca-agent role to that user. Consequently, self-service instance monitoring is not possible.

Use Cases

The change proposed by this spec will improve the situation outlined above as follows:

End Users will be able to acquire access credentials for their instances’ metrics and log agents in a self-service manner.
End Users’ attack surface from compromised instances with Monasca agents will be reduced to submission of bogus metrics/logs for their project.

Proposed change

This change takes inspiration from OpenStack Magnum, particularily the fix for CVE-2016-7404[0]. Before this fix, Magnum would create Keystone trustee users in a separate domain, with one or more of a cluster owner’s roles delegated via Keystone trusts. These user accounts would only get used for submitting certificate sign requests to the Magnum API in most cases so they had similarly generous permissions to the monasca agent users. The fix for CVE-2016-7404 reduced the use of trusts to only the scenarios where they were really needed: in the default case these users do not get any trusts delegated, nor do they have roles assigned, rendering them useless for most OpenStack APIs. The Magnum API’s policy rules for certificate:create and certificate:sign are the sole exception from this rule: they allow access if the user exists in the Magnum domain and the user’s ID matches the recorded user ID for a given cluster.

For Monasca, this spec proposes an analogous Keystone domain for agent users, just like Magnum’s trustee user domain. Likewise, the Monasca API service would get access to an admin account for this domain so it can create users inside the domain. The final puzzle piece are two extensions to the Monasca metrics and log APIs:

A monasca-api endpoint that allows end users to get these special agent users created for their project and to retrieve their credentials.
A modification to log and metrics submission endpoints for monasca-api and monasca-log-api that allows submission for the project associated with the agent user in question.

In the remainder of this section you will find a detailed description of how this change affects various parts of monasca-api.

Alternatives

Some things could be implemented in a different manner from the approach outlined below:

It would be possible to substitute the rather heavyweight Keystone domain for the agent users by a lightweight, automatically generated API key. This would make for leaner credentials, but it would allow authorization for monasca-api/monasca-log-api submission entirely without Keystone. This is less of a technical and more of a political issue. Also, monasca-client would need additional, homegrown code for authenticating with this API key which may introduce additional security bugs. On the whole this is probably a bad idea (while discussing this on IRC people were in favour of using Keystone as well).
Instead of recording project association and permissions for agent users in the database one could encode it in the agent users’ user names. This would be less than elegant, though. On the other hand, we would not need to create database tables/add database client code to monasca-log-api.
The current approach has monasca-api handling creation/deletion of agent users for both metrics and log submission. It would be conceivable to implement independent user creation for both APIs, but this would add considerable implementation overhead for no little benefit.

Data model impact

We will need to introduce a new table agent_users with the following fields:

id (string): the agent user’s keystone user UUID. Unique primary key.
creator_id (string): the creating user’s keystone user UUID
project_id (string): the project the user can submit logs metrics for
submit_metrics (boolean): optional flag specified upon creation. Defaults
to True if unspecified. Controls whether the user is allowed to submit metrics
submit_logs (boolean): optional flag specified upon Creation. Defaults to
True if unspecified. Controls whether the user is allowed to submit logs.

For this to work, monasca-log-api will need a database client implementation and the configuration options to go with that, of course. In order to reduce code duplication, as much database handling code from monasca-api as possible will be moved to monasca-common from where both monasca-api and monasca-log-api can use it.

REST API impact

The REST API needs to be modified in 3 places:

There needs to be a facility for self-service agent user creation
monasca-api needs to grant or deny metric submission based on the project an agent user is associated with and whether it has its submit_metrics flag set to True.
monasca-api needs to grant or deny log submission based on the project an agent user is associated with and whether it has its submit_metrics flag set to True.

In the remainder of this section these API changes are described in detail.

Agent User Handling

To be able to create, delete and list agent users, and retrieve agent users’ credentials this spec proposes the following extensions to monasca-api:

POST /v2.0/agent_users

This request creates agent users. The request body must follow the following JSON schema:

{
type: "map",
required: "true",
"mapping":  {
  "password": { "type": "int", "required": false },
  "submit_metrics": { "type": "boolean", "required": false },
  "submit_logs": { "type": "boolean", "required": false }
  }
}

The parameters behave as follows:

password: if this is set, the provided password will be used as the agent
user’s password. Otherwise, a randomly generated 40 character string will be used.
submit_metrics: if this is set to False, this agent user will not be
allowed to submit metrics to monasca-api. This parameter is optional. If it is omitted, the default is True and the agent user will be allowed to send metrics.
submit_logs: if this is set to False, this agent user will not be
allowed to send logs to monasca-log-api. This parameter is optional. If it is omitted, the default is True and the agent user will be allowed to send logs to monasca-log-api.

This request will

Return 200 with a JSON formatted database record for the agent user in the body if the request is successful. In addition to the database record the response will contain a password field with the newly created user’s password. This password will not be recorded in the database.
Return 500 with an error message in the body if user creation fails.
Return 401 for unauthenticated users or users without any roles.

GET /v2.0/agent_users

This request lists agent users. It will

Return 200 OK and a JSON formatted list of agent user database records for the requester’s project. For requesters with the admin role, agent users from all projects will be listed. The list may be empty.
Return 401 for unauthenticated users or users without any roles.

GET /v2.0/agent_users/<id>

This request retrieves the database record for an agent user. It will

Return 200 with a JSON formatted record for the agent user identified by the Keystone user ID <id> in the body, provided there is a record for <id> in the database and the requester is allowed to access it. The requester is allowed to access a record if the requester’s project matches project_id or the requester fullfills the oslo.policy is_admin criterion.
Return 404 if there is no record for the user or the requester is not allowed to access it.
Return 401 for unauthenticated users or users without any roles.

Extended Policy Check for Metric Submission

The request

POST /v2.0/metrics

will need to be extended to not only check for the roles configured in security/agent_authorized_roles but it will also need to verify if

The requesting user is in the designated agent user domain
If so, look the user up in the agent_users table and submit the metrics being sent for the agent user’s recorded project.
If the lookup fails for some reason (e.g. for a manually created user in the agent user domain that does not have a record in the database), the request is treated as unauthorized.

Extended Policy Check for Log Submission

The following requests

POST /v3.0/logs

…will need to be extended to not only check for the roles configured in roles_middleware/delegate_roles but it will also need to verify if

The requesting user is in the designated agent user domain
If so, look the user up in the agent_users table and submit the logs being sent for the agent user’s recorded project.
If the lookup fails for some reason (e.g. for a manually created user in the agent user domain that does not have an associated domain), the request is treated as unauthorized.

Client impact

python-monascaclient will need to be extended to implement the API operations described in the Agent User Handling section above.

monasca-agent will not need to be modified.

Configuration changes

monasca-api will need configuration settings to provide it with admin settings for the agent user domain.

monasca-log-api will need configuration settings to provide it with access to the monasca-api database.

Both monasca-api and monasca-log-api will need configuration that allows an operator to disable this feature as desired. Since it allows users to create monitoring users, it should be disabled by default. Optionally, it should be possible to configure a Keystone role required to create agent users. If this role is left unspecified, any user should be able to create agent users)

Security impact

This feature introduces a mechanism that allows regular users to create unprivileged special purpose user accounts in a dedicated Keystone domain. This might not be desirable for every operator, hence the provisions for disabling or restricting it in the previous section.

The special purpose users in question do not have Keystone roles and are therefore unusable for most purposes. There is a precedent for this in Heat and Magnum. The former creates such users as targets for a Keystone trust delegated by a Heat stack’s creating user. The latter creates such users as of the fix for CVE-2016-7404[0] and uses them in the exact same manner as the one proposed by this spec.

Other end user impact

Users will be able to create dedicated monitoring/logging users in a self-service manner, which is an improvement over the current situation (they need to ask somebody with admin privileges to create users with the special monasca-agent role).

Performance Impact

This change may add a small performance penalty due to adding a database lookup for every metrics submission attempt. This shouldn’t be too bad, though, because every metrics submission attempt already requires Keystone token validation, beside which one database lookup is pretty small. Nonetheless, this feature can be disabled until a fix is found if it turns out to cause major performance issues.

Other deployer impact

N/A

Developer impact

N/A

Implementation

Assignee(s)

Primary assignee:: jgr-launchpad

Work Items

Add common database code to monasca-common.
Modify monasca-api to use database code from monasca-common and remove its own database code.
Implement agent user CRUD operations in monasca-api
Extend monasca-api policy enforcement code to authorize agent users
Extend monasca-log-api policy enforcement code to authorize agent users

Dependencies

Testing

Documentation Impact

The self service creation of agent users will need to be documented.

The various newly introduced configuration settings will need to be documented.

Creating a domain for agent users will need to be documented.

References

[0] https://github.com/openstack/magnum/commit/2d4e617a529ea12ab5330f12631f44172a623a14

History

Revisions
Release Name	Description
Queens	Introduced

Use oslo.policy for Monasca APIs

Wed, 11 Oct 2017 00:00:00

https://storyboard.openstack.org/#!/story/2001233

Presently, neither monasca-api nor monasca-log-api use oslo.policy. Instead, they contain their own policy enforcement code. Without oslo.policy they will not be able to implement the policy in code community goal for Queens[0], which requires the use of oslo mechanisms for defining and enforcing the policy.

Problem description

Since neither monasca-api nor monasca-log-api use oslo.policy for policy enforcement, they cannot describe their policy in code using oslo.policy mechanisms as mandated by the community goal[0].

Use Cases

The change proposed by this spec will improve the situation outlined above as follows:

Policy will be enforced in the OpenStack wide standard manner by oslo.policy. This reduces the maintenance burden on Monasca developers because they will no longer need to maintain Monasca’s own policy enforcement code.
All default policy rules will be available in a standard format to all interested parties, which fulfils the community goal.
Operators will be able to override the Monasca API and Monasca Log API default policies using policy.json files and run command line tools to generate a full policy.json for either service (e.g. for auditing purposes).
The ability to use new tooling in oslo.policy that let’s developers deprecate or change policy defaults in a way operators can consume.

Proposed change

This spec proposes the following changes to monasca-common, monasca-api, monasca-log-api and the upcoming monasca-events-api:

Implement a policy enforcement engine in monasca_common.policy. We can probably model this on Nova’s policy enforcement engine[1]. We will have to modify it somewhat to account for the fact that we have multiple APIs that use the same policy engine.
Define a module that contains the default policy rules in both monasca-api and monasca-log-api and exposes them to the enforcement engine (in monasca-events-api there is such a module already). Nova’s approach of having a list_rules() method[2] should work just fine for us. We can either copy Nova’s approach of aggregating individual endpoints’ policies in that central module[3] or just have them in there directly.
Modify existing policy enforcement code in monasca-api, monasca-log-api and monasca-events-api by code to use the enforcement engine from monasca-common.
Add monasca-api-policy and monasca-log-api-policy command line entry points that allow the user to generate a policy.json file for both APIs.

Alternatives

None, since this is a community goal. One thing that could be done differently is the policy enforcement engine: it would be conceivable to have independent enforcer implementations in both monasca-api and monasca-log-api. This would needlessly violate the DRY principle, though.

Data model impact

This change does not impact the data model.

REST API impact

This change must be implemented in a way that creates no discernible impact for the API’s consumers.

Client impact

N/A

Configuration changes

This change will continue to use the same monasca-api and monasca-log-api configuration settings we already use for policy enforcement:

agent_authorized_roles, default_authorized_roles, delegate_authorized_roles and read_only_authorized_roles for monasca-api
default_roles, agent_roles and delegate_roles for monasca-log-api

The only difference is that a different implementation (monasca-common.policy) will use them now.

Additionally, the policy enforcer will allow operators to create a policy.json file for each API service that overrides the in-code defaults.

Security impact

This change introduces a way for operators to influence monasca-api and monasca-log-api policy through configuration files. If they misconfigure policy that way, they may allow unauthorized users to send or retrieve metrics and logs.

Other end user impact

N/A

Performance Impact

N/A

Other deployer impact

N/A

Developer impact

Developers that introduce new API operations will need to register policy rules for these endpoints once this feature is implemented.

Implementation

Assignee(s)

Primary assignee:: amofakhar

Work Items

Add policy enforcement module to monasca-common. It might be a good idea to have an extension mechanism for the policy’s target analogous to this one[1] in Magnum to account for the different role variables from the configuration file needed by monasca-api and monasca-log-api, respectively. That way a generic policy enforcer can be made flexible to the point where all the role differences between monasca-api and monasca-log-api can be expressed in terms of policy rules.
Retire policy enforcement code in monasca-events-api in favour of the policy enforcement module in monasca-common.
Add policy registration code to monasca-api.
Use policy enforcement module in monasca_api.v2.reference.helpers.validate_authorization()
Add policy registration code to monasca-log-api
Remove policy enforcement code from monasca_log_api.middleware.role_middleware. (for validating agent role) and monasca_log_api.app.base.request and replace it by using monasca-common.policy from either monasca_log_api.middleware.role_middleware. or monasca_log_api.app.base_request (to have centralized enforcement in one spot).
Add monasca-api-policy console script to monasca-api
Add monasca-log-api-policy console script to monasca-log-api

Dependencies

N/A

Testing

Existing policy enforcement tests are likely to need a major overhaul.

Documentation Impact

The following things need to be documented for operators:

Their newly added ability to create policy.json files for Monasca API services
The functionality of the monasca-api-policy and monasca-log-api-policy scripts

References

[0] https://governance.openstack.org/tc/goals/queens/policy-in-code.html

[1] https://github.com/openstack/nova/blob/master/nova/policy.py

[2] https://github.com/openstack/nova/blob/master/nova/policy.py#L207

[3] https://github.com/openstack/nova/blob/master/nova/policies/__init__.py

[2] https://github.com/openstack/magnum/blob/master/magnum/common/policy.py#L102

History

Revisions
Release Name	Description
Queens	Introduced