..
      Copyright 2012 New Dream Network, LLC (DreamHost)

      Licensed under the Apache License, Version 2.0 (the "License"); you may
      not use this file except in compliance with the License. You may obtain
      a copy of the License at

          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
      License for the specific language governing permissions and limitations
      under the License.

===============
 Configuration
===============

Polling
~~~~~~~

Polling rules are defined by the `polling.yaml` file. It defines the pollsters
to enable and the interval they should be polled. See :doc:`plugins` for
details on how to write and plug in your plugins.

Each source configuration encapsulates meter name matching which matches
against the entry point of pollster. It also includes: polling
interval determination, optional resource enumeration or discovery.

All samples generated by polling are placed on the queue to be handled by
the pipeline configuration loaded in the notification agent.

The polling definition may look like the following::

    ---
    sources:
      - name: 'source name'
        interval: 'how often the samples should be generated'
        meters:
          - 'meter filter'
        resources:
          - 'list of resource URLs'
        discovery:
          - 'list of discoverers'

The *interval* parameter in the sources section should be defined in seconds.
It determines the cadence of sample generation.

A given polling plugin is invoked according to each source section
whose *meters* parameter matches the plugin's meter name.  That is,
the matching source sections are combined by union, not intersection,
of the prescribed time series. It functions the same as Pipelines_
filtering

The optional *resources* section of a pipeline source allows a list of
static resource URLs to be configured. An amalgamated list of all
statically configured resources for a set of pipeline sources with a
common interval is passed to individual pollsters matching those pipelines.

The optional *discovery* section of a pipeline source contains the list of
discoverers. These discoverers can be used to dynamically discover the
resources to be polled by the pollsters defined in this pipeline. The name
of the discoverers should be the same as the related names of plugins in
:file:`setup.cfg`.

If *resources* or *discovery* section is not set, the default value would
be an empty list. If both *resources* and *discovery* are set, the final
resources passed to the pollsters will be the combination of the dynamic
resources returned by the discoverers and the static resources defined
in the *resources* section. If there are some duplications between the
resources returned by the discoverers and those defined in the *resources*
section, the duplication will be removed before passing those resources
to the pollsters.

There are three ways a pollster can get a list of resources to poll, as the
following in descending order of precedence:

    1. From the per-pipeline configured discovery and/or static resources.
    2. From the per-pollster default discovery.
    3. From the per-agent default discovery.


.. _Pipeline-Configuration:

Pipelines
~~~~~~~~~

Pipelines describe a coupling between sources of samples and the
corresponding sinks for transformation and publication of the samples.

A source is a set of samples that should be passed to specified sinks. These
samples may come from polling agents or service notifications.

A sink on the other hand is a consumer of samples, providing logic for
the transformation and publication of samples emitted from related sources.
Each sink configuration is concerned `only` with the transformation rules
and publication conduits for samples.

In effect, a sink describes a chain of handlers. The chain starts with
zero or more transformers and ends with one or more publishers. The first
transformer in the chain is passed samples from the corresponding source,
takes some action such as deriving rate of change, performing unit conversion,
or aggregating, before passing the modified sample to the next step.

The chains end with one or more publishers. This component makes it possible
to persist the data into storage through the message bus or to send it to one
or more external consumers. One chain can contain multiple publishers, see the
:ref:`multi-publisher` section.


Pipeline configuration
----------------------

Pipeline configuration by default, is stored in a separate configuration file,
called :file:`pipeline.yaml`, next to the :file:`ceilometer.conf` file. The
pipeline configuration file can be set in the *pipeline_cfg_file* parameter in
:file:`ceilometer.conf`. Multiple chains can be defined in one configuration
file.

The chain definition looks like the following::

    ---
    sources:
      - name: 'source name'
        sinks
          - 'sink name'
    sinks:
      - name: 'sink name'
        transformers: 'definition of transformers'
        publishers:
          - 'list of publishers'

The *name* parameter of a source is unrelated to anything else;
nothing references a source by name, and a source's name does not have
to match anything.

There are several ways to define the list of meters for a pipeline source. The
list of valid meters can be found in the :ref:`measurements` section. There is
a possibility to define all the meters, or just included or excluded meters,
with which a source should operate:

* To include all meters, use the ``*`` wildcard symbol.
* To define the list of meters, use either of the following:

  * To define the list of included meters, use the ``meter_name`` syntax
  * To define the list of excluded meters, use the ``!meter_name`` syntax
  * For meters, which identify a complex Sample field, use the wildcard
    symbol to select all, e.g. for ``disk.read.bytes``, use ``disk.*``

The above definition methods can be used in the following combinations:

* Only the wildcard symbol
* The list of included meters
* The list of excluded meters
* Wildcard symbol with the list of excluded meters

.. note::
    At least one of the above variations should be included in the meters
    section. Included and excluded meters cannot co-exist in the same
    pipeline. Wildcard and included meters cannot co-exist in the same
    pipeline definition section.

The *transformers* section of a pipeline sink provides the possibility to add a
list of transformer definitions. The names of the transformers should be the
same as the names of the related extensions in :file:`setup.cfg`. For a more
detailed description, please see the `transformers`_ section of the
Administrator Guide of Ceilometer.

.. _transformers: https://docs.openstack.org/admin-guide/telemetry-data-collection.html#transformers

The *publishers* section contains the list of publishers, where the samples
data should be sent after the possible transformations. The names of the
publishers should be the same as the related names of the plugins in
:file:`setup.cfg`.

The default configuration can be found in `pipeline.yaml`_. For more details about
how to configure publishers, see :ref:`publisher-configuration`.

.. _pipeline.yaml: https://git.openstack.org/cgit/openstack/ceilometer/tree/ceilometer/pipeline/data/pipeline.yaml

Pipeline Processing
-------------------

On large workloads, multiple notification agents can be deployed to handle the
flood of incoming messages from monitored services. If transformations are
enabled in the pipeline, the notification agents must be coordinated to ensure
related messages are routed to the same agent. To enable coordination, set the
``workload_partitioning`` value in ``notification`` section.

To distribute messages across agents, ``pipeline_processing_queues`` option
should be set. This value defines how many pipeline queues to create which will
then be distributed to the active notification agents. It is recommended that
the number of processing queues, at the very least, match the number of agents.

Increasing the number of processing queues will improve the distribution of
messages across the agents. It will also help batching which minimises the
requests to Gnocchi storage backend.

.. warning::

   Decreasing the number of processing queues may result in lost data as any
   previously created queues may no longer be assigned to active agents. It
   is only recommended that you **increase** processing queues.


.. _pipeline-publishers:

Publishers
~~~~~~~~~~

For more information about publishers see the `publishers`_ section of the
Administrator Guide of Ceilometer.

.. _publishers: https://docs.openstack.org/admin-guide/telemetry-data-pipelines.html#publishers
