Charm Specs

Enable Audit Middleware that comes with keystonemiddleware

Thu, 26 Oct 2023 00:00:00

This is a requirement from one of the customers to enable audit middleware.

Problem Description

Currently, manual changes are made to the configuration to enable audit middleware. This specification is for a configuration option that can be used to enable audit middleware in a charm. This can be applied, as required, to applicable OpenStack charms.

Proposed Change

Update existing charms to enable this feature.

The customer in question is currently running bionic queens. This spec is a basis for that request.

Alternatives

Do it manually.

Implementation

For each of the OpenStack charms that provides API, we need to do the following:

Add a configuration option to enable or disable audit middleware.
We need to add the specific sections that need to go into 3 files.
- /etc/<project>/<project>.conf
- /etc/<project>/api-paste.ini
- /etc/<project>/api_audit_map.conf
Test to see if the corresponding files are changed correctly.
Write unit and functional tests.

Templates for /etc/<project>/api_audit_map.conf file can be found in https://github.com/openstack/pycadf/tree/master/etc/pycadf.

For further details on the implementation see https://docs.openstack.org/keystonemiddleware/latest/audit.html.

Assignee(s)

Primary assignee:: None

Gerrit Topic

Use Gerrit topic “audit-middleware” for all patches related to this spec.

git-review -t audit-middleware

Work Items

Understand the changes required for each project, maybe by changing by hand.
Common changes will be implemented in the charmhelpers library.
Write tests in charmhelpers for these changes.
For each of the projects:
1. sync the new charmhelpers.
2. Add the relevant updated templates.
  - /etc/<project>/<project>.conf
  - /etc/<project>/api-paste.ini
  - /etc/<project>/api_audit_map.conf
3. Write the amulet or zaza tests to ensure that the changes are good.

Repositories

No new git repositories will need to be created. However, multiple git repositories will need to be touched for this implementation to work

These are the initial charms that are within the scope of this specification:

The following repo will also need to be updated, so ensure that similar information is stored in one central place, rather than duplicating the contents in the above repositories.

https://github.com/juju/charm-helpers

Initial work was tried in the following commits:

Documentation

It will be documented within each of the charms’ config.yaml.

Security

Enable API auditing for security compliance.

Testing

Unit tests will be added to charm-helpers.
Functional tests will need to be added for the new option, and checking that the configuration is changed correctly, and then disabled.

Dependencies

There are no further dependencies.

Ceph Storage Action

Thu, 26 Oct 2023 00:00:00

We should allow the user to specify the class of storage that a given storage device should be added to. This action would allow adding a list of osd-devices into specified Ceph buckets.

Problem Description

All osd-devices are currently added to the same default bucket making it impossible to use multiple types of storage effectively. For example, users may wish to bind SSD/NVMe devices into a fast/cache bucket, 15k spindles into a default bucket, and 5k low power spindles into a slow bucket for later use in pool configuration.

Proposed Change

Add an action that includes additional metadata into the osd-devices list allowing for specification of bucket types for the listed devices. These OSDs would be added to the specified bucket when the OSD is created.

Alternatives

User could specify a pre-build usage profile and we could map device types onto that profile by detecting the device type and deciding, based on the configured profile, what bucket the device should go into.

This solution is being discarded because of the difficulty of designing and implementing usage profiles to match any use case automigically. It will also require a lot of work to correctly identify the device type and decide where to place it in the desired profile.

In addition, it would require that we only support a single “profile” within a deployed ceph-osd cluster.
Charm could define additional storage attach points in addition to osd-devices that would allow the user to specify what bucket they should add devices to.

The reason for discarding this solution is that it was determined to be too limiting because of only supporting a fixed number of bindings, in addition to making changes harder because of backwards compatibility requirements.

Implementation

Assignee(s)

Primary assignee:: Chris MacNaughton <chris.macnaughton@canonical.com>
Contact:: Chris Holcombe <chris.holcombe@canonical.com>

Gerrit Topic

Use Gerrit topic “storage-action” for all patches related to this spec.

git-review -t storage-action

Work Items

Create a bucket with the required name. We do this so that we can create pool in the specified bucket type. This would be handled within the ceph-mon charm.
Create an action that returns a list of unused storage devices along with their device types.
Create an action that takes osd-devices and storage-type, that would be an enum into the buckets that we create. Rather than be a user specified string, this would be an enumerated list in the ceph charms provided through the shared charms_ceph library.
Add the ability for other charms to request the bucket that should back their created pools when creating pools through the broker.

Documentation

This will require additional documentation be added around how to use the action correctly and what to expect from it. This documentation will be added to the charm README

Security

This should have no security impact.

Testing

There will need to be new unit tests and functional tests to ensure that the necessary buckets are created and that the disks are added to them.

Repositories

None

Dependencies

None

Cinder subordinate charm cinder-nimblestorage

Thu, 26 Oct 2023 00:00:00

Problem Description

Cinder has a great number of drivers for different backends. Almost all currently available commercial storage arrays and many other software only solutions have a driver available, if not already in the upstream cinder project, as an installable package that enables cinder with the extra driver.

Because of how many possibilities exist, a model was developed where a separate subordinate charm will be deployed to take care of the specifics of the driver and pass configuration to the main cinder charm over a relation so that cinder can write to its own configuration file.

Proposed Change

Instead of changing the cinder charm, this proposal aims to create a separate subordinate charm specific for the nimblestorage driver. This cinder installation would enable access to Nimblestorage devices. The following points are of condiderable importance for the development of this subordinate charm:

The new implementation should not conflict with current implementation. Both should co-exist, even in the same deployment if possible.
The new charm should implement all features currently existing in the cinder charm, regarding the nimblestorage functionality.

Note that because the Nimblestorage driver is part of the cinder-common package, no additional packages need to be installed for this charm to work.

This charm would be supported in the Ussuri release of Openstack and newer.

Alternatives

At the time of this writing, there exist no alternatives.

Implementation

Assignee(s)

Primary assignee:: Gustavo Sanchez <gustavo.sanchez@canonical.com> - LP ID: gustavosr98
Secondary assignee:: Luciano Lo Giudice <luciano.logiudice@canonical.com> - LP ID: lmlogiudice

Gerrit Topic

Use Gerrit topic “charm-cinder-nimblestorage” for all patches related to this spec.

git-review -t charm-cinder-nimblestorage

Work Items

The charm, as listed above. Plus, both unit and functional tests will be written.

Repositories

Presently, the following repository hosts the charm code:

https://opendev.org/openstack/charm-cinder-nimblestorage

Documentation

Once this charm is completed, charm-guide will be updated to reference it.

Security

No security aspect is going to change in relation to the current solution.

Testing

Code changes will be covered by both unit and functional tests. Functional tests would require Nimblestorage hardware.

Dependencies

No dependencies in addition to the ones that exist in the current solution.

Cinder subordinate charm cinder-solidfire

Thu, 26 Oct 2023 00:00:00

Problem Description

Proposed Change

Instead of changing the cinder charm, this proposal aims to create a separate subordinate charm specific for the solidfire driver. This cinder installation would enable access to Solidfire devices. The following points are of condiderable importance for the development of this subordinate charm:

The new implementation should not conflict with current implementation. Both should co-exist, even in the same deployment if possible.
The new charm should implement all features currently existing in the cinder charm, regarding the solidfire functionality.

Note that because the Solidfire driver is part of the cinder-common package, no additional packages need to be installed for this charm to work.

This charm would be supported in the Ussuri release of Openstack and newer.

Alternatives

At the time of this writing, there exist no alternatives.

Implementation

Assignee(s)

Primary assignee:: Gustavo Sanchez <gustavo.sanchez@canonical.com> - LP ID: gustavosr98
Secondary assignee:: Luciano Lo Giudice <luciano.logiudice@canonical.com> - LP ID: lmlogiudice

Gerrit Topic

Use Gerrit topic “charm-cinder-solidfire” for all patches related to this spec.

git-review -t charm-cinder-solidfire

Work Items

The charm, as listed above. Plus, both unit and functional tests will be written.

Repositories

Presently, the following repository hosts the charm code:

https://opendev.org/openstack/charm-cinder-solidfire

Documentation

Once this charm is completed, charm-guide will be updated to reference it.

Security

No security aspect is going to change in relation to the current solution.

Testing

Code changes will be covered by both unit and functional tests. Functional tests would require Solidfire storage hardware.

Dependencies

No dependencies in addition to the ones that exist in the current solution.

Memory Fragmentation Tuning

Thu, 26 Oct 2023 00:00:00

In some high memory pressure scenarios, the memory shortage would make the high order pages hard to be allocated and also the page allocation would go to the synchronous frequently reclaim thanks to the default gap between min<->low<->high is too small to wake up the kswapd (asynchronous reclaim) earlier.

Problem Description

In the OpenStack compute node, especially the hyperconverged machine with the Ceph OSDs using a lot of page cache. It is easy to have the memory allocation stall issue. The issue would lead to several issues: The new instance cannot be brought up (KVM needs to allocate order sixth pages) or VM stuck, etc. The reasons are:

1). Compaction for big order page If the THP (Transparent Huge Page) is used with the VM, it will be more severe than the persistent huge pages reserved for the VM’s dedicated usage. The THP needs to allocate the 2MB (x86) huge pages at run time. Moreover, this is the order 9 (2^9 * 4K = 2MB). In running system, it will be hard to get the continuous 512 (2^9) 4K pages according to /proc/pagetypeinfo.

2). Synchronous reclaim. There are three levels of watermark inside the system: 1). min 2). low 3). high. When the number of free pages lowers down to the low watermark. The kswapd will be wakened up to do the asynchronous reclaim. Furthermore, it will not be stopped until the number of free pages reaches the high watermark. However, when the memory allocation is strong enough, the free pages will continue to lower down to the min watermark. At this point, the number of min pages is reserved for emergency usage, and the allocation will go into the direct-reclaim (synchronous) mode. This will stall the process.

Proposed Change

In the past experience, the 1GB gap between min<->low<->high watermark is a good practice in the server environment. The bigger gap can wake up the kswapd earlier and avoid the synchronous reclaim. Moreover, this can alleviate the latency. The sysctl parameters related to the watermark gap calculation:

vm.min_free_kbytes vm.watermark_scale_factor

For the Ubuntu kernel before 4.15 (Bionic), the only way to tune the watermark is to modify the vm.min_free_kbytes. The gap would be 1/4 of the vm.min_free_kbytes. However, increasing the min_free_kbytes is the minimum watermark reservation increase, which will decrease the actual memory that the runtime system can use.

For Ubuntu kernel after 4.15, vm.watermark_scale_factor can be used to increase the gap without increasing the min watermark reservation. The gap is calculated by “watermark_scale_factor/10000 * managed_pages”.

The proposed solution is to set the 1GB watermark gap by using the above two parameters when the compute node is rebooted.

The feature will be designed in flexible ways: 1). There will be a switch to turn on/off the feature. By default, it is turned off. For some small memory compute nodes (<32GB), the 1GB low memory is too many.

2). The manual config has a higher priority to overwrite the default calculation.

Alternatives

The config can be set up in the run time with the following command: juju deploy cs:sysconfig-2 juju add-relation sysconfig nova-compute juju config sysconfig sysctl=”{vm.extfrag_threshold: 200, vm.watermark_scale_factor: 50}”

However, each system might have different memory capacities. The watermark_scale_factor needs to be calculated manually.

Implementation

Assignee(s)

Primary assignee: - Gavin Guo <gavin.guo@canonical.com>

Gerrit Topic

Use Gerrit topic “memory-fragmentation-tuning” for all patches related to this spec.

git-review -t memory-fragmentation-tuning

Work Items

Implement the watermark_scale_factor value calculation to set up the gap to 1GB.

Repositories

No new git Repository is required.

Documentation

The documentation is needed to include the switch to turn on/off the feature.

Security

The use of this feature exposes no other security attack surface.

Testing

To verify if the calculated watermark value is correct. Also, in different kernel versions, different parameters should be used (min_free_kbytes v.s. watermark_scale_factor).

Dependencies

None

Sharing SSH auth credentials across cloud-compute-related applications

Thu, 26 Oct 2023 00:00:00

Some cloud environments require different configuration settings (e.g. allocation ratios) for different hosts, yet still need to allow SSH-based migration between hosts. The proposed change in this spec would allow sharing of credentials across multiple applications related to the cloud-compute relation.

Problem Description

At present, the nova-cloud-controller charm gathers SSH public keys for all cloud-compute-related applications. These SSH keys are then shared as needed, but only to peers within the same related application.

While this is likely a sane default, especially considering that different applications may represent different classes of compute hosts which don’t allow migration between them (e.g. SRIOV vs. non-SRIOV), there are cases where there is no fundamental reason to disallow migration.

In cases like this, at present it’s necessary to manually modify configuration files, perhaps setting them immutable, to provide the different configuration values needed for the different types of hosts, yet still allow them to perform migrations between each other.

Thus, it is believed it would be beneficial to allow for sharing all SSH keys across all nova-compute applications to overcome those limitations.

This specification is meant to address an implementation for the following feature request on Launchpad: https://bugs.launchpad.net/charms/+source/nova-cloud-controller/+bug/1468871

Proposed Change

Share SSH keys across all nova-compute applications through the cloud-compute relation with the nova-cloud-controller charm.

The code will now also have to go through each cloud-compute relation whenever there is an update to a node or SSH key, in order to keep the SSH keys on all compute nodes of all relations consistent.

Upgrade Impact

Upon upgrading the nova-cloud-controller charm version, the upgrade-charm hook code will cycle through all cloud-compute relations and set the relation data with all SSH authorized keys and known hosts to all compute nodes in the relations.

Performance Impact

Performance is slightly impacted as the hooks will take longer both on the nova-cloud-controller charm to set a now higher number of SSH keys and on each the nova-compute charms to process that higher number of SSH keys received on the relation. The number of hooks executed remains the same.

On cloud-compute-relation-changed hooks, when a change on SSH keys is triggered by the compute node, or if a new compute node is added, the code will also take slightly longer due to cycling through all the relations to make all SSH keys on all compute nodes consistent, where before it only needed to update the relation pertaining to the compute node that triggered the change.

Alternatives

One alternative may be to write a subordinate charm simply for the purpose of sharing SSH keys and known hosts entries. While this could in theory be done, it is less than ideal because that effectively would create two sources of truth regarding which hosts should share credentials with which hosts, and it would effectively replicate a non-trivial amount of logic already handled by charm-nova-cloud-controller.

Implementation

Assignee(s)

Primary assignee:: Rodrigo Barbieri <rodrigo.barbieri@canonical.com> Paul Goins <paul.goins@canonical.com>

Gerrit Topic

Use Gerrit topic “migration-ssh-auth-sharing” for all patches related to this spec.

git-review -t migration-ssh-auth-sharing

Work Items

Allow for sharing across all cloud-compute-related applications

Repositories

The changes would be isolated to the charm-nova-cloud-controller repository.

Documentation

The release notes for the new charm release that includes this change should mention the new behavior.

Security

No security concerns are raised given that all the nova-compute charm applications are part of the same environment and connected to the same nova-cloud-controller nodes.

Testing

This functionality can be tested via unit tests. Functional tests may be overly cumbersome for this particular feature, although direct testing in a lab environment should also be done as a sanity check.

Dependencies

N/A

SR-IOV Port Live Migration

Thu, 26 Oct 2023 00:00:00

A recent change in Nova for Train release introduced the ability to live migrate instances with NUMA topology (CPU Pinning and Huge Pages also considered). This spec is to discuss and ensure support of that feature in a charm deployment.

Problem Description

As an operator, I would like the ability to live migrate instances with NUMA topology.

Proposed Change

In Nova several features depend of the internal NUMA topology object. As indicated in the problem description an operator may ask the desir of migrating such instances.

NUMA awareness when flavor is configured with hw:numa_nodes=N
CPU Pinning, when flavor is confiugred with hw:cpu_policy=dedicated
Huge Pages, when flavor is configured with hw:mem_page_size=large

The support of this improvement will be considered as tech-preview for Ubuntu 19.10/Train and production-ready for Ubuntu 18.04 LTS/Train and Ubuntu 20.04 LTS/Ussuri.

No changes are expected to any of the charms. The aim here is to validate behavior and provide additional documentations.

Alternatives

An operator could identify a destination host with the same characteristics as the source host and ensure that the host CPUs where the guest vCPUs are pinned on are free. Also, in case of huge pages usage, ensure the availability of the huge pages in the destination host.

By default the availibity of live migrate instances with NUMA topology is disabled in Nova but operators can enable that behavior using:

workaround/enable_numa_live_migration=True

It is to note that this option is not currently exposed in charm-nova-compute.

Implementation

Assignee(s)

Primary assignee:: Sahid Orentino Ferdjaoui <lp:sahid-ferdjaoui>

Gerrit Topic

Use Gerrit topic “numa-live-migration” for all patches related to this spec.

git-review -t numa-live-migration

Work Items

The implementation consists of the following items:

Validate behavior using NUMA awarness.
Validate behavior using CPU Pinning.
Validate behavior using Huge Pages.

Repositories

Any additional documentation will be considered for charm-deployment-guide.

Documentation

n/a

Security

n/a

Testing

n/a

Dependencies

See support of NUMA aware live migration spec [0].

[0] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/numa-aware-live-migration.html

Enable clean removal of a ovn-central unit from the cluster

Thu, 26 Oct 2023 00:00:00

Add capability to ovn-central charm to gracefully leave Southbound and Northbound clusters. This should be performed automatically in the process of unit’s teardown. In addition, there should be an action that enables user to manually kick specified cluster member in case that the unit was not able to gracefully leave the cluster on its removal.

It’s important to keep in mind that each ovn-central unit is always part of two clusters simultaneously, Northbound and Southbound. Any action in regards to leaving or kicking of the unit must be performed on both of them.

Problem Description

Currently there’s no clean way to remove ovn-central unit from the cluster. As described in LP#1948680, removed units do not properly leave the Southbound and Northbound clusters. Remaining active cluster members will still expect them and their continued presence will affect fault tolerance of the whole cluster. For example starting with a cluster of five ovn-central units and removing two of them will cause the cluster to behave as a cluster of five members with two faults, not a cluster of three members with no fault.

Proposed Change

Part of the solution to this problem is to invoke “ovs-appctl cluster/leave” command when the unit is in the process of shutting down. This will ensure that remaining cluster members will reconfigure themselves and will no longer expect the leaving unit to come back. These two commands should be executed for the unit to leave both clusters:

ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/leave OVN_Northbound
ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/leave OVN_Southbound

However there’s also a possibility that departing unit won’t be able to execute “cluster/leave” command properly during the teardown so there should also be an explicit action to kick “zombie” cluster members that were left behind after their ovn-central unit disappeared.

Action: kick

This action will enable user to kick specific members from the Southbound and Northbound OVN clusters. It will require user to supply a short version of an ID of the server that should be kicked from the cluster. Since the ovn-central unit is member in two clusters, under two unique identities, this action will accept following parameters:

–sb-identity <ID> - ID of the server in Southbound cluster
–nb-identity <ID> - ID of the server in Northbound cluster
–i-really-mean-it - This option signifies that user is aware of destructive nature of this action

Example:

juju run-action ovn-central/0 kick sb-identity=13fe nb-identity=77eb i-really-mean-it=true

Above example would translate to unit executing following commands:

ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/kick OVN_Southbound 13fe
ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/kick OVN_Northbound 77eb

This action will be executable on any unit that’s still participating in the OVN cluster, it does not have to be leader and it only needs to be executed once. The only problematic part is that user will have to provide Southbound and Northbound IDs which are not really exposed via juju. To allow user to correlate specific ovn-central unit with its cluster identities, we propose a new action called “cluster-status”

Action: cluster-status

Prerequisite for this action is to add two new data attributes to the peer relation between ovn-central units implemented by OVSDB Interface:

sb-identity: Short version of the UUID that unit uses in Southbound cluster
nb-identity: Short version of the UUID that unit uses in Northbound cluster

Each unit will publish its Northbound and Southbound ID via the peer relation.

The action itself will convey outputs of following commands with minor formatting changes:

ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound
ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound

Example output of one of these commands looks something like this:

f3be
Name: OVN_Southbound
Cluster ID: 77eb (77eb6ddc-6910-403b-b8da-ce3d90815183)
Server ID: f3be (f3be898b-e8e4-4719-b344-902444227dd7)
Address: ssl:10.5.1.88:6644
Status: cluster member
Role: follower
Term: 8
Leader: d33d
Vote: d33d

Election timer: 4000
Log: [25, 25]
Entries not yet committed: 0
Entries not yet applied: 0
Connections: ->9ea4 ->fbfc <-fbfc <-9ea4 <-d33d ->d33d
Disconnections: 4
Servers:
    d33d (d33d at ssl:10.5.0.92:6644) last msg 839 ms ago
    9ea4 (9ea4 at ssl:10.5.3.144:6644) last msg 81662711 ms ago
    fbfc (fbfc at ssl:10.5.1.86:6644) last msg 1221658 ms ago
    f3be (f3be at ssl:10.5.1.88:6644) (self)

The only proposed change in the following output would be in the “Servers” section. There, we would drop information about “last msg” as it is not very informative because cluster followers (non-leaders) don’t keep active heartbeats between each other. Instead of it, we would add information about which unit the specific server belongs to.

Example:

Servers:
    d33d (d33d at ssl:10.5.0.92:6644) ovn-central/0
    9ea4 (9ea4 at ssl:10.5.3.144:6644) ovn-central/1
    fbfc (fbfc at ssl:10.5.1.86:6644) ovn-central/2
    f3be (f3be at ssl:10.5.1.88:6644) UNKNOWN  # in case that this identity does not belong to any other related juju unit

With these information, user can effectively use action “kick” to remove undesirable members from the cluster.

These features would be supported on Openstack Yoga (and later) on Ubuntu 22.04 LTS.

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: Martin Kalcok <martin.kalcok@canonical.com>

Gerrit Topic

Use Gerrit topic “ovn-central-downscaling” for all patches related to this spec.

git-review -t ovn-central-downscaling

Work Items

Implement automatic execution of “ovs-appctl cluster/leave” command when ovn-central unit is removed
Implement announcing of Southbound and Northbound identity via OVSDB Interface
Implement “cluster-status” juju action in ovn-central charm
Implement juju action “kick” that will enable removal of arbitrary cluster members from Southbound and/or Northbound clusters

Repositories

Documentation

README.md of the ovn-central charm will have to be updated to include information about new juju actions.

Security

This change does not pose any direct security risk. The action “kick” can be dangerous when used incorrectly so it will include “–i-really-mean-it” required flag, to signify to the user that it needs to be used carefully.

Testing

Aside from unit tests, this change will have to be accompanied also with functional tests that will verify that:

removing unit will remove it from Southbound and Northbound clusters
using juju action “kick” will remove the specified members from the clusters
juju action “cluster-status” displays correct information

Dependencies

None

Pausing Charms with subordinate hacluster without sending false alerts

Thu, 26 Oct 2023 00:00:00

Overall, the goal is to leave “warning” alerts instead of “critical” that will help a human operator understand that all services are not completely healthy while reducing the criticality due to an on-going operation. Nrpe checks will be reconfigured once services under a maintenance operation are set back to normal (resume).

The following logic will be applied when pausing/resuming a unit:

Pausing a principal unit, pauses the subordinate hacluster;
Resuming a principal unit, resumes the subordinate hacluster;
Pausing a hacluster unit, pauses the principal unit;
Resuming a hacluster unit, resumes the principal unit;

Problem Description

We need to stop sending false alerts when a hacluster subordinate of an Openstack charm unit is paused or when the principal unit is also paused for maintenance. This may help operations to receive more effective alerts.

There are several charms that use hacluster and NRPE that may benefit from this:

charm-ceilometer
charm-ceph-radosgw
charm-designate
charm-keystone
charm-neutron-api
charm-nova-cloud-controller
charm-openstack-dashboard
charm-cinder
charm-glance
charm-heat
charm-swift-proxy

Pausing Principal Unit

If eg. 3 keystone units (keystone/0, keystone/1 and keystone/2) are deployed and keystone/0 is paused:

1) haproxy_servers on the other units (keystone/1 and keystone/2) will alert, because apache2 service on keystone/0 is down

haproxy, apache2.service and memcached.service in keystone/0 will also alert

3) it’s possible that corosync and pacemaker have the VIP placed on the same unit at which point the service will fail as haproxy is disabled. So hacluster subordinate unit should also be paused.

Note: Services affected when pausing a principal unit may change depending on the principal charm

Pausing hacluster unit

Pausing hacluster set the cluster node, e.g keystone, in standby mode. A standby node will have its resources stopped (hacluster, apache2) which will fire false alerts. To solve this issue, the units of hacluster should inform the keystone unit that they are paused. A way of doing this is through the ha relation.

Proposed Change

Pausing Pausing Principal Unit

Pause action on a principal unit should share the event with its peers to modify the behavior on them (until the resume action is triggered). It should also share the status (paused/resumed) to the subordinate unit to be able to catch-up the same status.

File actions.py in the principal unit

def pause(args):
    pause_unit_helper(register_configs())

    # Logic added to share the event with peers
    inform_peers_if_ready(check_api_unit_ready)
    if is_nrpe_joined():
      update_nrpe_config()

    # logic added to inform hacluster subordinate unit has been paused
    relid = relation_ids('ha')
    for r_id in relid:
        relation_set(relation_id=r_id, paused=True)

def resume(args):
    resume_unit_helper(register_configs())

    # Logic added to share the event with peers
    inform_peers_if_ready(check_api_unit_ready)
    if is_nrpe_joined():
      update_nrpe_config()

    # logic added to inform hacluster subordinate unit has been resumed
    relid = relation_ids('ha')
    for r_id in relid:
        relation_set(relation_id=r_id, paused=False)

After pausing a principal unit, it will change the unit-state-{unit_name} to NOTREADY. E.g:

juju show-unit keystone/0 --endpoint cluster
keystone/0:
  workload-version: 17.0.0
  machine: "1"
  opened-ports:
  - 5000/tcp
  public-address: 10.5.2.64
  charm: cs:~openstack-charmers-next/keystone-562
  leader: true
  relation-info:
  - endpoint: cluster
    related-endpoint: cluster
    application-data: {}
    local-unit:
      in-scope: true
      data:
        admin-address: 10.5.2.64
        egress-subnets: 10.5.2.64/32
        ingress-address: 10.5.2.64
        internal-address: 10.5.2.64
        private-address: 10.5.2.64
        public-address: 10.5.2.64
        unit-state-keystone-0: NOTREADY

Note: unit-state-{unit_name} field is already implemented, I’m just proposing to use this field and change the value to NOTREADY when a unit is paused and return to READY when resumed.

With every unit knowing which one is paused, it is possible to change the script check_haproxy.sh to accept a flag to warn the keystone units that are paused. The bash script is not able now to receive flags.

Check_haproxy.sh could be rewritten from Bash to Python to accept a flag to warn specific hostname (e.g. check_haproxy.py –warning keystone-0) is under maintenance.

The file nrpe.py on charmhelpers/contrib/charmsupport should have changes to first check if there is any paused unit in the cluster and then add the warning flag if necessary

def add_haproxy_checks(nrpe, unit_name):
    """
    Add checks for each service in list

    :param NRPE nrpe: NRPE object to add check to
    :param str unit_name: Unit name to use in check description
    """
    cmd = "check_haproxy.py"

    peers_states = get_peers_unit_state()
    units_not_ready = [
        unit.replace('/', '-')
        for unit, state in peers_states.items()
        if state == UNIT_NOTREADY
    ]

    if is_unit_paused_set():
        units_not_ready.append(local_unit().replace('/', '-'))

    if units_not_ready:
        cmd += " --warning {}".format(','.join(units_not_ready))

    nrpe.add_check(
        shortname='haproxy_servers',
        description='Check HAProxy {%s}' % unit_name,
        check_cmd=cmd)
    nrpe.add_check(
        shortname='haproxy_queue',
        description='Check HAProxy queue depth {%s}' % unit_name,
        check_cmd='check_haproxy_queue_depth.sh')

When a principal unit changes the state e.g READY to NOTREADY, it’s necessary to rewrite the nrpe files on the other principal units in the cluster because, otherwise, they won’t be able to warn that a unit is under maintenance.

File responsible for hooks in the classic charms:

@hooks.hook('cluster-relation-changed')
@restart_on_change(restart_map(), stopstart=True)
def cluster_changed():
    # logic added to update nrpe_config in all principal units when
    # a status is changed
    update_nrpe_config()

Note: In reactive charms, it might be slightly different using handlers, but the mean idea is to update_nrpe_config every time that a config in the cluster is changed. This will prevent false alerts in the other units in the cluster.

Services from Principal Unit

Removing the .cfg files, when the unit is paused, for those services at /etc/nagios/nrpe.d would stop sending critical errors. The downside of this approach is that it won’t have user friendly messages in Nagios saying that the specific services (apache2, memcached and etc) is under maintenance, on the other hand, it’s simpler to be achieved.

File responsible for hooks in a classic charm:

@hooks.hook('nrpe-external-master-relation-joined',
            'nrpe-external-master-relation-changed')
def update_nrpe_config():
    # logic before change
    # ...

    nrpe_setup = nrpe.NRPE(hostname=hostname)
    nrpe.copy_nrpe_checks()

    # added logic to remove services
    if is_unit_paused_set():
        nrpe.remove_init_service_checks(
            nrpe_setup,
            _services,
            current_unit
        )

    else:
        nrpe.add_init_service_checks(
            nrpe_setup,
            _services,
            current_unit
        )

    # end of added logic

    nrpe.add_haproxy_checks(nrpe_setup, current_unit)
    nrpe_setup.write()

The new logic to remove those services is presented below.

File charmhelpers/contrib/charmsupport/nrpe.py

# added logic to remove apache2, memcached and etc...
def remove_init_service_checks(nrpe, services, unit_name):
    for svc in services:
        if host.init_is_systemd(service_name=svc):
            nrpe.remove_check(
                shortname=svc,
                description='process check {%s}' % unit_name,
                check_cmd='check_systemd.py %s' % svc
            )

The status of the services will disappear from nagios after some minutes. When the resume action is used, the services are restored initially as PENDING, but after some minutes the check is done.

Pausing hacluster unit

File actions.py in charm-hacluster:

def pause(args):
    """Pause the hacluster services.
    @raises Exception should the service fail to stop.
    """
    pause_unit()
    # logic added to inform keystone that unit has been paused
    relid = relation_ids('ha')
    for r_id in relid:
        relation_set(relation_id=r_id, paused=True)


def resume(args):
    """Resume the hacluster services.
    @raises Exception should the service fail to start."""
    resume_unit()
    # logic added to inform keystone that unit has been resumed
    relid = relation_ids('ha')
    for r_id in relid:
        relation_set(relation_id=r_id, paused=False)

Pausing a hacluster would result in sharing a new variable paused that can be used in the principal units.

File responsible for hooks in a classic charm:

@hooks.hook('ha-relation-changed')
@restart_on_change(restart_map(), restart_functions=restart_function_map())
def ha_changed():

    # Added logic to pause keystone unit when hacluster is paused
    for rid in relation_ids('ha'):
        for unit in related_units(rid):
            paused = relation_get('paused', rid=rid, unit=unit)
            clustered = relation_get('clustered', rid=rid, unit=unit)
            if clustered and is_db_ready():
                if paused == 'True':
                    pause_unit_helper(register_configs())

                elif paused == 'False':
                    resume_unit_helper(register_configs())

                update_nrpe_config()
                inform_peers_if_ready(check_api_unit_ready)
                # inform subordinate unit that is paused or resumed
                relation_set(relation_id=rid, paused=is_unit_paused_set())

By informing peers and updating the nrpe config this will be enough to trigger the necessary logic to remove the services checks.

In a situation where the principal unit is paused, hacluster should also be paused. For this to happen, it can use the ha-relation-changed from charm-ha-cluster:

@hooks.hook('ha-relation-joined',
            'ha-relation-changed',
            'peer-availability-relation-joined',
            'peer-availability-relation-changed',
            'pacemaker-remote-relation-changed')
def ha_relation_changed():
    # Inserted logic
    # pauses if the principal unit is paused
    paused = relation_get('paused')
    if paused == 'True':
        pause_unit()
    elif paused == 'False':
        resume_unit()

    # share if the subordinate unit status
    for rel_id in relation_ids('ha'):
        relation_set(
            relation_id=rel_id,
            clustered="yes",
            paused=is_unit_paused_set()
        )

Alternatives

One alternative to services from the principal unit checks is to change systemd.py in charm-nrpe to accept flag -w like the proposal for the check_haproxy.py

This way would not be necessary to remove the .cfg files for services from the principal unit, but would be necessary to adapt the function add_init_service_checks to be able to accept services with the warning flag.

Implementation

Assignee(s)

Primary assignee:: gabrielcocenza

Gerrit Topic

Use Gerrit topic “pausing-charms-hacluster-no-false-alerts” for all patches related to this spec.

git-review -t pausing-charms-hacluster-no-false-alerts

Work Items

charmhelpers
- nrpe.py
- check_haproxy.py
charm-ceilometer
charm-ceph-radosgw
charm-designate
charm-keystone
charm-neutron-api
charm-nova-cloud-controller
charm-openstack-dashboard
charm-cinder
charm-glance
charm-heat
charm-swift-proxy
charm-nrpe (Alternative)
- systemd.py
charm-hacluster
- actions.py

Repositories

No new git Repository is required.

Documentation

It will be necessary to document the impact of pausing/resuming a subordinate hacluster and the side effect on Openstack API charms.

Security

No additional security concerns.

Testing

Code changes will be covered by unit and functional tests. For functional tests, it will use a bundle with keystone, hacluster, nrpe and nagios.

Dependencies

None

Align Prometheus metrics exporters to any LMA stack

Thu, 26 Oct 2023 00:00:00

The main objective of this spec is to improve the observability of bind and haproxy across the OpenStack Charms using Prometheus that is a popular tool for metrics gathering. Currently, OpenStack charms (except Ceph) do not support metrics gathering on their components.

Problem Description

The goal of this spec is to standardize the Juju interfaces required (and their logic) on OpenStack charms in order to facilitate Prometheus metrics collection (via Prometheus charms) and dashboards exports (via Grafana charms), no matter which LMA stack is used.

The alert rules that will be configured on prometheus for monitoring bind and haproxy are not part of this scope.

Proposed Change

Prometheus-scrape interface

This interface is reactivated form the prometheus_scrape library, which is used in the new COS. It is written for the Operator framework, so it will be necessary to wrap MetricsEndpointConsumer(Object) to PrometheusScrapeRequires(Endpoint) and MetricsEndpointProvider(Object) to PrometheusScrapeProvides(Endpoint).

The interface charm-interface-prometheus-scrape already implements the PrometheusScrapeProvides(Endpoint) part and thus it can be used to relate reactive charms with prometheus-k8s-operator charm.

Bind

BIND service does not support a built-in Prometheus metrics exporter. However, this service can expose a statistics channel, used by the bind-exporter to retrieve metrics to Prometheus (see bind-exporter service).

To take advantage of the bind exporter, changes need to be made in the charm designate-bind.

Charm designate-bind

This charm needs to expose a statistics service for bind whenever it is related with the prometheus charm.

The stats.conf will have the following content:

statistics-channels {
  inet <stats-listen-net> port <stats-port> allow { <client-ip>; };
};

Where stats-listen-net, stats-port and client-ip default values are 127.0.0.1, 8053 and 127.0.0.1, respectively. This will open a statistics channel in bind that prometheus_scrape interface can expose metrics for prometheus to collect.

This new configuration file will be included at /etc/bind/named.conf appending the following content:

include "/etc/bind/stats.conf";

When prometheus is removed, it will trigger the reactive endpoint logic for departed that will be responsible for removing the stats.conf, named.conf files and removing it from the restart_map.

Configuration options

no configurations will be available. The charm will automatically configure ports and listen network (localhost).

Relations:

The charm-designate-bind will support the prometheus_scrape interface to relate directly with prometheus charm and will have all the logic necessary to install the snap responsible for the prometheus bind exporter. The charm will interact with the snap by setting listen-address and stats-groups.

Furthermore, charm-designate-bind would also relate to the Grafana charm to share a rendered template with an optimized Grafana dashboard.

Provides:

bind-exporter:prometheus_scrape - The relation connecting this charm with the prometheus charm, while providing hostname and port through which it will collect all the metrics in Prometheus.
grafana:grafana-dashboard - The relation connecting this charm with the Grafana charm and at the same time responsible for creating a new dashboard from charm resource.

Note: The render_grafana_dashboard function from charmhelpers will be responsible for rendering the dashboard.

Actions:

This charm requires no action

HAproxy

HAProxy enterprise Edition and from version 2.0 also HAProxy community edition has built-in support to export metrics to Prometheus via Prometheus exporter.

To take advantage of the Prometheus exporter, changes need to be made in the haproxy.cfg file in the stats section.

listen stats:
    mode http
    stats enable
    bind {{ stats_exporter_host }}:{{ stats_exporter_port }}
    option http-use-htx
    http-request use-service prometheus-exporter if { path /metrics }

To see more information about stats_exporter_host and stats_exporter_port, see the Charmhelpers >> haproxy.cfg section below.

Implementation

For the legacy charms the implementation should be split into five parts:

editing haproxy.cfg config file - by charmhelpers in classic charms and by charms.openstack for reactive charms (see section below)
rendering Grafana dashboard template - by charmhelpers (see section below)
adding configuration options - by charm itself
adding and managing relations changes - by charm itself
providing Grafana dashboard template (JSON) - by charm itself via resource

Charmhelpers

haproxy.cfg

The charmhelpers library implements the code responsible for creating the context of haproxy.cfg, because of that it will be also responsible for adding parts needed to enable prometheus exporter. The prometheus exporter will be enabled only if these conditions are met:

Ubuntu Focal or above (haproxy version >= 2.0)
Haproxy-exporter relation exists

If both conditions are satisfied, then two values will be needed:

stats_exporter_host - obtain IP address from relation (aka. using “get_relation_ip”)
stats_exporter_port - obtain from “haproxy-exporter-stats-port” (provided automatically by the charm)

Note: In reactive charms this can be done by charms.openstack instead of charmhelpers.

Dashboard

The dashboard template will be provided as a juju resource in a JSON file. This way it’s not necessary to produce a release of charmhelpers library in order to update the dashboard and gives more flexibility to edit the template as necessary.

Charms

The following charms will benefit from these changes, however the list may not be considered exhaustive at the time of writing.

charm-ceilometer
charm-ceph-radosgw
charm-keystone
charm-neutron-api
charm-nova-cloud-controller
charm-openstack-dashboard
charm-cinder
charm-glance
charm-heat
charm-swift-proxy
charm-designate
charm-neutron-gateway
charm-percona-cluster
charm-vault
charm-ironic-api

Configuration options

No configurations will be available. The charm will configured automatically the port to Prometheus exporter.

Relations

Provides:

haproxy-exporter:prometheus_scrape - The relation connecting this charm with the Prometheus charm, while providing hostname and port through which it will collect all the metrics in Prometheus.
grafana:grafana-dashboard - The relation connecting this charm with the Grafana charm and at the same time responsible for creating a new dashboard from charm resource.

Note: The render_grafana_dashboard function from charmhelpers will be responsible for rendering the dashboard.

Actions:

No actions required.

References

Alternatives

None

Implementation

Assignee(s)

Primary assignee: - Robert Gildein <robert.gildein@canonical.com>

Gerrit Topic

Use Gerrit topic “prometheus-metrics-exporter” for all patches related to this spec.

git-review -t prometheus-metrics-exporter

Work Items

The Proposed Change and Repositories sections describe the working items.

Documentation

Documentation on how to use prometheus-exporter for HAProxy and prometheus-bind-exporter for BIND will be necessary.

Security

None

Testing

Code written or changed will be covered by unit tests and functional test. The functional test will be implemented using the Zaza framework.

Dependencies

None

Add support for managing Quorum Queues

Thu, 26 Oct 2023 00:00:00

The RabbitMQ project is moving away from classic queues n favour of quorum queues (see 4_0_deprecations). Quorum queues provide greater data safety and are based on the widely adopted Raft consensus algorithm

OpenStack does not currently support creating quorum queues but there is a change up for review to add support for them Oslo Messaging Quorum Queues.

WARNING: There is not complete feature parity between classic queues and quorum queues. For example quorum queues do not support message TTLs which maybe an issue with notifications in OpenStack queues which often build up. See _feature_comparison for more details.

Problem Description

Quorum queues are available with RabbitMQ 3.8.0 or later (eg Focal). A deployment using the existing rabbitmq-server charm on focal will already support the creation of quorum queues. The replication of the queues needs to be managed externally in the event that rabbitmq server cluster is expanded or shrunk. This spec covers the features needed to manage quorum queues throughout the rabbit clusters life-cycle. For example if a rabbitmq-server unit is lost and it was hosting a mirror of a quorum queue then a new mirror on another unit should be added.

NOTE: When a client connects to rabbitmq-server via the amqp relation they declare the vhost that they want access to. The rabbitmq-server charm ensures the vhost exists and provides the client with credentials to use the vhost. It is up to the client to create any queues it needs and it is at the point that the queue is created that its type is defined.

Proposed Change

Add functionality to the peer joined, departed and broken hooks to rebalance queues in the event of a peer being added or lost. These hooks will most likely utilise the add_member, delete_member, grow and shrink options to the rabbitmq-queues command.

Charm actions to manually add or remove members from a quorum queue should also be added to aid with pro-active maintenance of the cluster like taking a unit down for maintenance.

Alternatives

Classic queues are still supported so an alternative would be to continue to use them in all cases.

Implementation

Assignee(s)

TBD

Gerrit Topic

Use Gerrit topic “quorum_queues” for all patches related to this spec.

git-review -t quorum_queues

Work Items

Add functions to rabbit utils to identify all quorum queues across all vhosts.
Add function to rabbit utils to report which nodes are hosting a queue.
Extend peer departed hook to identify which queues have a mirror hosted on the leaving node and for each queue add a mirror on another node if possible.
Report queues with insufficient mirrors in charms workload status.
Add actions to allow queues mirrors to be moved.

Repositories

No new repositories needed

Documentation

Actions will need to be documented.

Security

N/A

Testing

Add functional test to create quorum queues and check there are sufficient replicas after cluster is shrunk and expanded.

Dependencies

For OpenStack to utilise quorum queues Oslo Messaging Quorum Queues needs to land and each OpenStack client charm will need to grow support for the rabbit_quorum_queue option.

Release Notes Migration to Reno

Thu, 26 Oct 2023 00:00:00

The OpenStack Charms project has a ever increasing velocity with improvements and new features being committed by disperse group of people each cycle.

Currently we manage our release notes in a central document and it is typically put together manually at the end of each cycle.

Problem Description

Putting together the release notes document involves manual labor and the current process is error prone with high risk of leaving important information out of the release notes at release time.

The upstream OpenStack release process has been heavily automated in recent cycles and we are currently barred from announcing our release notes to the new release-announce mailing list. Our repositories must be aligned with upstream automated release tools to gain access. Modernizing the way we do release notes is one of the steps needed to accomplish this.

Proposed Change

To address these issues I propose we do a stepwise migration towards committing release notes along with code changes throughout our charm repositories.

Prior work we can make use of exists upstream in the Reno [0] project, with developer tooling, linters and gate jobs. The use of this methodology and tooling is widespread among other OpenStack projects.

As a first step I propose we enable the use of reno on the following repositories during the cycle leading up to the 18.11 Charm release:

charm-keystone
charm-nova-compute

After the 18.11 Charm release we evaluate and decide on further adoption.

0: https://docs.openstack.org/reno/latest/

Alternatives

None defined.

Implementation

Assignee(s)

Primary assignee:: fnordahl

Gerrit Topic

Use Gerrit topic “charm-reno” for all patches related to this spec.

git-review -t charm-reno

Work Items

Add necessary boiler plate for use with reno to affected charm repositories
Add tox environment that enables developer to run reno tooling from virtual environment, relieving developer from the responsibility of installing and maintaining it in their developer environment.
Add tox environment that enables gating on existence and correctness of release notes artifacts.
Research ways to best coordinate and extract release notes artifacts from multiple charm repositories and combine them in one (charm-guide) at release time.
Create preliminary guidelines for use of developers and reviewers.

Repositories

No new git repositories are needed.

Documentation

Preliminary guidelines for use of developers and reviewers must be created. Communication with the community is also necessary.

Security

This change does not introduce any additional security risks.

Testing

The process for creating release notes files and committing them with code changes will be discussed with the community as it is implemented.

Gate jobs checking for existence and correctness of the release notes will be implemented.

Dependencies

Reno - https://docs.openstack.org/reno/latest/

Service Discovery

Thu, 26 Oct 2023 00:00:00

Many optional services may now be deployed as part of an OpenStack Cloud, with each service having a different optional features that may or may not be enabled as part of a deployment.

Charms need a way to discover this information so that services can be correctly configured for the options chosen by the charm user.

Problem Description

Charms need to be able to determine what other services are deployed within an OpenStack Cloud so that features can be enabled/disabled as appropriate.

Examples include:

notifications for ceilometer (really don’t want notifications enabled when ceilometer is not deployed).
misc panels within the openstack dashboard (fwaas, lbaas, l3ha, dvr etc… for neutron).
notifications for designate (disable when designate is not deployed).

Services and features of services are determined by the API endpoint charms that register them into the service catalog via the keystone charm.

Proposed Change

The keystone charm will expose a new provides interface ‘cloud-services’ which is a rich-ish description of the services deployed with registered endpoints.

The identity-service relations would also advertise the same data as the cloud-services relations so that charms already related to keystone don’t have to double relate (identity-service is a superset of cloud-services).

By default, a registered endpoint of type ‘A’ will result in service type ‘A’ being listed as part of the deployed cloud on this interface.

Services may also enrich this data by providing ‘features’ (optional) alongside their endpoint registration - these will be exposed on the cloud-service and identity-service relations.

Data will look something like (populated with real examples - key and proposed values):

services: ['object-store', 'network', 'volumev2', 'compute',
           'metering', 'image', 'orchestration', 'dns']

Example - advertise features supported by the networking service, allowing features to be enabled automatically in the dashboard:

network: ['dvr', 'fwaas', 'lbaasv2', 'l3ha']

Example - allow ceilometer to know that the deployed object-store is radosgw rather than swift:

object-store: ['radosgw']

Values will be parseable as a json/yaml formatted list.

By using the basic primitive of tags, we get alot of flexibility with type/feature being easily express-able.

Interface will be eventually consistent in clustered deployments - all keystone units will present the same data.

Alternatives

Each charm could query the keystone service catalog; however this is very much a point in time check, and the service catalog may change after the query has been made. In addition the keystone service catalog does not have details on what optional features each service type may have enabled and keystone services will be restarted during deployment as clusters get built out etc.

Implementation

Assignee(s)

Primary assignee:: james-page

Gerrit Topic

Use Gerrit topic “service-discovery” for all patches related to this spec.

git-review -t service-discovery

Work Items

Core (keystone charm):

Add cloud-services relation to keystone charm
Add service and feature discover handler to keystone charm
Update keystone interface to advertise services and features in keystone charm.
Create cloud-services reactive interface

Enablement:

Update ceilometer charm for radosgw discovery.
Update openstack-dashboard charm to automatically enable panels for deployed services and features.
Update neutron-api charm for designate discovery.
Update cinder charm for ceilometer discovery.
Update glance charm for ceilometer discovery.
Update neutron-api charm for ceilometer discovery.
Update radosgw charm to advertise ‘radosgw’ feature.
Update neutron-api charm to advertise networking features.

Repositories

No new git repositories required.

Documentation

This change is internal for use across the OpenStack charms, no documentation updates are required for end-users.

Security

No security implications for this change.

Testing

Implementation will include unit tests for all new code written; amulet function tests will be updated to ensure that feature is being implemented correctly across the charm set.

Dependencies

No external dependencies

Extended Swift Cluster Operations

Thu, 26 Oct 2023 00:00:00

The Swift charms currently support a subset of operations required to support a Swift cluster over time. This spec proposes expanding on what we already have in order to support more crucial operations such as reconfiguring the rings post-deployment.

Problem Description

To deploy a Swift object store using the OpenStack charms you are required to deploy both the swift-proxy and swift-storage charms. The swift-proxy charm performs two key roles - running the api endpoint service and managing the rings - while the swift-storage charm is responsible for the running the swift object store services (account, container, object).

As they stand, these charms currently support a base set of what is required to effectively maintain a Swift cluster over time:

deploy swift cluster with configurable min-part-hours, replica count, partition-power and block storage devices.
once deployed the only changes that can be made are the addition of block devices and modification of min-part-hours. Changes to partition-power or replicas are ignored by the charm once the rings have already been initialised.

This forces operators to manually apply changes like adjusting the partition-power to accommodate for additional storage added to the cluster. This poses great risk since manually editing the rings/builders and syncing them across the cluster could easily conflict with the swift-proxy charm’s native support for doing this resulting in a broken cluster.

Proposed Change

The proposal here is to extend the charm support for ring management in order to be able to support making changes to partition-power, replicas and possibly others and have the charm safely and automatically apply these changes and distribute aross the cluster.

Currently we check whether the rings are already initialised and if they are we ignore the partition power and replica count configured in the charm i.e. changes are not applied. To make it possible to do this we will need to remove these blocks and implement the steps documented in [0] and [1]. I also propose that charm impose a cluster size limit (number of devices) above which we refuse to make changes until the operator has paused the swift-proxy units i.e. placed them into “maintenance mode” which will shutdown the api services and block any restarts until the units are resumed. The user will also have the option to set disable-ring-balance=true if they want check that their changes have been applied successfully (to the builder files) prior to having the rings rebuilt and sycned across the cluster.

For the swift-storage charms, where currently one can only add devices but not remove, the proposal is to support removing devices. This will entail messaging the swift-proxy on the storage relation which an updated list of devices and a new setting ‘purge-missing-devices’ to instruct the swift-proxy to remove devices from the ring that are no longer configured. We will also need to ensure that the device cache located on the swift-storage unit from which we are removing a device is also updated to no longer include the device since not doing so would block the device from being re-added in the future. As an extension to this we should also extend the swift-storage-relation-broken to support removing devices associated with that unit/host from the rings and sycning these changes across the cluster.

[0] https://docs.openstack.org/swift/latest/ring_partpower.html [1] https://docs.openstack.org/swift/latest/admin/objectstorage-ringbuilder.html#replica-counts

Alternatives

Juju charm actions are another way to implement operational actions to be performed on the cluster but do not necessarily fit all cases. Since ring management is at the core of the existing charm (hook) code itself, the proposal is to extend this code rather than move and rewrite it as an action. However, there will likely be a need for some actions to be defined as post-modification checks and cleanups which would be well suited to an action and not directly depend on the charm ring manager.

Implementation

Assignee(s)

Primary assignee:: hopem

Gerrit Topic

Use Gerrit topic swift-charm-extended-operations for all patches related to this spec.

git-review -t swift-charm-extended-operations

Work Items

add support for modifying partition-power
add support for modifying replicas
add support for removing devices
add support for removing an entire swift-storage host

Repositories

None

Documentation

All of the above additions will need to be properly documented in the charm deployment guide.

Security

None

Testing

Each additional level of support will need very thorough testing against a real Swift object deployed with the charms that contains data and is of a reasonable scale. All code changes will be accompanied by unit tests and where possible functional tests.

Dependencies

None

The Title of Your Specification

Thu, 26 Oct 2023 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

Ceph RADOS Gateway (RGW) Multi-site Sync Policies

Wed, 25 Oct 2023 00:00:00

Multi-site sync policies provide fine-grained control of data movement between buckets in different zones. The sync policy can be configured at the zonegroup level, but it can also be configured at the bucket level.

By default, no sync policies are configured in the multi-site scenario, and all the buckets are synced from the primary zone to the secondary zones.

The goal is to allow selective sync of buckets between zones. This is important, for security purposes, when configuring secondary zones in the multi-site scenario.

More info about multi-site sync policies here: https://docs.ceph.com/en/latest/radosgw/multisite-sync-policy.

Problem Description

The current ceph-radosgw charm does not support configuration of multi-site sync policies. At the moment, we do not have the option to stop particular buckets syncing from primary zone to secondary zones.

Adding sync policies to the ceph-radosgw charm will give us more control over how the data is synced from primary RGW zone to secondary RGW zones. This can be generally useful for syncing only essential buckets to the secondary cluster. It is particularly valuable for one-way data synchronizations.

Proposed Change

Allow configuration of a default zonegroup sync policy.

This would allow all the buckets sync disabled by default, in the zonegroup. And, for particular buckets, having the possibility to enable syncing to secondary zones via bucket level sync policies. This is very useful when having selective data sync to secondary RGW zones.

By default, the ceph-radosgw charm will not configure any sync policies to maintain backwards compatibility with the existing functionality.

Sync policies can be created by the command radosgw-admin sync group create. A sync policy group can be in three states:

enabled, sync is allowed and enabled.
allowed, sync is allowed (but not enabled).
forbidden, sync is not allowed.

In the sync policy group, we need to configure:

data-flows, via command radosgw-admin sync group flow create.
pipes, via command radosgw-admin sync group pipe create.

A data-flow defines the flow of data between the different zones, and it can define:

symmetrical data flow, in which multiple zones sync data from each other.
directional data flow, in which the data moves in one way from one zone to another.

A pipe defines the actual buckets that can use these data flows, and the properties that are associated with it (for example: source object prefix).

Alternatives

None

Implementation

Two new configs will be added to the ceph-radosgw charm:

sync-policy-state, allows to configure the state of the default sync policy group (enabled, allowed, or forbidden). This will be configured in the primary ceph-radosgw application, and it will dictate the default zonegroup sync policy. This config will be empty, by default. If it’s empty, sync policies are not configured in the zonegroup. This way, we maintain backwards compatibility with the existing functionality of the charm.
sync-policy-flow-type, defines the data flow type that will be configured in the default sync policy group between primary zone and a secondary zone. This config will be symmetrical by default, and it is used only when the primary ceph-radosgw application has config option sync-policy-state non-empty, and multi-site sync policies are configured.

Implement the primary-relation-changed hook, where we configure the default sync policy (if the sync-policy-state config is set). If the default policy is created:

A data flow with each secondary zone is configured. The data flow type is obtained from the sync-policy-flow-type config passed on the relation data by the secondary ceph-radosgw applications.
A pipe with each secondary zone is configured. The pipe allows all the buckets to use the previously defined data flow.

Three Juju actions will be added to the ceph-radosgw charm:

enable-buckets-sync, takes a comma-separated list of buckets to enable sync with bucket level sync policy. This is meant to be used in conjunction with allowed sync-policy-state config, which allows buckets syncing, but it’s disabled by default.
disable-buckets-sync, takes a comma-separated list of buckets to forbid sync with bucket level sync policy. This is useful when you want to disable syncing for some buckets only, regardless of the default zonegroup sync policy.
reset-buckets-sync, takes a comma-separated list of buckets to reset bucket level sync policy. After this is executed, the buckets will be synced according to the default zonegroup sync policy.

If the Juju actions are executed on the primary ceph-radosgw application, any bucket level sync policies are automatically propagated from the primary RGW site to all the secondary RGW sites.

On the other hand, if we execute the Juju actions on the secondary ceph-radosgw applications, the bucket level sync policies are not propagated to any other RGW site.

All these new Juju actions can be executed only on the primary ceph-radosgw application. They will fail if they are executed on the secondary ceph-radosgw applications.

Assignee(s)

Primary assignee: ionutbalutoiu

Gerrit Topic

Use Gerrit topic “ceph-radosgw-multisite-sync-policies” for all patches related to this spec.

git-review -t ceph-radosgw-multisite-sync-policies

Work Items

Add two new charm configs to ceph-radosgw:
- sync-policy-state, which allows to configure the state of the default sync policy group.
- sync-policy-flow-type, defines the type of data flow that will be configured in the default sync policy group between the primary zone and a secondary zone.
Implement the primary-relation-changed hook with all the details described in the Implementation section.
Add three Juju actions to the ceph-radosgw charm:
- enable-buckets-sync, takes a comma-separated list of buckets to enable sync with bucket level sync policy.
- disable-buckets-sync, takes a comma-separated list of buckets to forbid sync with bucket level sync policy.
- reset-buckets-sync, takes a comma-separated list of buckets to reset bucket level sync policy.

Repositories

https://opendev.org/openstack/charm-ceph-radosgw

Documentation

The new charm configs, and Juju actions will be documented in the ceph-radosgw charm.

Also, additional documentation to charm deployment guide should be added for the new optional multi-site sync policies.

Security

If used, this will improve the security of multi-site Ceph RGW deployment.

Testing

Code written or changed will be covered by unit tests; functional testing will be implemented using the Zaza framework.

Dependencies

No new dependencies.

Unify ceph nova-compute credentials

Tue, 24 Oct 2023 00:00:00

Ceph credentials should be the same across all nova-compute charm apps to allow live-migration of VMs booted from image (and stored in ceph via use of libvirt-image-backend=rbd config) to succeed.

Problem Description

Currently every nova-compute charm app registers a ceph credential named after the charm app itself. Therefore, if a cloud has two nova-compute charm apps named nova-compute-haswell and nova-compute-skylake, then their ceph credentials will be the respective names.

The result of this is that live-migrating a VM booted from image where both nova-compute charm apps have been configured to use libvirt-image-backend=rbd (resulting in the image being stored in ceph) fails because the libvirt XML for the instance will have the ceph credential and key associated with the name on the source, to which the destination does not possess the key to access the image, resulting in a failed migration due to permission denied error, as documented in [1].

Proposed Change

In order to solve the problem, all nova-compute charm apps need to have access to the same ceph credentials and by extent, to the same key, allowing credentials specified in the libvirt XML to be compatible with both the host and destination.

To achieve that, it is necessary to move away from having ceph credentials deriving their name from the nova-compute charm app names. Here we propose a new name to be a common one between all of them: ‘nova-compute-ceph-auth-<secret-uuid-first-block>’.

There are many reasons for choosing the above-mentioned new credentials name:

As part of the name transition, it is necessary to make changes to the libvirt secret entry. The secret entry in libvirt consists of:

secret UUID - name/usage - key

For each registered libvirt secret, we can only update the key. In order to allow continued function of existing VMs as we upgrade the charms performing the credential transition, we cannot delete and replace the entire secret. It is necessary then to add a new secret with the new credentials and new UUID, while also preserving the old secret.
The upgrade path from the old name to the new name is simpler if every single existing name is handled as old. For example, if we chose ‘nova-compute’ to be the new name, then deployments that had nova-compute charm apps named ‘nova-compute’ would need to be handled differently. The main technical difficulty of this approach is again, the libvirt secret management limitations, preventing us from adding a new libvirt secret with the same name and a new UUID, nor can we achieve the desired functionality while avoiding the need of adding a new secret UUID.
Having a credential name that is very unlikely to clash with any charm app name avoids the situation described above. By using the first block of the secret UUID in the credential name we also make it easier to identify the credential as being the new one. If more credential names are required to be performed in the future, by following this pattern we can easily have versioning between them.

Charm Impact

Only the nova-compute charm is affected by the code changes. The ceph-mon charm does not require code changes.

Upgrade Impact

On upgrade, transitioning from old credentials to new credentials consists of:

Every nova-compute unit needs to send the new application-name to ceph-mon. Ceph-mon will then register the new credential when receiving it for the first time (subsequent receipt of the new application-name will not need to re-register the new credentials, as they would already exist) and send back the key of the new credentials.
Nova-compute units, upon receiving the updated key, will rewrite the config files with the new credentials, keys and UUID, and invoke libvirt to register the new secret. New VMs created from this point onwards will use the new credentials.

The nova-compute units will retain their old credentials key file in their /etc/ceph folder and registered in libvirt, so existing workloads are not affected and allows the units to be able to receive live-migration of non-rebooted instances.

Existing Workloads Impact

Existing running VMs will have their old credentials declared in their libvirt XML and continue to run fine as the old credentials as retained. A full power cycle of those VMs is required to update their libvirt XML and switch to the new credentials. Those VMs cannot be migrated to nova-compute nodes operated by different nova-compute charm apps until rebooted.

Scale-out Impact

New nodes deployed with the updated charm need to install both old and new credentials. To achieve this, the nova-compute code will need to send the old application-name on ceph-relation-joined, then upon successful configuration of the old credential, send the new application-name and trigger the upgrade path. The end result is that a newly deployed node will have both old and new credentials and can receive migration of non-rebooted VMs that are still using the old credentials. This was a concern already pointed out by [2].

Revert/Downgrade Possibility

Unfortunately, downgrading and reverting the changes in case something unexpected happens is not smooth, but is fairly straightforward. The main problem is that the current code does not have a way to re-send the application-name to ceph-mon outside of the ceph-relation-joined hook function, so this information has to be exchanged manually through the juju relation-set command. This is enough to revert all the changes.

If something went wrong during the upgrade or downgrade, then libvirt secrets may need to be manually maintained to address a potential error or conflict of names, UUID, or re-set the keys.

Charm Configuration Options

No charm config options are affected or have to be added for this change. The change, however, only affects users that have the config libvir-image-backend set to ‘rbd’.

Configuration Files

The config files are affected in the following way:

/etc/ceph/secret.xml - This file will be updated to have the new UUID and credential name. It is only used for the secret to be registered in libvirt once.
/etc/ceph/ceph.client.nova-compute-ceph-auth-<secret-uuid-first-block> - This file will be added with the key for the new credentials.
/etc/nova/nova.conf - The properties rbd_user and rbd_secret_uuid will be updated with the new credentials and UUID respectively.

Non-Charm Configuration

The relation-data changes between the nova-compute and ceph charms affect the following 2 fields:

application_name - Ceph-mon will receive the application name nova-compute-ceph-auth-<secret-uuid-first-block> instead of the nova-compute charm app name.
key - Nova-compute will receive the new credential’s key instead of the old credential’s key.

OpenStack Versions

This feature will be enabled for Yoga and newer OpenStack releases.

Operating System Versions

This feature will be enabled for Ubuntu 20.04 (focal) and newer Ubuntu releases.

Juju Version Dependencies

This feature has no dependency on Juju versions.

Alternatives

A design alternative is possible to achieve the same result, although with some advantages and disadvantages:

Change the relation-data to exchange a dictionary containing all nova-compute charm app names and keys between all nova-compute units. To achieve this, either the nova-cloud-controller needs to be involved (as it already currently is for exchanging SSH keys), or a new relation needs to be created between nova-compute between different charm apps. This alternative requires more code changes and relation data structure changes, but the end result is generally more consistent and resilient against unexpected behavior, such as hook errors or hooks running out of order.

Implementation

Assignee(s)

Primary assignee:: ganso

Gerrit Topic

Use Gerrit topic “lp2028559” for all patches related to this spec.

git-review -t lp2028559

Repositories

No new repositories are required for this work.

Documentation

As part of this effort, the following documentation will need to be updated:

Charm Guide
Release Notes

Security

The changes required in the charm do not introduce any additional security implications beyond the security requirements and compromises already in place.

The extra credentials in ceph and extra keys in nova-compute are subject to the same vulnerability as the existing credentials and keys.

Testing

Unit tests and functional tests will be implemented for this feature. The functional tests will validate whether the nova-compute nodes have credential files for both the name derived from their charm app name, and the new proposed unique name.

Currently there are not multiple nova-compute charm apps configured in the CI, nor their presence supported by the functional tests code, nor any live-migration tests. As future work there could be live-migration tests across different nova-compute charm apps.

Work Items

Implement code changes in nova-compute charm
Add functional tests to zaza-openstack-tests
Provide user documentation on impact of changes

Dependencies

No hardware, software or version dependencies are required for this change to be functional.

References

Ceph RADOS Gateway (RGW) Cloud Sync

Mon, 23 Oct 2023 00:00:00

Ceph RGW has a module called Cloud Sync which allows syncing zone data to a remote cloud service. The sync is unidirectional, data is not synced back from the remote zone. The goal of this module is to enable syncing data to multiple cloud providers. The currently supported cloud providers are those that are compatible with AWS (S3).

More info about Ceph Cloud Sync here: https://docs.ceph.com/en/latest/radosgw/cloud-sync-module.

The Cloud Sync module is built atop of the multi-site framework that allows for forwarding data and metadata to a different external tier.

More info about Ceph sync modules here: https://docs.ceph.com/en/quincy/radosgw/sync-modules.

Problem Description

The current ceph-radosgw charm does not support the cloud-sync module. Given the fact that the cloud-sync module is built atop of the multi-site framework, we can leverage the existing radosgw-multisite Juju relation interface.

The cloud-sync module is enabled via a new relation with the primary ceph-radosgw application. The deployment is similar with the existing RGW multi-site replication steps: https://ubuntu.com/ceph/docs/setting-up-multi-site.

The only differences being:

Both primary-ceph-radosgw and secondary-ceph-radosgw (with the cloud-sync enabled zone) are related to the same Ceph cluster.
When data is replicated from primary-ceph-radosgw zone, the secondary-ceph-radosgw zone will write into a remote S3, instead of Ceph storage. The secondary-ceph-radosgw zone will have the appropriate S3 credentials configured for this task.
Data sync is unidirectional, therefore the secondary-ceph-radosgw zone will be read-only.

More info about how to configure the cloud-sync module here: https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/#how-to-configure.

Proposed Change

Add a new relation, called cloud-sync, to the ceph-radosgw charm. The new relation will implement the existing radosgw-multisite relation interface.

In the new cloud-sync relation, when the secondary multi-site secondary zone is created, we need to pass --tier-type=cloud to the radosgw-admin zone create command in order to have the cloud-sync module enabled. Besides this, we need to add the S3 target credentials via --tier-config parameter of the radosgw-admin zone modify command.

These steps are documented at: https://docs.ceph.com/en/latest/radosgw/cloud-sync-module.

The Ceph cloud-sync module allows multiple S3 targets to be configured in the same zone tier config. For this, we have profiles in the tier config. Each profile maps a single source bucket (or multiple buckets via prefix) to one S3 destination (Many-To-One mapping). The profiles in the tier config are optional.

A profile contains info about:

source_bucket, either a bucket name, or a bucket prefix (if ends with *) that defines the source bucket(s) for this profile.
target_path, a string that defines how the target path is created. The target path specifies a prefix to which the source object name is appended. The target path is configurable, and it can include any of the following variables:
- $sid: unique string that represents the sync instance ID
- $zonegroup: the zonegroup name
- $zonegroup_id: the zonegroup ID
- $zone: the zone name
- $zone_id: the zone id
- $bucket: source bucket name
- $owner: source bucket owner ID
For example: target_path = rgwx-${zone}-${sid}/${owner}/${bucket}
connection_id, ID of the connection (with credentials) that will be used for this profile.

A new charm config, called cloud-sync-target-path, will be added to configure the target path for all the profiles. This allows a consistent target path for the configured cloud-sync zone.

The profiles are configured through the use of s3-integrator Juju applications together with the new config option cloud-sync-target-path.

It is mandatory to have a default S3 target for all the buckets that don’t have a profile configured. The rationale is that every bucket needs to have a sync target, and the default target is the fallback for any bucket that doesn’t have a profile configured. A new charm config will be added, called cloud-sync-default-s3-target for this purpose.

It is obvious that we need to handle S3 credentials for the S3 targets configured in the cloud-sync zone. For this purpose, we will use the s3-integrator charm (https://github.com/canonical/s3-integrator). The ceph-radosgw charm will have new relation with the s3-integrator charm.

Each deployed application of the s3-integrator charm will handle credentials for a single S3 target. When relating multiple s3-integrator applications to the same secondary-ceph-radosgw cloud-sync application, the tier config will be updated with profiles for each S3 target.

For example, the following Juju deployment commands:

#
# Assuming ceph-mon is already deployed
#
juju deploy ceph-radosgw primary-ceph-radosgw \
  --config realm=eu \
  --config zonegroup=east \
  --config zone=primary

juju relate ceph-radosgw:mon ceph-mon:radosgw

juju deploy ceph-radosgw secondary-ceph-radosgw \
  --config realm=eu \
  --config zonegroup=east \
  --config zone=primary-cloud-sync \
  --config 'cloud-sync-target-path=${bucket}' \
  --config cloud-sync-default-s3-target=minio-dev

juju relate secondary-ceph-radosgw:mon ceph-mon:radosgw

juju deploy s3-integrator minio-dev \
  --config endpoint=http://10.7.133.248:9000 \
  --config region=us-east-1 \
  --config s3-uri-style=path

juju deploy s3-integrator minio-production \
  --config endpoint=http://10.7.133.250:9000 \
  --config region=us-east-2 \
  --config s3-uri-style=path \
  --config 'bucket=production*'

juju relate ceph-radosgw-cloud-sync:s3-credentials minio-dev:s3-credentials
juju relate ceph-radosgw-cloud-sync:s3-credentials minio-production:s3-credentials

#
# After all applications' units are idle
#
juju relate ceph-radosgw-cloud-sync:cloud-sync ceph-radosgw:primary

juju run minio-dev/leader sync-s3-credentials --string-args access-key=MY_DEV_ACCESS_KEY secret-key=MY_DEV_SECRET_KEY
juju run minio-production/leader sync-s3-credentials --string-args access-key=MY_PROD_ACCESS_KEY secret-key=MY_PROD_SECRET_KEY

will render the following tier config in the cloud sync zone:

{
    // ...
    "name": "primary-cloud-sync",
    // ...
    "tier_config": {
        "connections": [
            {
                "id": "minio-dev",
                "endpoint": "http://10.7.133.248:9000",
                "region": "us-east-1",
                "host_style": "path",
                "access_key": "MY_DEV_ACCESS_KEY",
                "secret": "MY_DEV_SECRET_KEY"
            },
            {
                "id": "minio-production",
                "endpoint": "http://10.7.133.250:9000",
                "region": "us-east-2",
                "host_style": "path",
                "access_key": "MY_PROD_ACCESS_KEY",
                "secret": "MY_PROD_SECRET_KEY"
            }
        ],
        "profiles": [
            {
                "connection_id": "minio-production",
                "source_bucket": "production*",
                "target_path": "${bucket}"
            }
        ],
        "connection_id": "minio-dev",
        "target_path": "${bucket}"
    },
    // ...
}

Alternatives

None

Implementation

Assignee(s)

Primary assignee: ionutbalutoiu

Gerrit Topic

Use Gerrit topic “ceph-radosgw-cloud-sync” for all patches related to this spec.

git-review -t ceph-radosgw-cloud-sync

Work Items

Add two new charm configs to ceph-radosgw:
- cloud-sync-default-s3-target, the default S3 target for buckets that don’t have a profile configured in the tier config.
- cloud-sync-target-path, string that defines how the target path is created. The target path specifies a prefix to which the source object name is appended.
Add a new relation called cloud-sync to the ceph-radosgw charm. The new relation implements the existing radosgw-multisite interface. The cloud-sync secondary zone will be configured with --tier-type=cloud, and connection info for the S3 targets will be fetched from the relation with the s3-integrator charm.

When the cloud-sync relation is established, the ceph-radosgw cloud-sync application will be blocked until a relation with the s3-integrator application is created, which provides S3 credentials for the configured cloud-sync-default-s3-target.
Add a new relation called s3-credentials, implementing s3 interface, used to fetch S3 credentials for each S3 target in the cloud-sync tier config.

The name of related s3-integrator application will be used as the profile name configured in the tier config. From the relation data, we also fetch the source bucket(s) for each profile.

Repositories

https://opendev.org/openstack/charm-ceph-radosgw

Documentation

The config options (cloud-sync-default-s3-target and cloud-sync-target-path) will be documented in the ceph-radosgw charm.

Also, additional documentation to charm deployment guide should be added for the new cloud-sync relation.

Security

ceph-radosgw
- The Ceph Cloud Sync module requires S3 connection credentials for the configured S3 targets. These credentials are fetched from the s3-credentials relation with an application that implements the s3 relation interface.

Testing

Code written or changed will be covered by unit tests; functional testing will be implemented using the Zaza framework.

Dependencies

No new dependencies.

The Title of Your Specification

Tue, 23 May 2023 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

The Title of Your Specification

Tue, 23 May 2023 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

Swift Backup Driver for Cinder

Tue, 08 Nov 2022 00:00:00

It is possible to use swift API for cinder backup purposes. In one of the configurations the swift API provider is deployed via non-Juju means. This spec is aimed to address this use-case specifically.

Problem Description

Cinder has support for backing up volumes via Swift API. In order to use it a URL is needed along with authentication details.

The following methods are supported to use to authenticate against a swift endpoint:

v2 auth (URL, username, password, tenant);
v3 auth (URL, username, password, project domain, project, user domain, username, password);

The backup to swift functionality is supported on all versions of OpenStack supported at the time of writing. It was added in 2013.1

Backup configuration should be added to cinder.conf and can be propagated from the cinder-backup-like charm to the cinder charm. Extending the existing cinder-backup charm is not favorable as it is ceph-specific.

Config options for the swift backend are present here

Proposed Change

Implement a new subordinate charm in reactive framework. This charm will be deployed alongside cinder deployments. This will be a subordinate charm to cinder, that installs cinder-backup and configures cinder to use swift as a backend for volume backups.

config options for configuring the swift backend. Following configs need to be modeled in the charm.
- backup_swift_url
- backup_swift_auth_url
- backup_swift_user
- backup_swift_key
- backup_swift_auth_version
- backup_swift_tenant
- backup_swift_user_domain
- backup_swift_project_domain
- backup_swift_project
- backup_swift_container
- backup_swift_object_size
- backup_swift_block_size
- backup_swift_ca_cert_file

Implement a new interface cinder-backup in reactive framework for sending cinder-backup external backend configuration.

Alternatives

extend cinder-backup to support swift options (discarded as it is already Ceph-specific);
extend cinder-backup to support clusters not deployed via charms by creating proxy charms (discarded as cinder-backup is Ceph-specific and proxy charms add additional complexity).

Implementation

Assignee(s)

Primary assignee:: yoshikadokawa alitvinov

Gerrit Topic

Use Gerrit topic “cinder-backend-swift” for all patches related to this spec.

git-review -t cinder-backend-swift

Work Items

Implement a new reactive subordinate charm, cinder-backup-swift
Implement a new reactive charm interface, cinder-backup
write unit tests;
write functional tests using the zaza test framework (a functional tests should include deploying swift via swift-proxy and swift-storage charms with config options used to connect to it from the cinder-backend-swift charm).

Repositories

The existing classic-style charm in the following repository:

openstack/charm-cinder-backup-swift

will be replaced by the new reactive charm.

New git repository:

openstack/charm-interface-cinder-backup

Documentation

The charm should contain documented options and authentication methods for the target Swift cluster.

Security

The Swift endpoint might be TLS-terminated, therefore, an option to provide a CA certificate is required.

Testing

unit tests;
functional tests (as mentioned above).

Dependencies

This charm will support OpenStack Queens and Ubuntu 18.04 Bionic as its baseline
A new project will need to be created based on the OpenStack Project Creator’s Guide

NVidia Virtual GPU support in Nova

Tue, 08 Nov 2022 00:00:00

Graphical Processing Units (GPUs) are optimized for performing various mathematical operations in silicon. These optimizations allow for extremely fast floating point operations that benefit various workloads such as graphics rendering, machine learning and others. Hardware manufacturers such as Nvidia [0] and Intel [1] offer physical GPU (pGPU) cards that can be virtually partitioned into virtual Graphics Processing Units (vGPUs), enabling multiple guests to make use of a single physical GPU installed in the system.

Recent changes in Nova [2][3][4][5] introduced partial support for exposing physical GPUs as virtual GPUs to Nova instances. This spec is about enabling this support for NVidia hardware in a charm deployment.

Problem Description

Machine Learning workloads benefit heavily from the hardware capabilities of GPUs. The common configuration for running ML based workloads leverage Kubernetes, which has poor multi-tenancy support and is often run on top of OpenStack to provide elastic resources and provide multi-tenant isolation.

Currently, deployments of Charmed OpenStack must use GPU passthrough [6] in order to enable a guest to make use of a GPU. This limits deployments to require one physical card for each guest that they intend to run on the hypervisor. vGPU support enables multiple guests to make use of the installed pGPUs which enables deployments to increase the number of guests which can leverage the system.

Proposed Change

The Nova project within OpenStack has provided vGPU support since the Queens release. [2][3] The Nova project continues to iterate over the design of vGPU support, making enhancements to allow multiple vGPU types in the Ussuri release. [4][5]

The Nova project provides support for vGPUs for allocation and assignment. The vGPU support upstream has evolved and thus, there are different capabilities available in different releases of OpenStack. The primary difference is whether or not multiple GPU devices are supported and how to specify which devices should be used for virtual GPU types.

A new nova-compute-nvidia-vgpu charm, subordinate to nova-compute, will be created. It will install the NVidia host software and set it up if and only if the presence of one or several such pGPU cards is detected. Indeed deploying the subordinate charm “blindly” to all compute hosts, no matter which of them provide NVidia pGPU cards or not, simplifies Day-0 and Day-1 operations. If no supported pGPU card is found, the subordinate charm will simply state so in its unit status message.

The proprietary software package (.deb) [7] to be installed on host will be passed to the subordinate charm as a charm resource.

To configure vGPUs, a new config option vgpu-device-mappings will be added to the nova-compute-nvidia-vgpu charm. The format of this configuration value will be a YAML-formatted dict (or JSON-formatted, since JSON is valid YAML) where the key is the vgpu_type and the value is a list of device addresses. E.g.

juju config nova-compute-nvidia-vgpu vgpu-device-mappings="$(cat << EOF
vgpu_type1:
  - device_address1
  - device_address2
vgpu_type2:
  - device_address3
EOF
)"

is equivalent to

juju config nova-compute-nvidia-vgpu vgpu-device-mappings=\
    "{'vgpu_type1': ['device_address1', 'device_address2'], 'vgpu_type2': ['device_address3']}"

vGPU support will be added to the following Operating System versions:

Bionic
Focal

vGPU support will be enabled for the following OpenStack releases:

Queens (bionic only)
Rocky (migration path only)
Stein (migration path only)
Train (migration path only)
Ussuri
Victoria
Wallaby
Xena
Yoga (and newer)

Juju 2.9 or newer is required.

Alternatives

Using PCI passthrough [6] virtual machines can have direct access of the physical GPUs installed in the compute host. This limits however deployment to require one physical card for each guest.

Implementation

Assignee(s)

Primary assignee:

Aurelien Lourot <lp:aurelien-lourot>

Gerrit Topic

Use Gerrit topic “vgpu-support” for all patches related to this spec.

git-review -t vgpu-support

Work Items

The implementation consists of the following items:

Create a nova-compute-nvidia-vgpu subordinate charm with a vgpu-device-mappings config option.
Let the subordinate charm feed the principal charm with:
- vGPU-related Nova configuration bits, using a subordinate_configuration key in order to pass a JSON blob [8][9] over a new nova-vgpu relation. The JSON blob will be a 1:1 mapping of the wanted Nova configuration, [3][5] essentially containing the enabled vGPU types and the device addresses for each vGPU type.
- Data around installed packages and services in order to properly implement pausing, resuming and upgrading (examples of this mechanism can be found here [10][11]).
Let the subordinate charm install the proprietary host software [7] whenever the required hardware is found:
- in the install hook,
- after a reboot, either via a hook or an action.
Let the subordinate charm perform any necessary action for setting up the proprietary software. [7]
See “Documentation” and “Testing” sections below.

Repositories

charm-nova-compute
charm-nova-compute-nvidia-vgpu (new)
charm-guide
charm-deployment-guide

Documentation

Documentation will be written for charm-guide and/or charm-deployment-guide explaining the usage of this feature, i.e. how to use and configure the charms. Useful links around these topics will be provided:

guest flavors,
proprietary guest software, [7]
license server.

Security

Special care will be taken to prevent injecting arbritrary Nova configuration through the vgpu-device-mappings config option.

The proprietary host software [7] will only be installed if suitable hardware is found.

Testing

Gate tests (unit and functional tests) will be added in order to cover the new charm’s behaviour without any special hardware.

If needed and possible, functional tests exercising special GPU hardware and NVidia proprietary software [7] will be written and run periodically.

Dependencies

We depend on proprietary NVidia software [7] to be provided to the subordinate charm as a .deb resource file.

[0]: https://docs.nvidia.com/grid/5.0/pdf/grid-vgpu-user-guide.pdf
[1]: https://01.org/igvt-g
[2]: https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html
[3]: https://docs.openstack.org/nova/queens/admin/virtual-gpu.html
[4]: https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/vgpu-multiple-types.html
[5]: https://docs.openstack.org/nova/ussuri/admin/virtual-gpu.html
[6]: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-pci-passthrough-gpu.html
[7]: https://docs.nvidia.com/grid/index.html
[8]: https://opendev.org/openstack/charm-nova-compute/src/branch/master/hooks/charmhelpers/contrib/openstack/context.py#L1508
[9]: https://opendev.org/openstack/charm-ceilometer-agent/src/branch/master/hooks/ceilometer_hooks.py#L85
[10]: https://opendev.org/openstack/charm-nova-compute/src/branch/master/hooks/nova_compute_utils.py#L76
[11]: https://opendev.org/openstack/charm-ceilometer-agent/src/branch/master/hooks/ceilometer_hooks.py#L86

Virtual TPM Enablement

Tue, 08 Nov 2022 00:00:00

Increasingly, applications and Operating Systems are using TPM devices to store secrets. In order to run these application in a virtual machine, it is necessary to be able to expose a virtual TPM device within the guest.

Problem Description

Guests requiring access to TPMs for secret storage are unable to do so in an OpenStack Charms deployed cloud.

Proposed Change

Nova is able to provide virtual TPM devices to guests starting in the Victoria release [1], [2]. TPM devices are provided to libvirt/qemu guests via the swtpm library.

The nova-compute charm should be able to install and configure the necessary libraries for providing emulated TPM devices. It will do so by default for new installations and installations that upgrade to a version of the nova-compute charm which has the feature set enabled. This will cause the nova-compute service on the local machine to report that it has the COMPUTE_SECURITY_TPM_1_2 and COMPUTE_SECURITY_TPM_2_0 traits.

While the compute nodes will report that they have the necessary traits, new instances will not have TPM devices attached unless the flavor and/or image has the appropriate properties configured. It is considered an administrative decision to determine which images or flavors should have TPM devices enabled and is out of scope for this implementation.

The above also makes it generally safe to enable by default for users who upgrade their charms to a version that has this capability enabled. While it may be safe to enable by default, initial versions will ship with the feature disabled by default in order to prevent package installation errors on charm or OpenStack upgrade.

Charm Configuration Options

The following configuration options will be available on the nova-compute charm:

A new config option will be introduced in order to enable or disable vTPM support:

enable-vtpm:
  type: boolean
  default: False
  description: |
    Enable emulated Trusted Platform Module support on the hypervisors.
    A key manager, e.g. Barbican, is a required service for this
    capability to be enabled.

Configuration Files

The swtpm package in Ubuntu does not use the tss/tss user/group that is the default for qemu, nova, etc. Instead, the swtpm package configures the user/group as swtpm/swtpm as the swtpm user does not need the same level of permissions that the existing tss user has. This requires some additional changes to configuration files.

Enabling virtual TPM support using OpenStack charms will require the following configuration files to be updated:

/etc/libvirt/qemu.conf - the swtpm_user and swtpm_group values need to be set to the same users that the swtpm software package expects. This will cause the qemu configuration file to look as follows:

##########################################################################
# [ WARNING ]
# Configuration file maintained by Juju. Local changes may be overwritten.
##########################################################################

# File installed by Juju nova-compute charm
cgroup_device_acl = [
   "/dev/null", "/dev/full", "/dev/zero",
   "/dev/random", "/dev/urandom",
   "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
   "/dev/rtc", "/dev/hpet", "/dev/net/tun",
   "/dev/vfio/vfio",
]

swtpm_user = "swtpm"
swtpm_group = "swtpm"

/etc/nova/nova-compute.conf - similar to the qemu config changes, the nova services need to specify which user and group should be configured in libvirt for qemu instances. It will also have the global flag for enabled or not enabled:

##########################################################################
# [ WARNING ]
# Configuration file maintained by Juju. Local changes may be overwritten.
##########################################################################
[DEFAULT]
compute_driver=libvirt.LibvirtDriver
swtpm_enabled=True
swtpm_user=swtpm
swtpm_group=swtpm

Non-Charm Configuration

Enabling vTPM support in the nova compute charm will cause the nova hypervisor to report the COMPUTE_SECURITY_TPM_1_2 and COMPUTE_SECURITY_2_0 traits to the placement service. Additional steps need to be taken by the cloud operator/administrator in order to make this feature available to guests by configuring the appropriate properties on images or flavors.

Nova uses information from the extra specs configured on the flavor or properties set on an image in order to determine whether or not to add a vTPM device. As such, the hypervisor may be configured to have the necessary traits exposed to allow for a vTPM device, but the device will not be provisioned for a guest unless the operator appropriately configures the images and/or flavors.

Refer to the Nova documentation [2] for the specific extra specs and properties that need to be set to provide vTPM devices to guests.

Barbican

The swtpm library which provides emulated TPM devices encrypts secrets locally in files on the file system. Nova uses the Barbican key manager service for secret storage, which is already available as a charmed application.

Conveniently, the default configuration for Nova will use the barbican services from the keystone catalog to store the necessary secrets. These secrets are scoped per project and the interactions with the secret store will happen with appropriate context of the user. As such, there’s no additional information that the nova-compute charm requires in order to configure the nova compute services so additional relations are unnecessary.

OpenStack Versions

This feature will be enabled for Wallaby and newer OpenStack releases.

Operating System Versions

This feature will be enabled for Ubuntu 20.04 (focal) and Ubuntu 22.04 (jammy).

Juju Version Dependencies

This feature has no dependency on Juju versions.

Alternatives

There are no alternatives for vTPM support within the charms that integrates nicely with OpenStack while using the OpenStack charms for deployment.

Implementation

Assignee(s)

Primary assignee:: billy-olsen

Gerrit Topic

Use Gerrit topic “charm-vtpm” for all patches related to this spec.

git-review -t charm-vtpm

Work Items

Add configuration changes to nova-compute charm
Add functional tests to zaza-openstack-tests
Provide user documentation around enabling the feature and how to use

Repositories

No new repositories are required for this work.

Documentation

As part of this effort, the following documentation will need to be updated:

Charm Deployment Guide
Charm Readme
Charm Guide
Release Notes

Security

The changes required in the charm do not introduce any security implications above and beyond what is outlined in the Nova specification for enabling emulated vTPM devices [1].

Testing

Unit tests and functional tests will be implemented for this feature. The functional tests will validate the various TPM device configurations and validate that the TPM device is available within the guest.

Dependencies

Nova Wallaby version or greater.
swtpm TPM emulator [3] [4]
Focal-Wallaby support depends on swtpm package being backported to either the Wallaby Ubuntu Cloud Archive or Focal. Ubuntu developers have indicated a willingness to backport swtpm to Focal.

designate-bind: Configurable Service IPs

Tue, 08 Nov 2022 00:00:00

Original LP bug: https://bugs.launchpad.net/charm-designate-bind/+bug/1804057

The problem described by the Bootstack team is that when designate-bind [1] unit get redeployed, there is a chance that they’ll receive a new IP address. This is a problem if the previous IP address was used in other, non-juju, parts of the infrastructure to reference the DNS service provided by the designate (e.g. if the IPs were used in upstream DNS server for zone delegation or if they were used in forwarding/firewall rules). Suggested solution is to allow configuration of static IP addresses that would be preserved on the units even after redeployment.

Problem Description

The possibility of unit’s IP address change requires that each deployment needs to keep track of all the external systems that reference designate-bind units by the IP addresses and manually change these references whenever the unit gets redeployed. These manual actions present an opportunity for human error and introduce a delay to the update process as, for example, it takes time for SOA DNS records to propagate depending on their timeout values.

Proposed Change

The core of the solution should be a new config option in designate-bind charm that would allow to specify static IP addresses from reserved pool not controlled by the DHCP. This config option would be called service-ips and it would contain a list of comma separated IP addresses that could be statically configured on the “DNS frontend” facing interface of the designate-bind units in addition to the address assigned normally by the DHCP.

There are two alternative solutions suggested in the original launchpad issue and comments. One Solution would have the designate-bind charm handle the address assignment and distribution itself, the other suggested solution would be based on using hacluster subordinate charm [2] that would manage service IPs as a resource.

Hacluster-based solution offers certain advantages so it’s listed as a main proposed solution.

Hacluster based solution

This solution proposes to use the hacluster subordinate charm. The service-ips would be configured via juju config on the designate-bind charm, but designate-bind would not be in charge of managing the IPs. Instead, the relation with charm-hacluster would be used to create each service IP as a cluster resource in pacemaker [3]. These resources could be configured with colocation rules such as they would never be placed on the same host if there was another host without assigned Service IP. This can be achieved by specifying a negative colocation score for the resources [6] (see score calculations at [7]).

Another restriction for the Service IP cluster resource might be the actual network configuration on the designate-bind units. The Service IP must be within the network configured on the unit and multiple units do not necessarily need to be on the same network. Cluster resource constraint “location” [9] can be used to configure cluster resources to prefer or avoid certain hosts.

At the cost of adding load to the designate-bind unit by running charm-hacluster, this solution is rather elegant and keeps most of the logic and responsibility out of the charm’s code. Additional benefit is that even in degraded state, when some of the units are down, all the Service IPs would remain functional as the hacluster charm would move them to the unit that is up.

Alternatives

This alternative solution would manage everything within the designate-bind charm without addition of new dependencies.

As the designate-bind units support HA deployments and are typically deployed in pairs (e.g. ns1.example.org, ns2.example.org), we need to manage how the units will choose IP from the service-ips list. This responsibility could be handled by the leader unit. When the leader unit comes online, it can pick the first IP address from the list and mark it as “used”. Then, when additional units join the cluster, the service IP will be chosen for them by the leader, marked as used, and shared via relationship data. The callback in the leader unit that chooses the IP address from service-ips, should implement some form of lock, to prevent collision if two units join the cluster relation at the same time. Leader unit should use leader-set [8] to store information about “used” IP addresses to ensure preservation of this information in case that the leader unit changes.

Actual configuration of the additional IP address on each designate-bind unit could be done using netplan which supports defining static IP on the interface that has DHCP enabled. It’s done by adding an addresses field to the interface object while also leaving the option dhcp4: True [4].

If a designate-bind unit is removed from the cluster, the leader unit should return the previously assigned IP address to the pool of available service IPs.

In case that more units join the designate-bind cluster than there are available Service IP addresses, the unit without the service IP should be put into the blocked state with appropriate message.

In case that the service-ips option is modified and the new number of IP addresses is less than the current number of deployed designate-bind units, the leader should redistribute available IP addresses and units without it should be put into the blocked state with appropriate message.

In case that the service-ips option is changed from having values to empty value, the leader unit should trigger reconfiguration on all the designate-bind units that would remove previously statically configured IP addresses. If there are units that were previously blocked due to the lack of available service IPs, they should be brought back to the active state.

In case that the service-ips option is not set at all, no special action should be performed by the leader unit.

Optional: In addition to the configuration change, a confirmatory juju action could be implemented that would apply the “service-ips” config values. Without the action, no changes to the live configuration would be made. Reasoning for this is that it would serve as a confirmation by the user that the change is intended and it would prevent unwanted changes to the “service-ips” configuration and possible outages. Another benefit of this action would be that it could be applied 1 unit at a time so if there was something wrong with the configuration it would not affect the whole cluster at the same time but only one unit.

Further Information

Regardless of the approach taken, user should only interact with this new feature by using juju config command, for example:

$ juju config designate-bind service-ips=”10.0.10.10,10.0.10.20”

Basic checks should be performed on the supplied IP addresses:

Validate that the supplied values are valid IPs
Validate that the Service IP belongs to the subnet configured on the unit.

[1] https://jaas.ai/designate-bind/36

[2] https://jaas.ai/hacluster

[3] https://github.com/openstack/charm-interface-hacluster#requires

[4] https://askubuntu.com/questions/1031278/does-netplan-support-dhcp-and-static-addresses-on-one-interface

[5] https://github.com/openstack/charm-designate-bind/blob/master/src/metadata.yaml#L23

[6] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-colocationconstraints-haar

[7] https://www.thegeekdiary.com/managing-resource-startup-order-in-pacemaker-cluster-managing-constraints/

[8] https://charm-helpers.readthedocs.io/en/latest/api/charmhelpers.core.hookenv.html#charmhelpers.core.hookenv.leader_set

[9] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/ch-resourceconstraints-haar

Implementation

Assignee(s)

Primary assignee:: martin-kalcok <martin.kalcok@canonical.com>

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t designate-bind-serivce-ips

Work Items

Add configuration option service-ips
Handler for service-ips config change event that configures apropriate IP addresses on designate-bind units

Repositories

Work will be located in the main designate bind repository:

https://opendev.org/openstack/charm-designate-bind

Documentation

To Be Updated

Security

None

Testing

Unit and Functional tests will be needed to verify this new functionality.

Dependencies

None

Enablement of Keystone OpenID Connect Support

Tue, 08 Nov 2022 00:00:00

Keystone Charm support pluggable backends that use Keystone federations to enable a variety of providers for the account users and groups. This spec attempts to discuss how to enable support for OpenID Connect.

Problem Description

When deploying the OpenStack charms in an enterprise environment currently it’s only possible to bring up the central database of users into Keystone via LDAP or Federation, this last one method support only SAML integration, hence environments where users are exposed with OpenID Connect (e.g. Google Apps), Charmed OpenStack provides no solution.

Proposed Change

The proposed solution is to make OpenID Connect available to Charmed OpenStack in a new subordinate charm to keystone.

The new feature would be supported on OpenStack Yoga (and later) on Ubuntu 22.04 LTS.

Alternatives

Allow operators to configure OpenID Connect federation with some support backed into the Keystone charm directly.

Implementation

Assignee(s)

Primary assignee:: Felipe Reyes <felipe.reyes@canonical.com>

Gerrit Topic

Use Gerrit topic “keystone-openidc” for all patches related to this spec.

git-review -t keystone-openidc

Work Items

Implement a new operator framework subordinate charm, keystone-openidc, that implements keystone-fid-service-provider and websso-fid-service-provider relations
Add a charm option that allows the operator to pass a custom HTML template to be used as a Single Sign On template, this template will be used by OpenStack Horizon to render the login screen (see Configuring Horizon as a WebSSO Frontend)
Create a new snap with a OpenID Connect provider software such as Ipsilon or Oidc-op, this snap will be designed with the intention of being deployed and used for testing purposes.
Create a new charm that deploys a OpenID Connect provider for testing purposes, openidc-test-fixture, this charm will be similar to ldap-test-fixture.
Write unit tests
Extend current functional tests using the zaza testing framework

Timeline

The goal is to implement this change in the OpenStack Charms 22.04 release.

Repositories

During the intial development the code will be managed here:

When ready the charm keystone-openidc will be managed via OpenDev gerrit at:

https://opendev.org/openstack/charm-keystone-openidc

Documentation

The charm should contain documented options:

Create charm options
Create charm relations
Create charm README
Update the deployment-guide to include a section that explains how to configure Charmed OpenStack to authenticate against a OpenID Connect provider.

Security

The OpenStack Identity Service (Keystone) has to interact exclusively with OpenID Connect providers served over valid HTTPS endpoints.

Testing

Code changes will be tested by unit tests.

Functional tests would be covered by zaza testing framework.

Dependencies

This enablement depends on libapache2-mod-auth-openidc available in Ubuntu since 18.04 (Bionic) in the Universe pocket. To maintain the security and availability of this component in the long term a Main Inclusion Request (MIR) will be submitted.

The Title of Your Specification

Tue, 08 Nov 2022 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

Charmstore to Charmhub Migration

Thu, 09 Dec 2021 00:00:00

This spec proposes a new, better, but alternative means for maintaining and releasing OpenStack charms in a world with a new charm store (charmhub.io) which provides richer mechanisms for charm delivery.

A new charm store, charmhub, has been introduced that shares features and capabilities with snaps. The legacy charm store, jaas.ai, is planned to be retired and all projects delivering charms should migrate to charmhub. One of the compelling features of charmhub is the ability to offer multiple tracks for releases.

The charms themselves are written to support multiple releases of OpenStack from pre-Mitaka to Xena. This is convenient and straightforward for having development working only on one branch at a time for a charm, but requires that code and dependencies work towards the lowest common supported denominator. This is problematic as these charms depend on python libraries that are moving forward and dropping support for legacy python versions such as Python 2, 3.5 and beyond, which causes maintenance challenges for current charms.

Problem Description

Currently, OpenStack Charms are delivered in two charmstore namespaces - openstack-charmers and openstack-charmers-next. The use of two namespaces predates the existence of channels within the jaas.ai charm store. These namespaces are equivalent to the stable and edge channels respectively.

The git repositories hosting the source code for the OpenStack charms generally live on opendev.org git servers. The repositories contain branches of each stable charm release and an unstable development branch (master). A post-merge job in the CI system publishes an updated version of the charm to the jaas.ai charm store after any change is merged. Changes landing in the master development branch are published in the openstack-charmers-next namespace and changes landing in the most recent stable branch are published in the openstack-charmers namespace.

One of the challenges with the current approach is that the two branches must focus on the lowest common denominator for dependencies; meaning that most python libraries leveraged and included in the charms must be limited to versions that support older python releases that are still included on older OpenStack releases. As the world of Python library development moves on, support for older python versions are being deprecated and removed causing a variety of maintenance headaches for the current charms.

Another challenge with this approach is that each new feature introduced adds potential risk to older install bases. In order to mitigate this, the gate testing of the OpenStack Charms runs a broad sampling of tests across a multitude of OpenStack and base distro releases. New contributors are often running into challenges with the CI system and needing to debug older releases even when their patches are not relevant to these older versions.

One of the positive sides to this arrangement is that development primarily only needs to consider 2 branches at a time - the master development branch and the most recent stable branch. All patches are primarily targeted at the master development branch and relevant bug fixes are backported to a single release.

Proposed Change

OpenStack charms will adopt a model of release and delivery which takes advantage of the various delivery channels offered by charmhub which closely matches channels in the Snap store. Note, that not all charms will support all releases/versions of a specific service and charms will only adopt branches and tracks for those releases which are relevant to the service itself.

Tracks

Tracks will be set up to track git branches in the upstream repositories and will result in a track per release of the charm payload (e.g. the software service it delivers). When the primary software payload of the charm uses well-known release names, the release name will be used in lieu of version numbers for the track name. When the primary software payload of the charm does not use well-known release names, the version of the software will be used.

For example, the nova-compute charm delivers nova hypervisor services and the charm tracks will correspond to the OpenStack release of the nova services installed (e.g. ussuri, victoria, etc.).

As another example, the OVN Central charm delivers OVN as the payload and OVN uses release names/versions of YY.MM. In this scenario, the OVN charm Central charm will provide a track corresponding to the YY.MM naming schema (e.g. 21.03, 21.09, etc).

Risk Levels

The charmhub store also has the ability to allow the developers and users to provide and consume different risk levels. This feature was also available in the legacy charmstore, however it was not leveraged by the release process until very recently. There are 4 risk levels available within any track: edge, beta, candidate, and stable. The CI system will automatically publish a new charm for any change which lands in the corresponding git branch to the edge risk level.

Charms will be promoted through the edge -> beta -> candidate -> stable risk levels by a human actor, not through an automated script. Tooling may be developed to assist in this process, but automatic promotion is out of scope at this time. Future enhancements may include triggering a promotion of a build upon the introduction of git tags, but that is considered out of scope for this work.

Branches

The charmhub store also provides for a further subdivision of tracks and risk levels into branches, which are meant to provide temporary, short-lived, releases for bug fixing purposes. There are various possibilities to consider regarding how branch flows should be worked into the development flow of OpenStack charms.

For the purposes of this work, branches are explicitly out of scope and should be considered separately.

Git Branches

Git branches will need to be created for the various tracks that are delivered in Charmhub. The relationship between git branches and charmhub tracks are one-to-many, meaning that a change landing in one branch may result in a charm being published into multiple tracks. This is useful when the tracks provide logically different targets, while the branch serves the same functionality. For example, the latest master branch of development will logically point to the latest OpenStack release as well as the latest/edge release.

Additionally, the charms in each of the git branches and tracks are intended to support the targeted release N and the previous release N-1 for upgrade purposes.

OpenStack Charms

For those charms specifically delivering OpenStack services a new git branch will be created for each OpenStack release the charm supports. This branch will be named in the form of stable/<openstack-release> for all stable versions of OpenStack. Branches will only be created back to the Queens release of OpenStack. Each of these branches will be created from the tip of the current stable charms release (stable/21.10) as a base. These branches will differ from each other regarding the primary version of software that they are targeting as well as the default configuration values for the track they are deployed from.

Git Branch	Charmhub Track(s)
stable/queens stable/rocky stable/stein stable/train	train
stable/ussuri	ussuri
stable/victoria	victoria
stable/wallaby	wallaby
stable/xena	xena
stable/yoga	yoga
master	zed, latest/edge

Ceph Charms

Currently, Ceph updates are delivered through the Ubuntu Cloud Archive which ties the ceph charms to OpenStack versions, causing some challenges in defining tracks per Ceph release. In scenarios where services are co-located (e.g. nova-compute and ceph-osd), updating the source repository for the ceph-osd charm will have unintended consequences on the nova-compute services and their versions. Rather than introduce complex pinning or a new package archive, the charms used to deliver Ceph will track the OpenStack release names in addition to the Ceph release names, up to the Octopus release. This is expected to make the migration easier for users.

The development git branches will be created corresponding to the appropriate Ceph version where the product was delivered. In order to make it easier for users to migrate, Ceph charms may have a single branch delivered to multiple Charmhub tracks.

Git Branch	Charmhub Track(s)
stable/luminous	luminous, queens, rocky
stable/mimic	mimic, stein
stable/nautilus	nautilus, train
stable/octopus	octopus
stable/pacific	pacific
master	quincy, latest/edge

On the newer branches, a mapping can be made to safely convert a Ceph release name into a cloud-archive pocket in a way that doesn’t unintentionally upgrade either one, with the assumption that a valid mapping between OpenStack and Ceph versions is clearly documented. What that mapping should look like:

Ceph Release	OpenStack Release
Luminous	Queens
Mimic	Stein
Nautilus	Train
Octopus	Ussuri
Pacific	Wallaby (mid-cycle Openstack, more support)
Unstable / Quincy	Yoga

The above works because we can include distro (bionic-queens) with Rocky safely, and we can include Pacific’s focal-wallaby with Xena safely, as the Xena packages have higher versions.

The Ceph charms include the following charms:

ceph-fs
ceph-iscsi
ceph-mon
ceph-osd
ceph-proxy
ceph-radosgw
ceph-rbd-mirror
ceph-dashboard

Provider-specific Subordinate Charms

Some services such as Cinder and Neutron allow for provider specific subordinate charms to be used in order to allow for a specific SDN, storage plugin, etc. These provider-specific subordinate charms are considered supporting charms rather than OpenStack specific charms, however they are often enabling specific functionality for a specific storage backend. As such, they will follow the OpenStack Charms track schema.

These charms will include the following subordinate charms, and as of this writing include:

cinder-ceph
cinder-lvm
cinder-netapp
cinder-purestorage
neutron-openvswitch
neutron-api-plugin-arista
neutron-api-plugin-ovn
keystone-saml-mellon

OVN Charms

OVN is a networking SDN which is used as the primary SDN for Charmed OpenStack deployments. OVN has been provided through the Ubuntu Cloud Archive as well as the Ubuntu archives. The OVN charms are a bit of a mix when it comes to providing source versions. Standalone charms such as OVN Dedicated Chassis and OVN Central have source options which allow for configuring the archive to use for package installation. As a subordinate charm, OVN Chassis will adopt the version that is installed alongside the principal charm.

As OVN is a general SDN and not implemented specifically for OpenStack, the tracks in charmhub should closely follow the versions of the software delivered rather than the OpenStack release. This may be a bit confusing at first for users migrating, but this should be addressed with clear documentation as well as tooling to help the end-users select the appropriate track to upgrade their charms to. Additionally, tracks will be put in place labelled openstack-{release-name} to ease the migration burden.

OVN charms will have the following sets of branches and tracks:

Branch	Track(s)
stable/20.03	20.03, ussuri, victoria
stable/20.12	20.12, wallaby
stable/21.09	21.09, xena
stable/22.03	22.03
master	latest/edge

Supporting Charms

In the OpenStack ecosystem, there are other core services which are required to produce an OpenStack cloud. These services include messaging, database, and certificate management services. While they are critical to the functionality of an OpenStack cloud, they are not tied to specific releases of OpenStack and these charms should result in tracks which are appropriate for their specific behavior.

These charms consist of the following services, and will result in tracks that are based on the versions of software provided. These services are generally delivered from the Ubuntu archives (not the Ubuntu Cloud Archive) and will track the versions of their versions in Ubuntu.

Charm	Branch	Track(s)
hacluster	stable/bionic	1.1.x
hacluster	stable/focal	2.0.3, latest/stable
hacluster	master	2.0.5, latest/edge
pacemaker-remote	stable/bionic	1.1.x
pacemaker-remote	stable/focal	2.0.3, latest/stable
pacemaker-remote	master	2.0.5, latest/edge
rabbitmq-server	stable/bionic	3.6
rabbitmq-server	stable/focal	3.8, latest/stable
rabbitmq-server	master	3.9, latest/edge
vault	stable/1.5	1.5
vault	stable/1.6	1.6
vault	stable/1.7	1.7
vault	master	latest/edge
percona-cluster	stable/bionic	5.7, latest/stable
percona-cluster	master	latest/edge
mysql-innodb-cluster	stable/jammy	8.0, latest/stable
mysql-innodb-cluster	master	latest/edge

Trilio Charms

Trilio Charms will also need to be migrated to the charmhub delivery mechanism. While implemented for OpenStack, the Trilio software is versioned separately from the OpenStack releases and should use Trilio specific tracks.

trilio-data-mover	stable/4.0	4.0
trilio-data-mover	stable/4.1	4.1, latest/stable
trilio-data-mover	master	4.2 (?), latest/edge
trilio-dm-api	stable/4.0	4.0
trilio-dm-api	stable/4.1	4.1, latest/stable
trilio-dm-api	master	4.2 (?), latest/edge
trilio-horizon-plugin	stable/4.0	4.0
trilio-horizon-plugin	stable/4.1	4.1, latest/stable
trilio-horizon-plugin	master	4.2 (?), latest/edge
trilio-wlm	stable/4.0	4.0
trilio-wlm	stable/4.1	4.1, latest/stable
trilio-wlm	master	4.2 (?), latest/edge

Installation Sources

Most charms in the OpenStack charm collection provide a config option that allows the user to set the archive or repository that debian packages/snaps are installed from. This is generally provided via either the openstack-origin or source charm config option, which defaults to install using the local installations default distribution repositories.

In order to ensure that the tracks are installing the appropriate versions of packages, the default value for this config option will be set to the appropriate installation source for the track. If a track spans multiple distro releases (e.g. the cloud-archive at the LTS release crossover), the charm will need to be able to determine whether this should be sourced from the cloud-archive or the distribution, whichever is appropriate.

Primarily, this will affect the tracks that use the Ussuri Ubuntu Cloud Archive and the Yoga Ubuntu Cloud Archive as an installation source.

Latest Track

The ‘latest’ track in charmhub is its own track, and like other tracks - charms can be pushed to the track and the various risk levels within the track. While it is a full blown track, to end users it is a means to get the “latest” version of a piece of software. However, the risk level must be taken into consideration for which version should be installed.

The latest track will be used into two ways:

The latest/stable track will be used to park the stable 21.10 charms release until (at least) the 22.10 release.
At 22.10, the latest/stable track will follow the current stable release, and for 22.10 this will be the zed track.

The reasoning is that the 21.10 charms’ metadata advertise focal support, against all architecturees and so it difficult to replace until a jammy, and greater, charm can be released to the channel. However, the zed charms will only support jammy and kinetic, and so they can be released alongside the existing 21.10 charms. Then existing 21.10 charms won’t upgrade as the new charms will not support focal. Therefore, it is safe to re-use the latest/stable track at that point.

Risks

The purpose of each risk level for each track is as follows:

Edge

For each charm, the latest/edge channel will always map the current master development branch.

Currently, for stable branches, the edge track will not be used, as the Launchpad builders will push directly to the stable track. This is in keeping with the existing policy of requiring 2 x +2 reviews on gerrit for stable branches. At some future point, automation/policy will be introduced to enable pushing to edge to start with and then having testing/policy to promote to more stable risks.

Beta

The beta risk level will be used for milestone releases during the charm development cycle.

Candidate

The <track>/candidate channel will be used to track the next software release as it enters the final weeks of development and stabilizes in preparation for a release. Note that this is on the next stable release track.

Stable

The stable track will track the stable, tested, released version of the charm.

Technology Preview Charms

Charms are initially released in the “Technology Preview” state. Charms which are in the technology preview state should not be promoted into the stable channel until it is determined it is stable for the corresponding payload release. Once it has graduated, all releases which have qualified as graduated from technology preview will then be promoted to the stable channel. It is possible that not all tracks of a charm will have a stable risk level. Charms which are still in technology preview will only promote those charms to the beta channel until it is determined that the charm has reached maturity, at which point it can be promoted to the candidate and/or stable channels as appropriate.

For example, a charm such as ovn-chassis may be delivered as technology preview in the Stein release track but may not reach maturity until the Train release track. The Train release track of the charm will be promoted to the stable channel but the Stein release track will only have the charm in the beta channel.

Special consideration is needed for those charms considered Technology Preview at the time of this transition. Technology Preview charms are not promulgated in the charm store and are available under the openstack-charmers namespace. Charmhub does not have namespaces in the same way and these charms exist but are prefixed with the ‘openstack-charmers’ name, e.g. openstack-charmers-ironic-api. These charms will be created in charmhub and promoted to the beta channel.

Continuous Integration

Charmhub will deliver charms that are built for specific architectures. OpenStack charms to date, are built using python modules on amd64 architectures and distributed as such. This is generally acceptable as they are primarily source charms which have deliveries, but can pose some interesting challenges for non-amd64 architectures with native extensions.

The OpenStack CI system is currently composed of a Zuul-CI system which is used to trigger a variety of pep8 (pycodestyle), unit and functional tests for each patch proposed to a charm. Post-merge hooks are registered which trigger a Jenkins task to publish the charm to the legacy charm store.

Publishing Charms

For newer charms, delivered through charmhub, they are expected to have platform specific charms so that native extensions or tooling required by the charm can be built for the appropriate target architecture. Launchpad now offers charm recipes, which can be used to build and publish charms in a similar manner to snap recipes. Changes merged into a repository will trigger a build of the charm on all supported architectures and publish the resulting charm to Charmhub. Leveraging this service removes a step from the OpenStack Charmers maintained CI system for publishing charms to the charm store and results in completely eliminating the need for jenkins in the CI infrastructure.

The Launchpad charm recipes require that the git trees are known and stored in Launchpad itself. In order to continue to use the upstream OpenStack gerrit review system, which is standard for OpenStack services and has been standard for the charms for years, each charm project in Launchpad will be configured to mirror the source from the upstream opendev git system.

A charm recipe will be created for each branch of each charm such that upon a patch being merged in the upstream gerrit repository, the Launchpad service will update the mirrored (a.k.a. imported) repository, detect that new changes are present, initiate a charm build and publish these results to the appropriate channel(s) in charmhub.

Note that charm recipes in Launchpad will only build by default once per hour. While there is normally a slight delay in publishing charms to the charm store, there is one publish per change to the charm. In the Launchpad build model, multiple changes can be queued up and included in a single build unless builds are explicitly requested within the hour time window. This is considered acceptable for charm development and publishing purposes.

Testing Charms

With the introduction of branches and tracks per OpenStack release, the need to test a multitude of OpenStack releases for each change is no longer required. Instead, targeted tests for the branch will be run. For example, a change proposed to the Keystone charm in the stable/xena branch will run only tests relevant to the Xena release of OpenStack.

Reducing the releases that are running in the gate allows for the charm to be focused on the specific payload release and need not worry about spurious failures on older releases.

The branches may also include tests that exercise the N-1 version of the release software, in order to validate that the upgrade of N-1 to N are functioning properly.

Charmcraft

In order to leverage the charm recipes in Launchpad, all of the charms will need to be built using the charmcraft tool. Charmcraft, however, was designed to build and publish operator framework charms, which excludes the majority of the charms in the OpenStack charms collection.

Charmcraft has recently grown parity with the snapcraft toolset, which allows a variety of plugins to be used instead of the default charm plugin. The ‘dump’ plugin is appropriate to use for classic charms, however it is unsuitable for reactive charms. In the long term, a new plugin should be created for building reactive charms using charmcraft. In the short term, the various steps for producing a charm can be overridden using the ‘override-build’, ‘override-stage’, and ‘override-prime’ steps which will use the existing tox infrastructure to build, stage, and prime the reactive charm.

Documentation

Documentation for the OpenStack Charms and Ceph Charms are generally available as a solution level, with charm README files serving as the primary source of documentation within the Charm store and the published OpenStack Charm Guide, OpenStack Charms Deployment Guide and Ubuntu Ceph Documentation providing solution level documentation.

The release notes need to clearly identify the change to use and leverage Charmhub as well as referencing migration documentation. Migration documentation needs to be clearly outlined to make it simple for users to migrate from using the Charm Store to Charmhub.

Developer documentation should also be provided in the OpenStack Charm Guide to provide details for developers wishing to contribute new features, bug fixes, etc.

Alternatives

There aren’t really any good alternatives. When the charmstore to charmhub cut-over is implemented, the existing charm command will stop working, which means that the entire build and publishing system in Jenkins will simply stop working too. So the only real alternative is to re-work the Jenkins software to use charmcraft and then use that to push to the charm store.

This means continuing to maintain Jenkins and not take advantage of the Launchpad builders to build architecture aware charms.

Implementation

Assignee(s)

Primary assignee:: Alex Kavanagh

Gerrit Topic

N/A

Work Items

Produce ‘charmcraft’ recipes for classic and reactive charms

Launchpad can be used to build charms when branches in git repositories are changed. These builds are controlled by recipes. Part of the work will be to ensure that these recipes work for all of the charms. They will be centrally controlled and ‘synced’ into charms as part of the development cycle.

Write classic recipe
Write reactive charm recipe using charm-tools build command.
Add recipes to release-tools repository.

Produce helper tools/library to manage launchpad projects/recipes

In order to manage the charms, recipes and mapping of git repository branches to charmhub tracks/channels, a suite of tools and configuration will be produced to manage the complexity of the number of charms, branches and tracks.

Essential functionality will be:

display gap between config and actual in git repositories and launchpad recipes.
Show desired tracks vs actual tracks for charms (in charmhub) to generate ‘requests’ for new tracks, or for tracks to be deleted.
update existing recipes against the configuration.
Create new recipes as required against the config.

Repositories

Initially, the existing release-tools repository will be the collecting point for the tools, but it has become a bit of a dumping ground for related tools and needs a major re-organistion. In the fullness of time, these tools may be broken out into their own repository where changes to the config automatically updates the recipes.

Additionaly, the manage the recipes a separate charmhub-lp-tools tool will be created to automate recipe creation, updates, deletes and changed, and to authorize uploads to the charmhub.

https://github.com/openstack-charmers/release-tools
https://github.com/openstack-charmers/charmhub-lp-tools

Documentation

Documentation for the tools, configuration schemas and how the system works will be inside the repository in the form of Markdown files. This is to ensure that the documentation ‘lives’ with the code. Where possible documentations for the schemas (config files) will be in the config files themselves.

Security

No additional security concerns.

The user that runs the tool must have a launchpad account in the openstack-charmers team, and a charmhub account in the openstack-charmers team.

Testing

The possibility to break the charmstore and charmhub is pretty high, so the tools and process will be developed incrementally against ‘test’ charms and low priority ‘real’ charms.

Stage 1

Initially copies of the existing charms will be registered into the charmhub (under new names), to test the recipes and ability to build charms. This is to verify that the charm build recipes work for classic and reactive charms.

Stage 2

Selecting two charms (notionally, neutron-dynamic-routing and manila-generic), break the link to the charmstore and ensure that tracks, branches and recipes can be created such that updates to the charms git repositories result in built charms in the correct tracks.

This will allow the correct changes to the charms to be enumerated and captured in the release-tools repository.

Dependencies

None

References

Enablement of Ceph Dashboard

Wed, 10 Nov 2021 00:00:00

Problem Description

The ceph-mon charm provides the cluster monitor daemon for the Ceph distributed file system. It currently does not support enablement of the Ceph Manager built-in Dashboard. It is a web-based Ceph management and monitoring application through which you can inspect and administer various aspects and resources within the cluster. It is implemented as a Ceph Manager Daemon module.

Proposed Change

The proposed solution is to make the dashboard available and default enabled in a new subordinate charm to ceph-mon. SSL endpoint termination would be done using the certificates relation or by providing the ssl_cert, ssl_key and ssl_ca configuration options. User management would be done with the use of charm actions.

The upstream documentation of Ceph mentions Dashboard is supported, (https://docs.ceph.com/en/latest/mgr/dashboard/). The new feature would be supported on Ceph Octopus, OpenStack Victoria (and later) on Ubuntu 20.04 LTS.

The subordinate charm will enable access to the dashboard via the hostnames of the mon units. However, the hostname may not map to the desired network space, the operator may not be able to resolve the juju unit hostnames and the failure of a single mon unit may make the dashboard inaccessable if the operator was accessing the dashboard via that mon. To resolve these issues the dashboard charm will support a relation to an OpenStack loadbalancer application (as yet unwritten) which will manage an haproxy instance and a vip via the hacluster subordinate. If this relation is present the dashboard standby units will be configured to return a 500 enabling haproxy to determine the active dashboard unit. Both the loadbalancer and the dashboard charm will support SSL endpoint termination via a relation to vault or via configuration options. The delivery of the dashboard subordinate should be the initial focus of this work with the loadbalacer following soon after.

Alternatives

Grafana offers some visualization possibilities, but it is limited to view only experience.

Out of Scope

Features
Feature	In Scope
Multi-User and Role Management	✓
Single Sign-On (SSO)	X
SSL/TLS support	✓
Auditing	✓
Embedded Grafana Dashboards	✓
Monitoring (Prometheus & Alertmanager)	X
iSCSI	X
RBD	✓
RBD mirroring	X
CephFS	X
Object Gateway	✓
NFS	X

Implementation

Assignee(s)

Primary assignee:: Liam Young <liam.young@canonical.com>

Gerrit Topic

Use Gerrit topic “ceph-dashboard” for all patches related to this spec.

git-review -t ceph-dashboard

Work Items

Implement a new operator framework subordinate charm, ceph-dashboard
Add a charm option for toggling ceph mgr dashboard
Add a relation that implements SSL endpoint configuration
Add actions to manage user accounts
- Create admin user
write unit tests
extend current functional tests using the zaza test framework (a functional test should include deploying ceph-mon with ceph-dashboard added, without user management actions)
Create generic service loadbalancer charm.
Add a relation to a loadbalancer service which will provide a HA vip for accessing the dashboard.

Timeline

The goal is to implement this change in the OpenStack Charms 21.10 release.

Repositories

During initial development the code will be managed here:

https://github.com/openstack-charmers/charm-ceph-dashboard

When ready the charm will be managed via OpenDev gerrit.

https://opendev.org/openstack/charm-ceph-dashboard

Documentation

The charm should contain documented options:

Create charm options
Create charm actions
Create charm relations
Create charm README
Update of the official charm deployment guide: [0]

[0]: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/

Security

The Dashboard endpoint has to be TLS-terminated, therefore, an option to provide a CA certificate is required.

Testing

Code changes will be covered by unit tests. Functional tests would require ceph OSD hardware or software emulation. This is possible through Ceph cluster deployment. The deployment of the ceph cluster would be independent from the OpenStack bundle.

Dependencies

This charm will support OpenStack Victoria and Ubuntu 20.04 Focal as its baseline
A new project will need to be created based on the OpenStack Project Creator’s Guide <https://docs.openstack.org/infra/manual/creators.html>

Cinder subordinate charm cinder-lvm

Wed, 10 Nov 2021 00:00:00

Problem Description

Proposed Change

Instead of changing the cinder charm, this proposal aims to create a separate subordinate charm specific for the lvm driver. The following points are of condiderable importance for the development of this subordinate charm:

The new implementation should not conflict with current implementation. Both should co-exist, even in the same deployment if possible.
The new charm should implement all features currently existing in the cinder charm, regarding the lvm functionality.
New LVM capabilities in the exisiting cinder charm are to be considered deprecated and will be removed at a future release.

Note that because the LVM driver is part of the cinder-common package, no additional packages need to be installed for this charm to work.

This charm will support the then Queens release of Openstack and newer.

Alternatives

At the time of this writing, there exist no alternatives.

Implementation

Assignee(s)

Primary assignee:: Andre Ruiz <andre.ruiz@canonical.com> - LP ID: andre-ruiz
Secondary assignee:: Luciano Lo Giudice <luciano.logiudice@canonical.com> - LP ID: lmlogiudice

Gerrit Topic

Use Gerrit topic “charm-cinder-lvm” for all patches related to this spec.

git-review -t charm-cinder-lvm

Work Items

The charm, as listed above. Plus, both unit and functional tests will be written.

Repositories

Presently, the following repository hosts the charm code:

https://github.com/openstack-charmers/charm-cinder-lvm

Eventually, it will be moved to the _openstack_ namespace.

Documentation

Once this charm is completed, charm-guide will be updated to reference it.

Security

No security aspect is going to change in relation to the current solution.

Testing

Code changes will be covered by both unit and functional tests. No special hardware is expected to be needed to test this charm.

Dependencies

No dependencies in addition to the ones that exist in the current solution.

Service Restart Control In Charms

Wed, 10 Nov 2021 00:00:00

Openstack charms continuously respond to hook events from their peers related applications which frequently result in configuration changes and subsequent service restarts. This is all fine until these applications are deployed at large scale and having these services restart simultaneously can cause (a) service outages and (b) excessive load on external applications e.g. databases or rabbitmq servers. In order to mitigate these effects we would like to introduce the ability for charms to apply controllable patterns to how they restart their services.

Problem Description

An example scenario where this sort of behaviour becomes a problem is where we have a large number, say 1000, of nova-compute units all connected to the same rabbitmq server. If we make a config change e.g. enable debug logging on that application this will result in a restart all nova-* services on every compute host in tandem which will in turn generate a large spike of load on the rabbit server as well as making all compute operations block until these services are back up. This could also clearly have other knock-on effects such as impacting other applications that depend on rabbitmq.

There are a number of ways that we could approach solving this problem but for this proposal we choose simplicity by attempting to use all information already available to an application unit combined with some user config to allow units to decide how best to perform these actions.

Every unit of an application already has access to some information that describes itself with respect to its environment e.g. every unit has a unique id and some applications have a peer relation that gives them information about their neighbours. Using this information coupled with some extra config options on the charm to vary timing we could provide the operator the ability to control service restarts across units using nothing more than basic mathematics and no juju api calls.

For example, let’s say an application unit knows it has id 215 and the user has provided two options via config; a modulo value of 2 and an offset of 10. We could then do the following:

time.sleep((215 % 2) * 10)

which, when applied to all units, would result in 50% of the cluster restarting its services 10 seconds after the rest. This should hopefully alleviate some of the pressure resulting from cluster-wide synchronous restarts, ensuring that part of the cluster is always responsive and making restarts happen quicker.

As mentioned above we will require two new config options to any charm for which this logic is supported:

service-restart-offset (default to 10)
service-restart-modulo (default to 1 so that default behaviour is same as
before)

The restart logic will skip for any charms not implementing these options.

Over time some units may be deleted from and added back to the cluster resulting in non-contiguous unit ids. While for applications deployed at large scale this is unlikely to be significantly impactful, since subsequent adds and deletes will cancel each other out, it could nevertheless be a problem so we will check for the existance of a peer relation on the application we are running and, if one exists, use the info in that relation to normalise unit ids prior to calculating delays.

Lastly, we must consider how to behave when the charm is being used to upgrade Openstack services whether directly using config (“big bang”) or using actions defined on a charm. For the case where all services are upgraded at once we will leave it to the operator to set/unset the offset parameters. For the case where actions are being used, and likely only a subset of units are being upgraded at once, we will ignore the control settings i.e. delays will not be used.

Proposed Change

To implement this change we will extend the restart_on_change() decorator implemented across the openstack charms so that when services are stop/started or restarted they will include a time.sleep(delay) where delay is calculated from unit id combined with two new config options; service-restart-offset and service-restart-modulo. This calculation will be done in a new function that will be implemented in contrib.openstack the output of which will be passed into the restart_on_changed() decorator.

Since a decorator is used we do not need to worry about multiple restarts of the same service. We do, however, need to consider how apply offsets when stop/start and restarts are performed manually as is the case in the action managed upgrades handler.

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: hopem

Gerrit Topic

Use Gerrit topic “controlled-service-restarts” for all patches related to this spec.

git-review -t controlled-service-restarts

Work Items

implement changes to charmhelpers
sync into openstack charms and add new config opts

Repositories

None

Documentation

These new settings will be properly documented in the charm config.yaml as well as in the charm deployment guide.

Security

None

Testing

Unit tests will be provided in charm-helpers and functional tests will be updated to include config that enables this feature. Scale testing to prove effectiveness and determine optimal defaults will also be required.

Dependencies

None

OpenStack Endpoint Load Balancer

Wed, 10 Nov 2021 00:00:00

To enable Openstack services for a single cloud to be installed in a highly available configuration without requiring that each unit of a service is in the same broadcast domain.

Problem Description

As a cloud administrator I would like to simplify my deployment so that I don’t have to manage a corosync and pacemaker per OpenStack API service.
As a cloud architect I am designing a new cloud where all services will be in a single broadcast domain. I see no need to use the new central loadbalancer and would like to continue to have each service manage its own VIP.
As a cloud architect I would like to spread my control plane across N racks for redundancy. Each rack is in its own broadcast domain. I do not want the users of the cloud to require knowledge of this topology. I want the endpoints registered in Keystone to work regardless of a rack level failure. I am using network spaces to segregate traffic in my cloud and the OpenStack loadbalancer has access to all spaces so I only require one set of loadbalancers for the deployment.
As a cloud architect I would like to spread my control plane across N racks for redundancy. Each rack is in its own broadcast domain. I do not want the users of the cloud to require knowledge of this topology. I want the endpoints registered in Keystone to work regardless of a rack level failure. I am using network spaces to segregate traffic in my cloud. I want the segregation to extend to the load balancers and so will be requiring a set of load balancers per network space.
As a cloud architect I am designing a new internal cloud and have no interest in IPv6, I wish to deploy a pure IPv4 solution.
As a cloud architect I am designing a new cloud. I appreciate that it has been 18 years since the IETF brought us IPv6 and feel it maybe time to enable IPv6 within the cloud. I am happy to have some IPv4 where needed and am looking to deploy a dual stack IPv4 and IPv6.
As a cloud architect I am designing a new cloud. I appreciate that it has been 18 years since the IETF brought us IPv6 and wish to never see an IPv4 address again. I am looking to deploy a pure IPv6 cloud.
As a cloud architect I wish to use DNS HA in conjunction with the OpenStack loadbalancer so that loadbalancer units can be spread across different subnets within each network space.
As a cloud administrator I would like to have the OpenStack load balancers look after HA and so will be deploying in an Active/Passive deployment. I will need to use a VIP for the loadbalancer in this configuration.
As a cloud architect I have an existing hardware loadbalancers I wish to use. I do not want to have to update it with the location of each API service backend. Instead I would like to have the OpenStack load balancers in an Active/Active configuration and have the hardware loadbalancers manager traffic between haproxy instance in the OpenStack loadbalancer service. I do not need to use a VIP for the loadbalancer in this configuration. My hardware loadbalancers utilise vip(s) which will need to be registered as the endpoints for services in Keystone.
As a cloud administrator haproxy statistics are fascinating to me and I want the statistics from all haproxy instances to be aggregated.
As a cloud administrator I would like haproxy to be able to perform health checks on the backends which assert the health of a service more conclusively than simple open port checking.
As a cloud administrator I want to be able to configure max connections and timeouts as my cloud evolves.
As a charm author of a service which is behind the OpenStack load balancer I would like the ability to tell the loadbalancer to drain connection to a specific unit and take it out of service. This will allow the unit to go into maintenance mode.

Proposed Change

New interface: openstack-api-endpoints

This interface allows a backend charm hosting API endpoints to inform the OpenStack loadbalancer which services it’s hosting and on which IP address and port frontend API requests should be sent to on the backend unit. It also allows the backend charm to inform the loadbalancer which frontend port should be used for each service.

Example - neutron-api (single API endpoint per unit):

endpoints:
  - service-type: network
    frontend-port: 9696
    backend-port: 9689
    backend-ip: 10.10.10.1
    check-type: http

Example - nova-cloud-controller (multiple API endpoints per unit):

endpoints:
  - service-type: nova
    frontend-port: 8774
    backend-port: 8764
    backend-ip: 10.10.10.2
    check-type: http
  - service-type: nova-placement
    frontend-port: 8778
    backend-port: 8768
    backend-ip: 10.10.10.2
    check-type: http

A single instance of the OpenStack Loadbalancer application will only service a single type of OpenStack API endpoint (public, admin or internal). The charm will use the network space binding of the frontend interface to determine which IP or VIP (if deployed in HA configuration) should be used by the backend API service for registration into the Cloud endpoint catalog.

Having processed the requests from all backend units, the loadbalancer now needs to tell the backend application the external IP being used to listen for connections for each endpoint service type:

endpoints:
  - service-type: nova
    frontend-ip: 98.34.12.1
    frontend-port: 8774
  - service-type: nova-placement
    frontend-ip: 98.34.12.1
    frontend-port: 8778

The backend service now updates the endpoints in the Keystone registry to point at the IPs passed back by the loadbalancer.

This interface is provided by each backend API charm and consumed via the backend interface on the OpenStack loadbalancer charm. Each backend charm would provide three instances of this interface type:

provides:
  public-backend:
    interface: openstack-api-endpoints
  admin-backend:
    interface: openstack-api-endpoints
  internal-backend:
    interface: openstack-api-endpoints

Taking this approach means that the backend charm can continue to be the entry point/loadbalancer for some endpoint types, and push the loadbalancing for other entry points out to the OpenStack Loadbalancer charm (or multiple instances).

Updates to keystone endpoint calculation code

Currently the following competing options are used to calculate which EP should be registered in Keystone:

os-*-network set do resolve_address old method
dnsha use dnsha
os-*-hostname set use hostname
juju network space binding via extra-bindings
prefer ipv6 via configuration option
presence of {public,internal,admin}-backend relations to openstack loadbalancers

OpenStack Loadbalancer charm

New charm - OpenStack Loadbalancer - with corresponding tests & QA CI/setup.

Alternatives

Extend existing HAProxy charm.
Use DNS HA.

Implementation

Assignee(s)

Primary assignee:: unknown

Gerrit Topic

Use Gerrit topic “osbalancer” for all patches related to this spec.

git-review -t osbalancer

Work Items

Provide OpenStack Loadbalancer Charm

Write draft interface for LB <-> Backend
Write unit tests for Keystone endpoint registration code
Write Keystone endpoint registration code

Mojo specification deploying and testing Mistral

Write Mojo spec for deploying LB in an HA configuration

Repositories

A new git repository will be required for the Mistral charm:

https://git.openstack.org/openstack/charm-openstack-loadbalancer

Documentation

The OpenStack Loadbalancer charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

None

The Title of Your Specification

Wed, 10 Nov 2021 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

Ceph Benchmarking

Sat, 31 Jul 2021 00:00:00

Proposing a new charm, ceph-benchmarking, and additions to the Zaza test framework, in order to provide a simple repeatable method for testing ceph performance.

This new charm will allow for A/B testing. Performance can be compared between a deployment with a feature turned on and a deployment with the feature turned off in order to make informed choices on feature usage.

Benchmark Ceph Cluster Performance will be used as template for the various tests.

Problem Description

It is often unclear what performance one should expect from a software stack like Ceph. For example, it is difficult to give performance expectations in absolute terms (e.g. IOPS, throuput, etc), because performance is hardware and technology dependent. It may also be difficult to determine which suite of features to use and what their relative cost / benefit quotient might be.

The proposed changes will allow for acquiring relative performance (IOPs) on a given set of hardware that can be compared with various features turned on or off. Testing can be performed on hardware prior to a production deployment in order to give a before and after perspective on how the Ceph cluster performs in order to assist in bottleneck detection.

Proposed Change

ceph-benchmarking charm

A charm with several actions to run Ceph performance benchmarking tests and gather their results. (e.g. ‘rados bench’, ‘rbd bench’, ‘fio’, ‘swift bench’) The new charm will utilised the Ops Framework and leverage ops_openstack code base.
- Proposed actions and parameters
  - net-iperf
    - IP(s) of Ceph cluster
  - fio-disk
    - disk(s)
    - readwrite
    - blocksize
    - iodepth
  - rados-bench
    - pool
    - duration
    - task (write, random, sequential)
  - rbd-bench
    - pool
    - image
  - fio-ceph
    - readwrite
    - blocksize
    - iodepth
  - swift-bench
Additions to the Zaza test framework

The Zaza test framework uses libjuju under the hood and allows for standing up deployments from bundles and executing suites of tests against them. Test targets for each action on the new charm will be added allowing for a suite of Ceph performance tests to be repeated as often as desired.

Alternatives

All of the tests in Benchmark Ceph Cluster Performance can be run manually. The goal is to make this process more efficient.

Implementation

Assignee(s)

Primary assignee: David Ames (thedac)

Gerrit Topic

Use Gerrit topic “ceph-benchmarking” for all patches related to this spec.

git-review -t ceph-benchmarking

Work Items

ceph-benchmarking charm
Zaza targets

Repositories

The ceph-benchmarking charm will initially reside in the openstack-charmers namespace and may eventually move to the openstack namespace:

https://github.com/openstack-charmers/charm-ceph-benchmarking

Additions to Zaza will be added to the zaza-openstack-tests repository:

https://github.com/openstack-charmers/zaza-openstack-tests

Documentation

Documentation will be written for the usage of ceph-benchmarking and related Zaza tests. The actions for the charm will be documented in the charm’s README. The usage of the actions with or without Zaza will be an appendix to the charm-guide and / or the deployment-guide.

Security

The various tests may leave behind test detritus. Cleanup will be attempted where possible, but the operator will need to take responsibility for any security implications.

Testing

The proposed changes are the tests themselves. The Zaza targets will provide functional testing for the ceph-benchmarking charm.

Dependencies

Benchmark Ceph Cluster Performance is the template for the various tests.

Magnum

Sat, 31 Jul 2021 00:00:00

Magnum is an OpenStack API service, developed by the OpenStack Containers team, making container orchestration engines (COEs) such as Docker Swarm, Kubernetes, and Apache Mesos available as first class resources in OpenStack.

It uses Heat to orchestrate an OpenStack image, which contains the COE (for example: Docker and Kubernetes), and it runs that image in either virtual machines or bare metal in a cluster configuration.

Also, Magnum offers complete life-cycle management of COEs in an OpenStack environment, integrated with other OpenStack services. This allows a seamless experience for OpenStack users who wish to run containers in an OpenStack environment.

Problem Description

There is currently no charm that lets you easily deploy Magnum as part of Charmed OpenStack.

Proposed Change

Implement a set of reactive charms that enables the deployment of all required Magnum components. Changes will also be made to existing charms to integrate with the new Magnum charms.

Baseline supported OpenStack version will be Ussuri.

Alternatives

None

Implementation

The implementation consists of the following new charms:

magnum
- Provides the Magnum API. OpenStack administrators and Openstack Dashboard will call into this API to provision and manage Kubernetes clusters (or any other COE).
- Requires relations to AMQP, certificates, database and Keystone.
magnum-dashboard
- Create a new subordinate charm for Openstack Dashboard
- Implements the necessary changes to enable Magnum UI in Horizon

The implementation also requires changes to the following charms:

keystone
- Add Magnum to the list of valid services.

Assignee(s)

Primary assignee:: oprinmarius

Gerrit Topic

Use Gerrit topic “magnum-charm” for all patches related to this spec.

git-review -t magnum-charm

Work Items

The implementation section describes the working items.

Repositories

charm-magnum
charm-magnum-dashboard
charm-keystone

Documentation

Each charm will contain a README with instructions on deploying the charm. In addition, an appendix will be added to the deployment guide which will outline various deployment scenarios, hardware and networking requirements and config options.

Security

magnum
- Requires keystone, database and rabbitmq credentials which will be stored in magnum.conf
- The API endpoints for Magnum will optionally be accessible via TLS, either by creating a relation to Vault / easy-rsa, or by manually supplying the TLS key/cert via config

Testing

Code written or changed will be covered by unit tests; functional testing will be implemented using the Zaza framework.

Dependencies

None

Cinder Ceph Replication

Sat, 31 Jul 2021 00:00:00

Cinder replication allows Disaster Recovery (DR) via the Ceph RBD driver. During a failover scenario, the Cinder Ceph RBD driver will promote the secondary Ceph images to primary and use those going forward.

The Ceph RBD driver implements the Cinder replication mechanism. More info about it here: https://docs.openstack.org/cinder/victoria/contributor/replication.html.

Problem Description

There are two possible Ceph RBD mirroring modes (pool and image), and the ceph-rbd-mirror charm knows to do only pool mode. More info about RBD mirroring modes here: https://docs.ceph.com/en/latest/rbd/rbd-mirroring/#enable-mirroring.

The image mirroring mode is required for the Ceph replication mechanism, because the Ceph RBD driver will explicitly enable mirroring for each Ceph image, and set the required image features (exclusive-lock and journaling). When mirroring mode is set to pool, mirroring is enabled by default on all the Ceph images in the pool, and the Ceph clients will not take care of this part.

We want image mirroring mode (instead of pool) for the Cinder replication mechanism, because we don’t want all the Cinder Ceph images blindly mirrored. The mirroring will be orchestrated via the Cinder volume types. More info about this here: https://docs.openstack.org/cinder/ussuri/contributor/replication.html#volume-types-extra-specs.

Also, Cinder needs connection credentials to the remote cluster where the images are mirrored, to be able to perform a failover. This information needs to go into the cinder.conf, under the backend section with replication enabled.

Right now, if ceph-mon has a relation with ceph-rbd-mirror, it instructs its clients from the other relation (implementing the ceph-client interface) to use the exclusive-lock and journaling features, by default, for all the images in the pool.

We need to have the RBD mirroring features, enabled by default for all the images, only when mirroring mode is set to pool. When the mirroring mode is set to image, the Ceph client will explicitly set the RBD mirroring features on each image, if needed.

Therefore, we need to ignore the advertised default RBD image features from the ceph-mon charm, if the Ceph clients will have the mirroring mode set to image.

Proposed Change

The cinder-ceph charm should have a new config option called rbd-mirroring-mode that should be passed as part of the create-pool broker request to the ceph-mon charm.

The ceph-mon charm will forward the broker request with the new rbd-mirroring-mode flag to the ceph-rbd-mirror charm, which should be able to handle it and enable the requested mirroring mode for the pool.

Also, the ceph-rbd-mirror charm should use pool mirroring mode if it doesn’t find any mode explicitly set in the broker request. This functionality should maintain the backwards compatibility of the charm.

On top of this, the Ceph clients implementing the new rbd-mirroring-mode config option (for example cinder-ceph charm) should not use the advertised default RBD mirroring features from the ceph-mon charm, when mirroring mode is image. We can safely assume that these clients will take care of explicitly setting those image features.

When all of the above are in place, Cinder needs to be able to connect to the mirrored images, in the remote cluster, when a failover happens. It will promote the mirrored image to primary and use that going forward.

The cinder-ceph charm will get the connection credentials to the remote Ceph cluster through a new relation with the ceph-mon charm. The new relation will implement the existing ceph-client interface, and it will be named ceph-replication-device.

Also, nova-compute needs to be able to connect to the Ceph cluster where the Cinder volumes are mirrored (otherwise, the VMs won’t be able to access Cinder volumes after a failover).

At the moment, the interface cinder-ceph-key is used by cinder-ceph to grant access to nova-compute for the main Ceph backend. We need to grant access to nova-compute for the secondary backend (with the replicated devices).

Thus, we extend the ceph-access relation (that implements the cinder-ceph-key interface), and we set a new relation variable called keyrings. This is a list of Ceph keys corresponding to the Cinder main and replication backends. This new relation variable is only set when Cinder replication is enabled, otherwise fallback to setting the previous relation data.

Alternatives

None

Implementation

Assignee(s)

ionutbalutoiu (primary)
oprinmarius

Gerrit Topic

Use Gerrit topic “cinder-ceph-replication” for all patches related to this spec.

git-review -t cinder-ceph-replication

Work Items

cinder-ceph
- Provides a new config option rbd-mirror-mode (with the pool as the default value), and it will send this to ceph-mon as part of the broker request to create the pool (which is going to be forwarded to ceph-rbd-mirror).
- Implements a new relation, using the ceph-client interface, with the ceph-mon charm. The relation is called ceph-replication-device. This is going to be used to get the connection information to the remote Ceph cluster where the pool is mirrored. This information will go into the cinder.conf as replication_device config option, under the backend with replication enabled.
- Extend the ceph-access relation. If replication is enabled, make sure to set the list of Ceph keys (keyrings) for both Cinder main and replication backends.
nova-compute
- Handles the extended ceph-access relation. If more than a single Ceph key is set as part of the relation data, make sure that both of them are configured.
  
  Makes sure that the previous relation data is handled as well (to maintain backwards compatibility).
ceph-mon
- Forwards the broker requests to the ceph-rbd-mirror with information about the RBD mirroring mode.
ceph-rbd-mirror
- Handles the new RBD mirroring mode flag from the forwarded broker request given by ceph-mon charm.
charm-helpers
- Update the broker request handler to take into consideration the new rbd-mirroring-mode flag.
- Update CephContext to ignore advertised default RBD features from the relation with ceph-mon, if the client charm implements the new rbd-mirroring-mode flag, and that’s set to image.
- Add Ceph relation name parameter to the CephContext constructor. The current implementation assumes that the clients will name their relation ceph. This will remove the hard-coded value, and make it the default parameter value. Thus, we will maintain the backwards compatibility.
  
  This is going to be useful for charms that implement the ceph-client interface, but the relation name is not ceph.

Repositories

openstack/charm-cinder-ceph
openstack/charm-nova-compute
openstack/charm-ceph-mon
openstack/charm-ceph-rbd-mirror
juju/charm-helpers

Documentation

The new rbd-mirroring-mode config option will be documented in the cinder-ceph charm, and in the Ceph RBD mirroring charm deployment guide.

Security

cinder-ceph
- It requires connection credentials to the remote Ceph cluster where the images are being mirrored by the Ceph RBD mirroring daemon.
  
  These are passed to cinder principal charm via the container-scoped relation. They will go into the cinder.conf, under the backend with replication enabled.

Testing

Code written or changed will be covered by unit tests; functional testing will be implemented using the Zaza framework.

Dependencies

No new dependencies.

OpenStack Compute Nodes Juju Availability Zones Sync

Sat, 31 Jul 2021 00:00:00

For some providers, Juju may set the environment variable JUJU_AVAILABILITY_ZONE with information about the availability zone of the machine deployed.

OpenStack has the concept of availability zones too, and it would be useful for Juju operators to have the option of automatically sync the Juju availability zones with the OpenStack availability zones.

Problem Description

At the moment, a Juju operator needs to manually sync the nova-compute Juju machines’ availability zones (AZs), with the OpenStack AZs.

Proposed Change

A new Juju action to the nova-cloud-controller charm will be implemented to automatically sync the Juju AZs with the OpenStack AZs. The availability zone for each Juju unit is exposed to the nova-cloud-controller charm through the cloud-compute relation.

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: ionutbalutoiu

Gerrit Topic

Use Gerrit topic “compute-nodes-az-sync” for all patches related to this spec.

git-review -t compute-nodes-az-sync

Work Items

nova-compute
- Set a new relation variable called availability_zone in the cloud-compute relation (used to relate to the nova-cloud-controller application). This variable will take the value of the automatic Juju variable JUJU_AVAILABILITY_ZONE, if this is set by the underlying provider.
nova-cloud-controller
- Implement a new Juju action called sync-compute-availability-zones used to sync the Juju availability zones, set by the remote units in the cloud-compute relation, with the OpenStack availability zones.

Repositories

openstack/charm-nova-compute
openstack/charm-nova-cloud-controller

Documentation

The new Juju action needs to be documented in the appropriate docs that reference the nova-cloud-controller actions. This is the only user-facing change.

Security

None

Testing

Code written or changed will be covered by unit tests.

Dependencies

The nova-cloud-controller charm does not install the Nova client Python library. This is needed for the new Juju action, so it is added to the charm Python dependencies.

Deferred Service Restarts

Sat, 31 Jul 2021 00:00:00

When a charm or package is upgraded services are restarted. This can impact the data plane causing outages. This spec is aimed at giving the cloud operator the option of deferring these restarts to a later, more convenient, time.

Problem Description

One example of the issue would be a charm upgrade which includes a minor template change. Charm upgrades are executed on all units of an application simultaneously (the cloud operator has no control over this). The charms are designed to restart services when configuration files are changed so the same set of services, across all units, will be simultaneously restarted.

Another example is when a package update is triggered by the charm, it may be useful to defer any service restarts until after a lengthy package update completes.

Proposed Change

Add charm configuration option to control automatic service restarts.

enable-auto-restarts:
  type: boolean
  default: True
  description: |
    Allow the charm and packages to restart services automatically when
    required

Add charm action to run service restarts.

restart-services:
  description: |
    Restarts services this charm manages.
  params:
    deferred-only:
      type: boolean
      default: false
      description: |
        Only restart deferred services.
    services:
      type: string
      default: ""
      description: |
        Optional space seperated list of services to restart. If unset
        then it applies to all services the charm controls.
show-deferred-restarts:
    description: |
        Show the outstanding restarts

If a charm has enable-auto-restarts disabled this will be reflected in the charms workload status message. The starts that have been requested, but deferred, by either the charm or a package update can be seen by running the show-deferred-restarts action.

To block restarts, charms will first check whether enable-auto-restarts has been disabled and only perform the restart if permitted. Ideally preventing services from being stopped or restarted would be done within systemd but this does not seem to be possible in a clean way. Relying on the charm to police itself is not ideal as there are many places a restart command can emanate from and there is always the risk that a future charm update will introduce a restart call that does not honour enable-auto-restarts.

Packages may try and restart a service during an update. To stop this from happening a custom /usr/sbin/policy-rc.d script will be installed to block the package changes from causing any restarts. Charms will indicate which services they want to block restarts for by writing a file to /etc/policy-rc.d. /usr/sbin/policy-rc.d will collect all the files from /etc/policy-rc.d to construct a list of services which are not permitted to be restarted.

The flow for deferring restarts:

Operator updates application to set enable-auto-restarts=False
A /usr/sbin/policy-rc.d file is written
A charm specific file is placed in /etc/policy-rc.d which list which services should be blocked from restarting.

The flow for re-enabling restarts:

Operator updates application to set enable-auto-restarts=True
The charm specific file is removed from /etc/policy-rc.d
Optionally, the operator runs the restart-services action against each application unit.

The existing Pause and Resume actions will continue to operate as they do now irrespective of the value of enable-auto-restarts

Alternatives

The problem could be handled if Juju were to implement a feature allowing charm updates to be performed on a unit by unit basis.

Implementation

Assignee(s)

Primary assignee:: gnuoy

Gerrit Topic

Use Gerrit topic “deferred-restarts” for all patches related to this spec.

git-review -t deferred-restarts

Work Items

Write policy-rc.d script and add to charm-helpers
Write functions in charm-helpers for managing the masking of services and files in /etc/policy-rc.d.
Each charm will need to add the new action and configuration option and the charm will need to gate any service interupting restarts on the value of enable-auto-restarts in a similar way to the existing is_unit_paused_set

Repositories

No new repositories are required.

Documentation

The new actions will need to be documented in the charm-guide and the upgrade sections updated.

Security

None

Testing

The regularly scheduled upgrade testing would be a good place to test this feature.

Dependencies

None

NetApp Data ONTAP as Manila storage backend

Sat, 31 Jul 2021 00:00:00

The existing Manila NetApp Clustered Data ONTAP driver should be used to configure the shares storage backend.

Problem Description

There isn’t any backend configuration charm that relates to the principal Manila charm, and configures the NetApp storage backend.

Proposed Change

A new subordinate charm that relates to the Manila principal charm, and sends the proper NetApp Data ONTAP backend configuration.

The new charm is a backend configuration charm that allows relevant config options to connect to an already deployed NetApp Data ONTAP cluster. The config options need to suffice for the Manila backend configuration.

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: ionutbalutoiu

Gerrit Topic

Use Gerrit topic “manila-netapp” for all the patches related to this spec.

git-review -t manila-netapp

Work Items

manila-netapp
- Brand new subordinate charm that implements the manila-plugin interface, and it relates to the principal Manila charm. The charm is responsible to do the appropriate backend configuration to have NetApp Data ONTAP as Manila shares storage.

Repositories

openstack/charm-manila-netapp

Documentation

This change will require new documentation to the charm-guide, in addition to new charm documentation.

Security

The Manila NetApp driver needs the admin credentials to manage the external storage cluster. These credentials will be stored in the charm config, and they will be part of the Manila backend configuration.

Testing

Unit tests will be added to cover the new charm functionality.

New functional tests will be added to validate the new charm, including end-to-end testing of the entire solution.

Dependencies

The packages that are required for this work are already included in the Ubuntu archives. There are no other new, external dependencies.

The Title of Your Specification

Sat, 31 Jul 2021 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

Ceph BlueStore Compression

Thu, 29 Oct 2020 00:00:00

The Ceph BlueStore storage backend has support for inline compression.

Problem Description

The Ceph charms currently do not provide documentation, validation or support for enabling BlueStore inline compression.

Proposed Change

Ceph supports configuring compression through per-pool properties and global configuration options. What configuration to use depends on the type of data and the characteristics of the underlying storage devices in use.

To accomodate this we should allow Ceph consuming charms to configure compression on a per-pool basis and have centrally managed defaults for any options not provided.

The following configuration options will be added:

bluestore-compression-algorithm
bluestore-compression-mode
bluestore-compression-required-ratio
bluestore-compression-min-blob-size
bluestore-compression-min-blob-size-hdd
bluestore-compression-min-blob-size-ssd
bluestore-compression-max-blob-size
bluestore-compression-max-blob-size-hdd
bluestore-compression-max-blob-size-ssd

Support will be provided for Ceph Mimic and onwards on Ubuntu 18.04 LTS and onwards.

Alternatives

None.

Implementation

Assignee(s)

Primary assignee:: fnordahl

Gerrit Topic

Use Gerrit topic “ceph-bluestore-compression” for all patches related to this spec.

git-review -t ceph-bluestore-compression

Work Items

Extend the Ceph broker API to support providing compression options on pool requests.
Add context that can be used by both consuming and providing charms to map charm configuration options into CephBrokerRq op properties as well as Jinja2 template context data.
Extend the ceph-client and ceph-rbd-mirror reactive interfaces to support relaying compression options for pool creation.
Add charm configuration options to control global compression defaults to the ceph-osd charm.
Add charm configuration options to control compression on pool creation to the following charms:
- glance
- cinder-ceph
- nova-compute
- gnocchi
- ceph-iscsi
- ceph-radosgw
Add support for copying compression properties on pool mirroring to the ceph-rbd-mirror charm.
Establish a performance baseline and perform benchmark for use in a hyper-converged deployment topology and document results and recommendations.

Repositories

No new repositories are required to implement this spec.

Documentation

The individual configuration options will get a description that explains basic usage.

A new appendix or section should be added to the deployment-guide to document results of benchmark and recommendations for deployment of Ceph with BlueStore Compression.

Security

Apart from enabling code paths in Ceph that was previously not exposed in the charms, there are no security specific considerations for this feature work.

Testing

Existing charm gates should be extended to validate the operation of the new charm configuration options.

No special hardware is required to validate the feature.

Dependencies

None.

Ceph Erasure Coded Pool Support

Thu, 29 Oct 2020 00:00:00

Problem Description

Ceph Erasure Coded pools provide an alternative to Replicated pools with comparable performance whilst requiring less raw block device storage for the same effective usable capacity.

The Ceph charms and charms that consume ceph services don’t currently provide support for use of Erasure Coded pools.

Proposed Change

The ceph-{mon,proxy} broker will be extended to support the creation of erasure-coded pools. Some code exists in the broker to support EC pools however it has never been tested so may require some updates and refactoring.

Some existing support exists in both charmhelpers and charms.ceph to support the creation of erasure coded pools; however it is largely untested and will need to be validated and updated as part of the work to enable support for Erasure Coding in the OpenStack Charms. Existing gaps include support for EC overwrites (required for most RBD use-cases) and providing the minimum size for the EC pool.

Erasure Coding has been supported by Ceph for some time however overwrites (required for RBD uses cases) where not implemented until Luminous so the minimum version requirement for Ceph is Luminous as shipped with Ubuntu 18.04 LTS.

Ceph deployments must be using Bluestore - Filestore based deployments cannot support RBD use-cases without the use of a cache tier (which is not supported in charmed deployments).

Consuming charms will be updated to include new configuration options to support the use of EC pools - this includes the following charms:

glance (rbd)
cinder-ceph (rbd)
nova-compute (rbd)
ceph-radosgw
ceph-iscsi (rbd)
ceph-fs

Erasure Coding charm options
Option	Type	Default Value	Description
pool-type	string	replicated	Ceph pool type to use for storage - valid values are ‘replicated’ and ‘erasure-coded’.
ec-profile-name	string		Name for the EC profile to be created for the EC pools. If not defined a profile name will be generated based on the name of the pool used by the application.
ec-rbd-metadata-pool	string		Name of the metadata pool to be created (for RBD use-cases). If not defined a metadata pool name will be generated based on the name of the data pool used by the application. The metadata pool is always replicated (not erasure coded).
ec-profile-k	int	1	Number of data chunks that will be used for EC data pool. K+M factors should never be greater than the number of available AZs for balancing.
ec-profile-m	int	2	Number of coding chunks that will be used for EC data pool. K+M factors should never be greater than number of available AZs for balancing.
ec-profile-locality	int		(lrc plugin - l) Group the coding and data chunks into sets of size l. For instance, for k=4 and m=2, when l=3 two groups of three are created. Each set can be recovered without reading chunks from another set. Note that using the lrc plugin does incur more raw storage usage than isa or jerasure in order to reduce the cost of recovery operations.
ec-profile-crush-locality	string		(lrc plugin) The type of the crush bucket in which each set of chunks defined by l will be stored. For instance, if it is set to rack, each group of l chunks will be placed in a different rack. It is used to create a CRUSH rule step such as ‘step choose rack’. If it is not set, no such grouping is done.
ec-profile-durability-estimator	int		(shec plugin - c) The number of parity chunks each of which includes each data chunk in its calculation range. The number is used as a durability estimator. For instance, if c=2, 2 OSDs can be down without losing data.
ec-profile-helper-chunks	int		(clay plugin - d) Number of OSDs requested to send data during recovery of a single chunk. d needs to be chosen such that k+1 <= d <= k+m-1. The larger the d, the better the savings.
ec-profile-scalar-mds	string		(clay plugin) Specifies the plugin that is used as a building block in the layered construction. It can be one of: jerasure, isa or shec.
ec-profile-plugin	string	jerasure	EC plugin to use for this applications pool. These plugins are available: jerasure, lrc, isa, shec, clay.
ec-profile-technique	string	reed_sol_van	EC profile technique used for this applications pool - will be validated based on the plugin configured via ec-profile-plugin. Supported techniques are ‘reed_sol_van’, ‘reed_sol_r6_op’, ‘cauchy_orig’, ‘cauchy_good’, ‘liber8tion’ for jerasure, ‘reed_sol_van’, ‘cauchy’ for isa and ‘single’, ‘multiple’ for shec.
ec-profile-device-class	string		Device class from CRUSH map to use for placement groups for erasure profile - valid values: ssd, hdd or nvme (or leave unset to not use a device class).

RBD use cases require a metadata pool which is always configured as a replicated pool in addition to the underlying EC coded pool for volume data storage. The consuming application is configured to use the metadata pool, with the data pool being configured via ‘ceph.conf’ for the specific application - for example:

rbd default data pool = glance

As OpenStack services specify a full path to an application specific ceph.conf configuration file, the default data pool configuration does not need to be scoped to a specific client identifier.

The ceph-mon interface implementations for both reactive and operator framework charms will be updated to include support for EC.

OpenStack version support will be validated as part of the implementation of this specification.

This feature is not dependent on a specific version of Juju.

Alternatives

EC pools can be created post deployment using actions provided by the ceph-mon charm. However other than forking the consuming charms, no way currently exists to provide the required configuration to Ceph and OpenStack services to consume EC pools in RBD use-cases.

Implementation

Assignee(s)

Primary assignee:

james-page
gnuoy

Gerrit Topic

Use Gerrit topic “ceph-erasure-coding” for all patches related to this spec.

git-review -t ceph-erasure-coding

Work Items

Ceph core EC enablement:

charms.ceph: Review and add required features for RBD EC pool usage.
charmhelpers: Review and add required features for RBD EC pool usage.
ceph-mon: Resync updates to charms.ceph + charmhelpers and update the broker code base to support any required changes.
ceph-proxy: Resync updates to charms.ceph + charmhelpers and update the broker code base to support any required changes.

Interfaces:

charm-interface-ceph-client: Update to add support for EC pool and profile creation.
ops-interface-ceph-client: Update to add support for EC pool and profile creation.
charm-interface-ceph-mds: Update to add support for EC pool and profile creation.
charm-interface-ceph-rbd-mirror: Review for any EC pool requirements based on support for EC in ceph rbd-mirror.

OpenStack RBD use cases:

glance: Add required configuration option support and updates to the ceph broker interaction to support EC pools. Add functional testing to cover at least earliest and latest supported releases as part of a recheck-full pre-commit test run.
cinder-ceph: Add required configuration option support and updates to the ceph broker interaction to support EC pools. Add functional testing to cover at least earliest and latest supported releases as part of a recheck-full pre-commit test run.
nova-compute: Add required configuration option support and updates to the ceph broker interaction to support EC pools. Add functional testing to cover at least earliest and latest supported releases as part of a recheck-full pre-commit test run.

Ceph RBD use cases:

ceph-iscsi: Add required configuration option support and updates to the ceph broker interaction to support EC pools. Add functional testing to cover at least earliest and latest supported releases as part of a recheck-full pre-commit test run.
ceph-rbd-mirror: Validate EC pools are not supported with rbd-mirror, document in release notes and charm deployment guide.

Other use cases:

ceph-radosgw: Add required configuration option support and updates to the ceph broker interaction to support EC pools. Add functional testing to cover at least earliest and latest supported releases as part of a recheck-full pre-commit test run. All pools aside from the .data pool must continue to be replicated pools.
ceph-radosgw: Validate support for multi-site RADOS gateway replication.
ceph-fs: Add required configuration option support and updates to the ceph broker interaction to support EC pools. Add functional testing to cover at least earliest and latest supported releases as part of a recheck-full pre-commit test run.

Repositories

No new git repositories are required.

Documentation

The charm deployment guide will be updated to detail use of the OpenStack Charms with Ceph Erasure Coded pools. This will include details on how to structure zones within a Ceph deployment using EC pools as the requirements are potentially quite different to replicated pools.

Security

No additional security attack surface is exposed by use of this feature.

Testing

No special hardware is required for testing of EC pools.

Functional tests will be added to consuming charms to validate the creation and use of EC pools across OpenStack and Ceph use-cases.

Performance testing of EC pools will be undertaken as part of this spec to benchmark EC pool configurations against replicated pool configurations.

Dependencies

No dependencies on other specs.

Ceph iSCSI Gateway with Operator Framework

Thu, 29 Oct 2020 00:00:00

The aim of this piece of work is to produce an Ceph iSCSI Gateway charm written in the operator framework.

Problem Description

There is currently no charmed mechanism for providing Ceph backed block devices to clients which cannot mount rbd devices directly. In addition a new charm framework is under development and this charm is an opportunity to test it for writing OpenStack charms.

Proposed Change

To write a ceph-iscsi charm using the operator framework. The charm will be marked as a preview charm but should have functional equivalence with a corresponding OpenStack charm written using the reactive framework.

The ceph-iscsi charm should interact with the Juju environment via the operator framework. This means that calls to libraries like charmhelpers.core.hookenv should be avoided. This applies to all aspects of the charm including interfaces.

What versions of the operating system are affected or required? Focal or Bionic + HWE Kernel

What version of Juju is required? 2.7+

Alternatives

Write the charm using classic or reactive framework.

Implementation

Assignee(s)

Primary assignee:: Liam Young <lp:gnuoy>

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t ceph-iscsi

Work Items

The implementation consists of the following items:

Validate that Ubuntu ceph-iscsi packages are working on Focal
Plan charm interfaces, supporting modules etc
Decide how dependencies are to be handled (pip, git submodulees etc)
Decide on how framework and dependency versions will be pinned during development and upon release.
Write basic charm to deploy and configure gateway services.
Write functional tests to perform a multipath mount a iscsi volume.
Write README
Certificates interface
Security checklist
Add pause & resume
Implement ceph-iscsi api security (admin_password, trusted_ips etc)
zaza tests for creating nd mounting a target, doing basic i/o and initiator reconnect testing.
Add iscsi target create action
Ensure update statue detects missing or incomplete relations.
Add series upgrade

Repositories

This will almost certainly change but:

Documentation

Any additional documentation will be considered for charm-deployment-guide.

Security

The gateway api will be secured with a password and access limited to the IP addresses of the other gateways.

The certificates interface will also be implemented to encrypt the api endpoint.

Testing

The charm will have the standard lint, unit and functional tests. The functional tests will be written in zaza.

Dependencies

The operator framework having sufficient maturity.
A way to reuse existing code is identified. It would probably be unacceptable to add a third version of common functions.

The Title of Your Specification

Thu, 29 Oct 2020 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

CephFS NFS with Manila

Thu, 09 Jul 2020 00:00:00

CephFS should be used to provide shared storage to OpenStack servers via Manila.

Problem Description

The existing CephFS charm doesn’t have any usable integration with Manila.

Proposed Change

To add support for CephFS as a shared filesystem for an OpenStack Charm deployed OpenStack, a few things are required. Primarily, a new charm will be needed to bridge the NFS to CephFS protocols. The best option for this application is Ganesha. In addition to charming Ganesha, additional work will need to be done to improve the testability and supportablity of the existing CephFS charm.

To allow the new Ganesha charm to integrate with Manila, a new remote-manila-plugin interface will be added to the existing Manila charm that will act exactly the same as the existing manila-plugin interface, but will not be container scoped. This remote-manila-plugin interface will be used to disable the manila-share service on the Manila charm. The new manila-ganesha charm will configure the manila-share service in addition to Ganesha.

Access to the Ganesha network will include security groups restricting tenants on the provider network from being able to access anything over that network excepting their own NFS share, and IP restrictions will be used on the NFS shares to restrict tenants from being able to access each others’ shares. This provider network will be documented as requiring creation by the OpenStack administrator. The network will need to be created to match the space that the Ganesha charm provides its NFS service over.

Alternatives

As an alternative to installing Ganesha as the NFS gateway, it is possible to implement a shared filesystem via CephFS and Manila directly. This approach is not preferred because of the requirement to add the ceph-fs clients to all tenants that would like to access the shared filesystems. In addition, letting clients connect directly to Ceph reqiures exposing the Ceph cluster (mons and OSDs) directly to tenant workloads, dramatically increasing the security exposure to the shared storage.

Implementation

A bundle overlay to add Ceph backed storage to Manila with Ganesha would look like:

applications:
  manila-ganesha:
    charm: manila-ganesha
    num_units: 2
relations:
  - - ceph-mon
    - ganesha
  - - manila-ganesha:remote-manila-plugin
    - manila:remote-manila-plugin

A more complete bundle could look like:

applications:
  ceph-mon:
    charm: cs:~openstack-charmers-next/ceph-mon
    num_units: 3
  ceph-osd:
    charm: cs:~openstack-charmers-next/ceph-osd
    num_units: 3
    storage:
      osd-devices: '10g'
  ceph-mds:
    charm: cs:~openstack-charmers-next/ceph-mds
    num_units: 2
  manila-ganesha:
    charm: manila-ganesha
    num_units: 2
  manila:
    charm: cs:~openstack-charmers-next/manila
    num_units: 1
relations:
  - - ceph-mon
    - manila-ganesha
  - - ceph-mon
    - ceph-osd
  - - manila-ganesha:remote-manila-plugin
    - manila:remote-manila-plugin

Assignee(s)

Primary assignee:: Chris.MacNaughton

Gerrit Topic

Use Gerrit topic “charm-cephfs-manila” for all patches related to this spec.

git-review -t charm-cephfs-manila

Work Items

New Manila-Ganesha charm
Updates to the Manila charm
Improvements to CephFS charm

Repositories

There will be a few new repositories needed:

charm-manila-ganesha

In addition, charm-manila and charm-ceph-fs charm will be updated.

Documentation

This change will require new documentation to the charm-guide, in addition to new documentation in the new charms. Additionally, charm-ceph-fs will need updated documentation to illustrate new functionality.

Security

This change will require some validation of the permission model and restrictions on the ceph storage provided by manila to ensure that accessing a share doesn’t break tenant restrictions.

Testing

Unit tests will be added to each charm to cover any new functionality.

New functional tests will be added to each of the new and existing charms to validate functionality, including end-to-end testing of the entire solution.

Dependencies

The packages that are required for this work are already included in the Ubuntu archives in Universe. The required packages in Universe (ganesha-nfs) will be proposed for inclusion to Main as a part of this work. There are no other new, external dependencies. The first release that will be supported for Manila with CephFS and Ganesha is Bionic Rocky.

ML2/OVS Mellanox Hardware Offload

Thu, 09 Jul 2020 00:00:00

High end networking adapters such as the Mellanox Connect-X series support up to 200Gbps networking; instances using traditional software virtualized networking via OVS and virtio-net cannot achieve networking throughput of more than 10Gbps without a high CPU load; it’s possible to use SR-IOV to increase performance to nearer 100Gbps but that has limitations in terms of the flexibility of networking options (no support for overlay networks) and no support for security groups.

The Mellanox Connect-X 5+ supports hardware offload via a feature called ASAP2; integrated with Open vSwitch (OVS), this pushes the network data path from the kernel directly onto a high performance embedded switch on the network card. This will allow instances to achieve much higher network performance with very low CPU overheads as well as utilizing as much of the overall network capacity to the hypervisor as possible.

Problem Description

Hardware offload support is not currently enabled in the OpenStack charms.

Proposed Change

The neutron-openvswitch charm will be updated to support configuration of Mellanox Connect-X 5+ network adapters to support hardware offloading (card needs to be placed in switchdev mode on boot); OVS will be configured to make use of hardware offloading where possible.

A new tool (mlnx-switchdev-boot) will be developed to configure the Connect-X card in the right mode on boot prior to it being used by OVS for networking. This may be superseded at some future date by features in networkd and/or netplan.

This feature does make use of a number of leading edge features across the hypervisor software stack:

Linux Kernel >= 5.2
OVS >= 2.11.0

This limits the scope of support to OpenStack Stein or later in-conjunction with the Ubuntu LTS hardware enablement kernel from Ubuntu 19.10.

The Connect-X series of cards also supports Virtual Function (VF) Link Aggregation (LAG) providing upstream switch/cable/port resilience for hardware offloaded ports which are supported using a VF on the underlying network card - this is driven by the normal configuration process for bonding network devices via Linux - in this case the Physical Functions (PF’s) for the network cards.

The existing SR-IOV VF configuration code and scripts in the neutron-openvswitch charm is a little inconsistent and not very reliable on reboot; this change will cover refactoring the SR-IOV VF code into a new tool (sriov-netplan-shim) dealing with configuration of VF functions via the charm and on reboot. This tool will be superceeded by SR-IOV support in netplan at some future date.

Note

The most recent kernel and OVS versions don’t yet support connection tracking offload which means that security groups cannot be offloaded to the network card switch; this is under development and will land in a later release of OVS and Linux.

Note

The total number of hardware offloaded ports a hypervisor can support is limited by the total number of VF’s that the underlying network cards can support.

Alternatives

It’s worth noting that for VLAN or flat networking requirements, it’s possible to achieve high throughput networking to instances by using SR-IOV which is already supported by the neutron-openvswitch charm.

Implementation

Assignee(s)

Primary assignees:: wgrant james-page

Gerrit Topic

Use Gerrit topic “mlnx-hardware-offload” for all patches related to this spec.

git-review -t mlnx-hardware-offload

Work Items

Develop and test tool for configuration of Connect-X adapters for hardware offload support.

Update neutron-openvswitch to support configuration of OVS in hardware offload mode.

Add appendix to charm deployment guide to detail configuration and use of OpenStack with Mellanox Connect-X 5+ cards in hardware offload mode.

Repositories

A new repository (mlnx-switchdev-mode) will be required for tooling to configure the Mellanox Connect-X networking cards on boot. This tool has scope outside of the OpenStack charms project so can be developed on github.

A new repository (sriov-netplan-shim) will be required for tooling to configure SR-IOV VF functions; this tool will superceed the existing SR-IOV VF configuration code in the neutron-openvswitch charm.

Documentation

Documentation of this feature will be provided in the charm deployment guide.

Security

No additional security risks are introduced by this feature.

Testing

Testing of this feature requires specific hardware in the form of Mellanox Connect-X 5+ networking cards.

Dependencies

Linux >= 5.2
OVS >= 2.11.0
Mellanox Firmware >= 16.26.1040

MySQL 8

Thu, 09 Jul 2020 00:00:00

Introduce charmed and enterprise grade MySQL 8 + InnoDB + MySQL Router as a database solution for OpenStack models with the intention of making it the default as of Ubuntu 20.04.

Problem Description

Percona XtraDB Cluster is in Universe, has failed a Main Inclusion Request process, and will not be carried into the next LTS relative to Charmed OpenStack deployments.

Proposed Change

These will be wholly new charms, not based in any way on existing MySQL or Percona-Cluster charms, and without a charm upgrade path.

Instead of in-place upgrades a migration solution will be provided to allow existing Ubuntu 18.04 LTS deployments using the percona-cluster charm to migrate databases directly to an Ubuntu 20.04 LTS deployment based on mysql-innodb-cluster. Full details will be provided as an update to this spec.

The charms will be promulgated to the following locations:

cs:mysql-innodb-cluster
cs:mysql-router

What versions of the operating system are affected or required? 19.10 (Eoan) and 20.04

What versions of OpenStack are affected or required? Train and above

What version of Juju is required? Juju 2.7+

Alternatives

The alternative is to bring percona-cluster into main for 20.04.

Implementation

Assignee(s)

Primary assignee:: thedac

Gerrit Topic

Use Gerrit topic “mysql8” for all patches related to this spec.

git-review -t mysql8

Work Items

The implementation consists of the following charms:

mysql-innodb-cluster

The primary charm that will handle the mysql implementation.
- Install MySQL 8
- Handle InnoDB clustering
interface-mysql-innodb-cluster

The peer relationship for mysql-innodb-cluster
interface-mysql-shared provides

The interface mysql-shared provides side
mysql-router

The subordinate charm that will relate to the application charm and to mysql-innodb-cluster.
- Proxies DB requests and responses
interface-mysql-router

The interface between the application and mysql-router
mysqlsh snap

The mysqlsh tool that manages the mysql innodb cluster
- snapped from upstream source

Repositories

charm-mysql-innodb-cluster
charm-interface-mysql-innodb-cluster
charm-mysql-router
charm-interface-mysql-router
mysqlsh-snap

Documentation

Each charm and interface will contain a README with instructions on deploying and relating the charm.

Potentially an OpenStack deployment guide appendix to discuss clustering and cluster management.

Security

MySQL 8 will now be in main with security auditing which percona-cluster never had.

The code reviews of the work items should include a security conscious review approach.

Testing

Unit tests for each repository.

Zaza functional tests for each charm. This will include end to end testing with applications, mysql-router and mysql-inndob-cluster.

Dependencies

Eoan+ mysql8 packaging
MySQL shell

Enablement of S3 storage backend for Gnocchi

Thu, 09 Jul 2020 00:00:00

Problem Description

The gnocchi charm is a multi-tenant times eries, metrics and resources database. It currently only supports connections to a Ceph storage backend. For anyone using a S3 storage backend, this is an issue.

Proposed Change

The proposed solution is to make the choice of storage backend a configuration option. The relation between gnocchi and ceph-mon would become an optional one, that would be activated only when Ceph is the selected backend.

The upstream documentation of Gnocchi mentions that S3 storage backends are supported, along with others, such as Swift, Redis and Ceph (https://gnocchi.xyz/install.html). The new feature would be supported on OpenStack Stein (and later) on Ubuntu 18.04 LTS.

Alternatives

None.

Implementation

Assignee(s)

Primary assignee: camille.rodriguez

Gerrit Topic

Use Gerrit topic “s3-storage” for all patches related to this spec.

git-review -t s3-storage

Work Items

Add a charm option for selecting S3
Add options for S3 in the configuration file:
- S3 endpoint URL
- S3 region name
- S3 access key id
- S3 secret access key
- Prefix to namespace metric bucket: s3_bucket_prefix = gnocchi
- Maximum time to wait checking data consistency when writing toi S3: s3_check_consistency_timeout = 60
- The maximum number of connections to keep in a connection pool: s3_max_pool_connections = 50
Make the Ceph relation and configuration optional, dependent on the charm option
Implement https endpoint for S3

Timeline

The goal is to implement this change in the OpenstStack Charms 20.08 release. The freeze date for this release is July 24th 2020, for a release on Augusti 5th (see release schedule). This change should be proposed for merging ati least two weeks ahead of freeze, so ideally submitted by July 10th 2020.

Repositories

Changes will be pushed to the existing gnocchi charm repository:

https://git.openstack.org/openstack/charm-gnocchi

Documentation

Update of the charm options
Update of the charm README

Security

No additional security concerns. The feature will be compatible with an HTTPS endpoint to allow for encryption.

Testing

Code changes will be covered by unit tests. Functional tests would require S3 storage hardware or software emulation. This is possible through a Swift deployment, or with RADOS Gateway. The deployment of the storage backend with S3 capability would be independent from the OpenStack bundle, to represent a real scenario.

Dependencies

No dependencies outside of this specification.

The Title of Your Specification

Thu, 09 Jul 2020 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

Routed Provider Networks for Neutron Charms

Sun, 23 Feb 2020 00:00:00

As L3-oriented fabrics get more widely deployed in DCs there is a need to support multi-segment networks in Neutron charms. Routed provider networks were implemented in Neutron itself in order to model deployments with multiple isolated L2 networks each with their own VLANs (switch fabrics) and network or compute nodes attached to those networks.

Problem Description

Deployments that have machines connected to separate switch fabrics and hence provider networks need to be aware of multiple subnets attached to different L2 segments. While VLAN IDs on different switch fabrics may match, they define isolated broadcast domains and should have different subnets used on them for proper addressing and routing to be implemented.

Provider Networks in Neutron used to have a single segment but starting with Newton there is support for multi-segment networks which are called routed provider networks. From the data model perspective a subnet is associated with a network segment and a given provider network becomes multi-segment as a result of subnet to segment assignments. This concept is the same as creating a Virtual Routing and Forwarding (VRF) instance because end hosts on an L3 network are assumed to be in a single address space but possibly different subnets and routing needs to be implemented to make sure all hosts and instances are mutually L3-reachable.

Considerations for routed provider networks include:

scheduling (Neutron needs to use Nova Placement API);
subnets become segment-aware;
ports become segment-aware;
IP address to port assignment is deferred until a segment is known;
placement with IP address exaustion awareness;
live migration and evacutation of instances (constrain to the same L2);
DHCP and meta-data agent scheduling.

A nova spec for routed provider networks contains a detailed overview of the implemenation details to address the above points, including usage of generic resource pools.

In order to account for IPv4 address ranges associated with segments Neutron uses Nova placement API to create IPv4 inventories and update them if needed. Neutron also creates a Nova host aggregate per segment.

There are several points to consider when it comes to charm modifications to support this functionality:

“segments” service plugin enablement (unconditionally or via an option);
Deployment scenarios in conjunction with other features (L3 HA, DVR, BGP);
Floating IP usage;
Impact on existing deployments (upgrades).

Also, from the Charm and Juju perspective the following needs to be addressed:

Handling of physnet mappings on different switch fabrics using Juju AZs or tag contraints;
Appropriate placement of units to make sure necessary agents are deployed.

Deployment Scenarios

Deployment scenarios should include both setups with charm-neutron-gateway and without it. Moreover, for the use-case where charm neutron-gateway is present we need to account for DVR and L3 HA cases.

provider networks at networks nodes only (SNAT and FIP sub-cases);
provider networks on compute nodes without dedicated network nodes;
OVS HA using VRRP;
DVR and L3 HA using VRRP;
Deployments with dynamic routing agents;

Proposed Change

Given that there are certain limitations in how routed provider networks are implemented in Newton release this spec is targeted at Ocata release.

Segments Service Plugin

A related Ocata documentation section contains 3 major configuration requirements:

“segments” needs to be added to the service_plugins list (neutron-api);
A compute placement API section needs to be added to neutron.conf;
Physical-network related config options on gateway and compute nodes need to match a given switch fabric configuration (see a section on that below);

The proposed approach, therefore, is as follows:

Unconditionally render “segments” service_plugin in charm-neutron-api if a deployed OpenStack version is Ocata or newer;
Add Nova placement API section rendering to charm-neutron-api if a deployed OpenStack version is Ocata or newer.

The reason for unconditional addition of the segments service plugin is due to how feature is implemented and documented.

Unless a subnet is associated with a segment, a provider network is considered to be single-segment and old behavior applies. There is also a requirement that either all subnets associated with a network need to have a segment or none of them must have one. This is documented both in public documentation and in code.

The Networking service enforces that either zero or all subnets on a particular network associate with a segment. For example, attempting to create a subnet without a segment on a network containing subnets with segments generates an error.

This distiction is what allows unconditional addition of this service plugin for the standard ML2 core plugin.

Multiple L2 Networks (Switch Fabrics)

Each network segment includes network type information this means that for ML2 plugin flat_networks and network_vlan_ranges options there may be different values depending on a switch fabric. In this case, there need to be multiple copies of the same charm deployed as applications with different names to create per-fabric configurations. This can be modeled in Juju in terms of availability zones although this mapping is not exact because there may be several availability zones on the same switch fabric or there may be a fabric per availability zone. In a general case, tags contraints can be used in Juju to provide placement metadata related to switch fabrics. Either way, to achieve the separation of config namespaces the following charms may need per-fabric configuration:

charm-neutron-openvswitch (subordinate, provider networks or DVR cases);
charm-nova-compute (as a primary for neutron-ovs charm - defines placement);
charm-neutron-gateway (in scenarios where it is deployed).

Neutron charms (ovs or gateway) contain the following fabric-specific options:

bridge-mappings;
flat-provider-networks;
vlan-ranges;
enable-local-dhcp-and-metadata (needs to be true everywhere);

Albeit it would be tempting to assume that this configuration will be the same for all nodes in a given deployment and that switches the nodes will be connected to will have identical configuration, we need to account for a general case.

Multi-application approach allows to avoid any charm modifications besides charm-neutron-api and have the following content in bundle.yaml:

variables:
   data-port:           &data-port           br-data:bond1
   vlan-ranges:         &vlan-ranges         provider-fab1:2:3 provider-fab2:4:5 provider-fab3:6:7
   bridge-mappings-fab1: &bridge-mappings-fab1 provider-fab1:br-data
   bridge-mappings-fab2: &bridge-mappings-fab2 provider-fab2:br-data
   bridge-mappings-fab3: &bridge-mappings-fab3 provider-fab3:br-data
   vlan-ranges-fab1:     &vlan-ranges-fab1     provider-fab1:2:3
   vlan-ranges-fab2:     &vlan-ranges-fab2     provider-fab2:4:5
   vlan-ranges-fab3:     &vlan-ranges-fab3     provider-fab3:6:7

# allocate machines such that there are enough
# machines in attached to each switch fabric
# fabrics do not necessarily correspond to
# availability zones
machines:
   "0":
     constraints: tags=compute,fab1
   "1":
     constraints: tags=compute,fab1
   "2":
     constraints: tags=compute,fab1
   "3":
     constraints: tags=compute,fab1
   "4":
     constraints: tags=compute,fab1
   "5":
     constraints: tags=compute,fab2
   "6":
     constraints: tags=compute,fab2
   "7":
     constraints: tags=compute,fab2
   "8":
     constraints: tags=compute,fab2
   "9":
     constraints: tags=compute,fab3
   "10":
     constraints: tags=compute,fab3
   "11":
     constraints: tags=compute,fab3
   "12":
     constraints: tags=compute,fab3
 services:
   nova-compute-kvm-fab1:
     charm: cs:nova-compute
     num_units: 5
     bindings:
     # ...
     options:
     # ...
       default-availability-zone: az1
     to:
     - 0
     - 1
     - 2
     - 3
     - 4
   nova-compute-kvm-fab2:
     charm: cs:nova-compute
     num_units: 5
     bindings:
     # ...
     options:
     # ...
       default-availability-zone: az2
     to:
     - 5
     - 6
     - 7
     - 8
   nova-compute-kvm-fab3:
     charm: cs:nova-compute
     num_units: 5
     bindings:
     # ...
     options:
     # ...
       default-availability-zone: az3
     to:
     - 9
     - 10
     - 11
     - 12
   neutron-openvswitch-fab1:
     charm: cs:neutron-openvswitch
     num_units: 0
     bindings:
       data: \*overlay-space-fab1
     options:
       bridge-mappings: \*bridge-mappings-fab1
       vlan-ranges: \*vlan-ranges-fab1
       prevent-arp-spoofing: True
       data-port: \*data-port
       enable-local-dhcp-and-metadata: True
   neutron-openvswitch-fab2:
     charm: cs:neutron-openvswitch
     num_units: 0
     bindings:
       data: \*overlay-space-fab2
     options:
       bridge-mappings: \*bridge-mappings-fab2
       vlan-ranges: \*vlan-ranges-fab2
       prevent-arp-spoofing: True
       data-port: \*data-port
       enable-local-dhcp-and-metadata: True
   neutron-openvswitch-fab3:
     charm: cs:neutron-openvswitch
     num_units: 0
     bindings:
       data: \*overlay-space-fab3
     options:
       bridge-mappings: \*bridge-mappings-fab3
       vlan-ranges: \*vlan-ranges-fab3
       prevent-arp-spoofing: True
       data-port: \*data-port
       enable-local-dhcp-and-metadata: True
 # each of the apps needs to be related appropriately
 # ...

The above bundle part is for a setup without charm-neutron-gateway, although it can be added easily using the same approach. Given that there are no relations between charm-neutron-gateway and charm-neutron-openvswitch there is no problem with relating per-fabric services. What has to be kept in mind is the infrastructure routing for overlay networks on different fabrics so that VXLAN or other tunnels can be created between endpoints.

Documented Requirements and Limitations

The Ocata Routed Provider Networks guide mentions scheduler limitations, however, they are made in reference to Newton and are likely outdated. Looking at the Newton documentation it is certain that this documentation section was not updated for Ocata.

Minimum API version requirements are present in the Pike release documentation

Floating IP usage is not possible with routed provider networks at the time of writing (Pike) due to the fact that a routed provider network cannot be used with external-net extension.

This might be considered a serious limitation, however:

Deployments that rely on routed provider networks are inherently L3-oriented and it is likely that NAT functionality will not be required that often anyway alleviating the need for floating IPs;
Routed provider networks are not enforced at deployment time.

An RFE exists to address this limitation based on subnet service types and dynamic routing.

Alternatives

N/A

Implementation

Assignee(s)

Primary assignee:: dmitriis

Gerrit Topic

Use Gerrit topic “1743743-fe-routed-provider-networks” for all patches related to this spec.

git-review -t 1743743-fe-routed-provider-networks

Work Items

add “segments” to service_plugins in charm-neutron-api for Ocata+;
add section-placement to charm-neutron-api and import it in Ocata+ templates

Repositories

No new repositories.

Documentation

Release notes should mention that this functionality can be used.

It might be worthwhile to create a dedicated guide to cover how different OpenStack network deployment scenarios map to charm options.

Security

No apparent security risks.

Testing

The following functional test can be introduced even on a single fabric but with different VLANs used to simulate separate fabrics and L3 connectivity:

deploy a bundle with two neutron-openvswitch and nova-compute charms and configure them with different provider network settings as described above;
create a multi-segment network as described in the documentation;
create several instances with interfaces attached to the routed provider network created above and make sure that L3 connectivity is provided externally between segments;
verify L3 connectivity between instances.

Dependencies

None

Keystone Fernet Token Implementation

Sun, 23 Feb 2020 00:00:00

Fernet tokens were added to OpenStack to solve the problem of keystone being required to persist tokens to the a common database (cluster) like UUID tokens, and solve the problem of size for PKI or PKIZ tokens.

From Fernet - Frequently Asked Questions

A fernet token is a bearer token that represents user authentication. Fernet tokens contain a limited amount of identity and authorization data in a MessagePacked payload. The payload is then wrapped as a Fernet message for transport, where Fernet provides the required web safe characteristics for use in URLs and headers. The data inside the fernet token is protected using symmetric encryption keys, or fernet keys.

Problem Description

Keystone has had support for Fernet tokens since Kilo, so all services support fernet token authorization. In the upcoming Rocky release the sql token driver and uuid token provider will be removed. The main part of the work on this task is changing the keystone charm to enable configuration of fernet tokens, provide the initialisation of fernet tokens, provide rotation of fernet keys and their subsequent synchronisation to other keystone units.

There are also some issues around what aspects should be configurable vs controlled by the charm. The Proposed Change section provides more details about this.

Finally, upgrading an existing OpenStack implementation from another token system to Fernet tokens is discussed, and the support the charm(s) will need to implement to support this, along with documentation.

Proposed Change

Theory of Operation

The keystone-charm will, with the appropriate configuration options, configure keystone to generate fernet tokens.

In order to generate tokens, fernet keys are used. These are generated by keystone and have an expiry date. The key repository is a directory, and each key is an integer number, with the highest number being the primary key. Key ‘0’ is the staged key, that will be the next primary. Other keys are secondary keys.

New tokens are only ever generated from the primary key, whilst they secondary keys are used to validate existing tokens. The staging key is not used to generate tokens, but can be used to validate tokens as the staging key might be the new primary key on the master due to a rotation and the keys have not yet been synchronised across all the units.

Fernet keys need to be rotated at periodic intervals, and the keys need to be synchronised to each of the other keystone units. Keys should only be rotated on the master keystone unit, and must be synchronised before they are rotated again. “Over rotation” occurs if a unit rotates its keys such that there is no suitable decoding key on another unit that can decode a token that has been generated on the master. This happens if two key rotations are done on the master before a synchronisation has been successfully performed. This should be avoided. Over rotations can also cause validation keys to be removed before a token’s expiration which would result in failed validations.

There are 3 parts to the Key Rotation Strategy:

The rotation frequency
The token lifespan (set with [token] expiration)
The number of active keys: (set with [fernet_tokens] max_active_keys)

There needs to be at least 3 keys as a minumum. The actual number of keys is determined by the token lifespan and the rotation frequency. The max_active_keys must be one greater than the token lifespan / rotation frequency

To quote from the FAQ:

The number of max_active_keys for a deployment can be determined by dividing the token lifetime, in hours, by the frequency of rotation in hours and adding two. Better illustrated as:

token_expiration = 24
rotation_frequency = 6
max_active_keys = (token_expiration / rotation_frequency) + 2

In the keystone charm, there is already the config parameter token-expiration in seconds, and there will be a proposed config item of fernet-max-active-keys. Thus, the rotation frequence will be calucated as:

token_expiration = 24   # actually 3600, as it's in seconds
max_active_keys = 6
rotation_frequency = token_expiration / (max_active_keys - 2)

Thus, the fernet-max-active-keys can never be less than 3 (which will be enforced in the charm), which would make the rotation frequency the same as the token expiration time.

Upgrading an Existing System

To upgrade an existing system to use Fernet tokens, the keystone charm should be upgraded first. The (new) configuration option token-provider has a no default value set, which means on a pre-rocky install the token type will remain as UUID. In order to move a pre-rocky install to Fernet, the token-provider option should be set to fernet.

Note that if a pre-rocky system is upgraded to rocky, then the default token type will be fernet. The rocky cycle removes support for UUID tokens. Thus upgrading a system to rocky will automatically use fernet as the only token type. The token-provider option is only valid for pre-rocky systems.

Note that althought the charms enable token cachinng with memcache by default, this is only for the default of 300 seconds as the token_cache_time is not being set (see (Keystone) Middleware Architecture: Improving response time for further details. The impact of this is that some services may try to re-authenticate during the upgrade, and depending on which keystone unit they “pick”, and what stage the upgrade is at, will determine whether the action succeeds. As different services may be part of the same action, this might lead to odd failure modes. However, once the system is upgraded, after the default of 300 seconds, all services will continue to operate normally.

Therefore, there may be some percieved outage of the openstack control plane. Already running instances will not be affected and all services will continue to operate normally as soon as they request new tokens from keystone.

Additional Configuration Items

The following configuration items will be needed in the keystone charm.

token-provider - the token system to use: Either ‘uuid’ or ‘fernet’. The default will not be set. Pre-rocky systems will have a default of uuid. On rocky systems, the configuration option has no effect. As the default is uuid for pre-rocky systems, the token-provider won’t change on an upgrade unless the operator sets the configuration value to fernet.
fernet-max-active-keys - the maximum active keys configured in keystone. This controls the key rotation trigger times based on this config item and the config item token-expiration.

Keystone Actions

The following action will be required:

purge-tokens – purge existing tokens from the database. This is used after upgrading from UUID to Fernet tokens,

Internal Cron Jobs

The charm will set up a cron job to rotate the keys and then synchronise them to the other peered units. The cron job will call juju run from within the charm to rotate the keys and then synchronise the keys to the other peered units. It will also only perform this action if it is the leader. The cron job will run on all peered units, but only have an effect on the leader.

Synchronisation of the Fernet keys will be via Juju leader settings. The keys are small, and “leader settings” provides a convenient and secure mechanism to synchronise the keys between units without having to explicitly provide networking for all keystone peered units. The delay in transferring the keys using hooks is not an issue as the synchronisation does not need to be immediate; indeed, it could be just before the next key rotation in the worst case, although, this is extremely unlikely to be the case.

Alternatives

In the Openstack rocky release, fernet is the only token provider available. Therefore, there is no alternative.

Implementation

Assignee(s)

Primary assignee:: ajkavanagh
Secondary assignees:: fnordahl

Gerrit Topic

Use Gerrit topic “fernet-keystone-charm” for all patches related to this spec.

git-review -t fernet-keystone-charm

Work Items

Add fernet token functionality to the keystone-charm. This includes: * setup * upgrade * rotate / sync actions * cron job for automatic rotate / sync.
Add Fernet token information to the documentation: * charm-store text for keystone charm * Notes in the charm guide re: uuid vs Fernet.
Update tests: * Amulet/bundle for actions / verify installation. * Update other bundles to ensure defaults * Upgrade from uuid to Fernet tokens.

Repositories

No new git repositories required.

Documentation

Documentation will be provided as part of the keystone charm and notes in charm guide.

Security

A change of token provider does have security implications and well tested and proved best practices for using the fernet token provider will be implemented.

Testing

Unit tests will be developed along with new code. Functional tests will be implemented. A scenario test for change of token provider will also be written.

Dependencies

No external dependencies.

Neutron Dynamic Routing

Sun, 23 Feb 2020 00:00:00

OpenStack supports dynamic routing, specifically BGP. BGP forms a significant part of the routing infrastructure for many TelCos and large other organizations with complex networks. Our charms need to grow support for dynamic routing.

Problem Description

In a typical setup with Neutron Gateway, infrastructure routers do not have knowledge of the private networks allocated to OpenStack tenants. As a result, services external to OpenStack do not know how to route traffic to OpenStack instances. As OpenStack networks are elastic and therefore can evolve in many ways, a way of communicating changes between OpenStack and external routers is needed.

Since networks in OpenStack are self serviced, any addition or removal of OpenStack network should not depend on network administrator’s actions. Network limitations should be set up front. This is further facilitated by the use of subnet pools in OpenStack.

Other implications include decoupling subnets from layer 2, allowing different next-hops for floating IPs.

Proposed Change

Neutron dynamic routing solves this problem by introducing a BGP speaker, which then peers with the existing infrastructure router(s) and announces the next-hop virtual routers for OpenStack owned networks. This allows dynamic routing changes to be communicated to the physical infrastructure based on changes in OpenStack networks. Since OpenStack Networks are managed by users, address scope limitations need to be in place.

A new charm is proposed called neutron-dynamic-routing which will configure and deploy the neutron-bgp-dragent (Neutron BGP dynamic routing agent). The neutron-bgp-dragent is a BGP speaker which can be peered with existing routers in the BGP infrastructure. This charm will require interfaces for AMQP and potentially neutron-api-plugin (TBD). For testing purposes it will also make use of the bgp interface to peer with the quagga charm, both written by Frode Nordahl (See Testing below.)

Requirements

The charm will need to select an interface on the provider network for communication with OpenStack. It may also need to select an interface to speak with BGP routers in the infrastructure.

High availability is accomplished by deploying N number of units of neutron-dynamic-routing with care for placement on different physical hosts. This will introduce multiple neutron-bgp-dragents. BGP infrastructure routers will need to be manually informed of each neutron-bgp-dragent.

IPv4 and IPv6 support is handled with separate bgp-speakers defined for each.

The charm will need to support both DVR and centralized OpenStack routing.

Outside the scope

It is important to understand what the OpenStack dynamic routing implementation does not do and lies outside of the scope of this spec.

It does not provide ingress BGP to OpenStack, it is purely egress advertisement of OpenStack owned networks.
It does not automatically advertise all OpenStack networks or whole subnet pools. A considerable amount of administrator configuration is still required, including associating OpenStack networks with the bgp-speaker. See the Testing documentation for a sense of the administrative overhead required.
It does not provide a mechanism for injecting arbitrary BGP routes (for example for /32 FIPs). It merely advertises OpenStack networks that have been associated with the bgp speaker.

Alternatives

Floating IPs, and NAT in general, are one approach to this problem, but there are couple of drawbacks. For starters, NAT comes with a performance price as number of connections grows. In cloud, where many VMs can establish connections to various external peers, this can require significant memory and cpu resources. Neutron gateway doesn’t really scale without manual intervention. On top of that, if only NAT is used, peers can’t really know which instance established connection, they only know connection came ‘from the cloud’. Same problems apply to DVR.

Static routes on physical gateways are the closest thing to BGP solution. The only drawback is operational; in case address scope in Neutron changes, these changes need to be implemented on physical routers too.

Implementation

Assignee(s)

Primary assignee:: thedac
Additional work:: fnordahl

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t dragent

Work Items

New charm neutron-dynamic-routing

A new reactive charm will be developed to configure and deploy the neutron-bgp-dragent.

Update neutron-api charm

The neutron-api charm will need to load the service_plugin neutron_dynamic_routing.services.bgp.bgp_plugin.BgpPlugin and subsequently run a db migrate.

Potential need for new neutron-api-plugin interface

It is unclear at this time if a relationship to neutron-api will be required. If it is required no layered interface for neutron-api-plugin yet exists and will need to be created.

New charm quagga

For testing purposes the new quagga charm will act as a BGP peer to validate and test neutron-dynamic-routing. Frode Nordahl has already begun work on this project: https://github.com/openstack-charmers/charm-quagga

New interface bgp

For testing purposes the new bgp interface will define the peering relationship between neutron-dynamic-routing and the quagga charm. Frode Nordahl has already begun work on this project: https://github.com/openstack-charmers/interface-bgp

Repositories

A new repository for the neutron-dynamic-routing charm is needed.

https://git.openstack.org/openstack/charm-neutron-dynamic-routing

A new repository for the quagga charm is needed.

https://git.openstack.org/openstack-charmers/charm-quagga

A new repository for the bgp interface is needed.

https://git.openstack.org/openstack-charmers/interface-bgp

Documentation

An update to the charm deployment guide will be required.

Upstream documentation is found at the following: https://docs.openstack.org/neutron-dynamic-routing/latest/

Security

The neutron-dynamic-routing charm is dependent on the OpenStack implementation of BGP. Which uses a simple string as a password.

Testing

The neutron-dynamic-routing charm will need to be tested with a BGP peer. Frode Nordahl has begun work on the quagga charm which will act as an infrastructure BGP router allowing for testing and validation.

A mojo spec, or other suitably automated test, will be a requirement for this feature implementation.

Dependencies

None

Swift Global Cluster

Sun, 23 Feb 2020 00:00:00

Add support for Swift Global Cluster feature to Swift charms.

Problem Description

At the moment Swift charms do not support the Swift Global Cluster feature which is already available in the upstream:

https://docs.openstack.org/swift/latest/overview_global_cluster.html

As a result it is not possible to deploy Swift clusters in a multi-region mode using Swift charms.

Proposed Change

The proposed change can be split to the following items:

Add new config options to swift-storage charm to handle the following parameters:

“region”:

purpose: specifies region to request membership data type: int valid values: natural number
“account-server-port-rep”:

purpose: specifies listening port of the swift-account-replicator service data type: int valid values: 0 - 65535
“container-server-port-rep”:

purpose: specifies listening port of the swift-container-replicator service data type: int valid values: 0 - 65535
“object-server-port-rep”:

purpose: specifies listening port of the swift-object-replicator service data type: int valid values: 0 - 65535

Add new config options to swift-proxy charm to handle the following parameters:

“enable-multi-region”:

purpose: specifies whether the Swift Global Cluster feature should be used data type: boolean valid values: True / False
“read-affinity”:

purpose: specifies which backend servers to prefer on reads data type: string valid values: matches “^rd+z?(d+)?=d+(,s?rd+z?(d+)?=d+)*$” pattern
“write-affinity”:

purpose: specifies which backend servers to prefer on writes data type: string valid values: matches “^rd+(,s?rd+)*$” pattern
“write-affinity-node-count”:

purpose: specifies how many local objects will be tried before falling back to non-local ones data type: string valid values: matches “^d+(s*sreplicas)?$” pattern

Update template files to contain options required by the Swift Global Cluster feature.
Rename and change location of template configuration files responsible for Swift account, container and object services to match the schema required by the Swift Global Cluster feature.
Extend relation data exchanged between swift-storage and swift-proxy nodes with the region and other replication information.
Implement ‘rings-distributor’ - ‘rings-consumer’ relation based on a common interface (‘swift-global-cluster’) in swift-proxy charm to handle rings distribution across all proxy nodes participating in the multi-region deployment.
Update test files.
Update the documentation with information on how to setup Swift in multi-region mode.

Alternatives

At the moment there is no alternative to the Swift Global Cluster feature.

Implementation

Assignee(s)

Primary assignee:: tkurek

Gerrit Topic

Use Gerrit topic “swift-global-cluster” for all patches related to this spec.

git-review -t bug/1815879

Work Items

Update swift-proxy charm

Update ‘config.yaml’ file to add new options
Update template files with new options
Update ‘metadata.yaml’ file to add the new relation
Add hook files as symlinks to the ‘swift_hooks.py’ file
Update ‘swift_hooks.py’ file with hook definitions for the new relation
Update test files
Update ‘README.md’ file to describe the feature
Update other files (if needed) to implement the feature

Update swift-storage charm

Update ‘config.yaml’ file to add new options
Rename and change location of template configuration files responsible for Swift account, container and object services to match the schema required by the feature.
Update ‘metadata.yaml’ file to add extra bindings (‘cluster’ and ‘replication’)
Update ‘swift_storage_relation_joined’ hook definition with extended relation data
Update test files
Update ‘README.md’ file to describe the feature
Update other files (if needed) to implement the feature

Repositories

No new git repositories will be required to implement this feature.

Documentation

The README files of swift-proxy and swift-storage charms should be updated with instructions on how to setup Swift in multi-region mode.

Security

No security implications for this change.

Testing

Unit and functional tests of swift-proxy and swift-storage charms will need to be updated to support implementation of this feature.

Dependencies

No external dependencies.

Masakari

Sun, 23 Feb 2020 00:00:00

Masakari provides an API and ENGINE which receive messages from MONITORS (agents) on compute nodes to enable:

instance evacuation on host failure (masakari-hostmonitor) - requires shared storage
instance restart on certain instance errors (masakari-instancemonitor)
openstack related process monitoring and restarting (masakari-processmonitor)

This spec is for two charms which will install the masarki api and engine, and monitors respectively.

Problem Description

If an openstack hypervisor fails, there is no method of automatic recovery. Masakari provides a way of migrating instances from a failed host to a functional host, when used in conjunction with a cluster manager. There are similar issues with instance and process failures which masakari addresses.

Proposed Change

A solution using corosync and pacemaker on each hypervisor, along with masakari-hostmonitor has been validated. Unfortunately a cluster where each node runs the full cluster stack is only recommended up to around 16 nodes:

http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/#_overview

The scale issue with this initial solution can be mitigated by breaking compute nodes up into smaller clusters but this should be seen as a work around.

They suggested approach to scale to a greater number of nodes is by using pacemaker-remote on the hypervisors rather than the full cluster stack. Unfortunately masakari-hostmonitors does not currently work with pacemaker-remote due to the following bug:

https://bugs.launchpad.net/masakari-monitors/+bug/1728527

This work will involve the following new charms:

masakari (provides api, engine and python-masakariclient) The masakari charm will have identity-service, amqp and shared-db relations
masakari-monitors (optionally provides one or all of hostmonitor, instancemonitor, processmonitor) The masakari-monitors charm will be related to nova-compute via the juju-info relation and related to keystone via the identity-credentials relation. The monitors needs credentials for posting notifications to the masakari api service. When not using pacemaker_remote masakari-monitors will rely on the hacluster charm having been related to the nova-compute charm via juju-info.
pacemaker-remote. This charm simple installs the pacemaker remote service and requires a relation with fully fledged cluster.

This work will involve the following new relations:

hacluster <-> pacemaker-remote This relation will allow the pacemaker-remote charm to inform the hacluster charm of the location of remote nodes to be added to the cluster. It will also be used to expose the pacemaker-remotes stonith information and whether or not the pacemaker remote node should run resources. In the case of a masakri deployment the pacemaker-remote nodes should be set to not run resources.

The following charm updates:

hacluster charm. The hacluster charm need to be able to support adding pacemaker-remote nodes to the cluster and also confgiuring resources such that if requested no resources are run on the remote nodes.
hacluster charm. The hacluster charm need to be able to support creating maas stonith devices

Additional work:

Fix masakari-hostmonitors to work with pacemaker remote As mentioned masakari-hostmonitors does not work wih pacemaker-remote at the moment. A patch will need to be written to fix this and ideally landed upstream.
Write maas stonith driver. It is important that instances that are using shared storage are only running on one compute node at a time to avoid a split-brained cluster leading to two instances writting to the same storage simultaniously which would result in data corruption.

These charms will support TLS for API communications

Alternatives

Although not as feature rich as masakari much of the functionality can be achieved using pacemaker/corosync resource configuration.

Implementation

Assignee(s)

Primary assignee:: gnuoy
Secondary assignee:: admcleod

Gerrit Topic

Use Gerrit topic “masakari” or “masakari-monitors” as appropriate for all patches related to this spec.

git-review -t masakari
git-review -t masakari-monitors

Work Items

Masakari charm cleanup
Add domain id information to identity-credentials relation. The keystone charm and the identity-credentials interface will both need updating.
Masakari-monitors charm cleanup
Add pacemaker-remote charm
Add pacemaker-remote interface
Add functional and unit tests
Mojo specs for full stack functional testing.
Write patch to fix pacemaker-remote support in masakari-hostmonitors
Write Maas stonith plugin
Extend hacluster charm to support registering pacemaker_remote nodes.
Extend hacluster charm to support only running resources on a subset of available nodes.
Extend hacluster charm to support creating maas stonith devices.
Write deployment guide instructions.
Add new charms and interfaces to OpenStack gerrit.

Repositories

https://git.openstack.org/openstack/charm-masakari https://git.openstack.org/openstack/charm-masakari-monitors

Documentation

Both charms will contain README.md files with instructions

Security

We will have credentials for the cloud stored on the compute node. Dropping from the guest to the host in this case could allow a user to compromise the cloud by signally the masakari api about one or more false compute node failures. Keystone credentials which are used by the placement api are already stored on the compute node so this does not increase the attack surface but is worth mentioning for completeness.

We will need to enable a certificate relation in the nova compute host to facilitate the use of a vault charm to enable masakari ssl functionality.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of zaza and Mojo specification.

Dependencies

Requires cluster management such as corosync or pacemaker. At the very least, hacluster charm is required
Shared storage is required
Some administrative intervention will be required after a host failure.

Ceph PG Auto Tuning

Fri, 20 Dec 2019 00:00:00

In the Nautilus release, Ceph has introduced placement group (PG) autotuning. The charms should support this module allowing a Ceph cluster to grow or shrink naturally.

Problem Description

If an existing cluster is incorrectly sized at deploy time, it is very difficult for an operator to resolve the imbalance. PG Autotuning was designed to remove the black-magic of PG tuning.

Proposed Change

A new configuration option (pg-autotune) will be introduced to enable the new pg_autoscaler module. This config option will have no effect for Ceph releases before Nautilus and will have no default.

In a new Ceph deployment with a release greater than Nautilus, a null value for ‘pg-autotune’ will cause the autoscaler module to be enabled and configured on all pools, with the global default additionally configured as ON. In an upgrade to Nautilus, a null value will not cause the autoscaler module to be loaded, nor will pool metadata change.

For an existing deployment to enable the autoscaler, an administrator should change the config value to ‘True’, after which the charm will process the new option, as well as updating the existing pools as necessary.

The ceph-mon charms already handle sizing the default placement groups via the expected ratios so the required data is already present.

Alternatives

The current solution is a viable alternative, as long as the expectations about pool sizing is correct at deploy time. Other alternatives include manual operator intervention to resolve an incorrectly sized pool.

Implementation

Assignee(s)

Primary assignee:: chris.macnaughton

Gerrit Topic

Use Gerrit topic “pg-autotune” for all patches related to this spec.

git-review -t pg-autotune

Work Items

Enable the pg_autoscaler manager module if the Ceph version is Nautilus or higher
Disable the autoscaler globally, by default
Enable the autoscaler and configure pools when toggled via configuration

Repositories

No new repositories will need to be created.

The charms.ceph and charm-ceph-mon repositories will be the primary focus of this development.

Documentation

There will be a new configuration option added to toggle on the autoscaler for the Ceph cluster. This configuration option should additionally be called out in the README for the ceph-mon charm.

Security

There are no expected security risks associated with this change.

Testing

An additional test-case will be added enabling this config option to ensure that the functionality works as expected. Additionally, the functionality will be validated via inspection of Ceph’s suggestions for auto-tuning.

Dependencies

There are no new dependencies.

OVN

Fri, 20 Dec 2019 00:00:00

OVN provides open source network virtualization for Open vSwitch (OVS):

Logical switches
IPv4 and IPv6 logical routers
L2/L3/L4 ACLs (Security Groups)
Multiple tunnel overlays (Geneve, STT and VXLAN)
Logical L4 load-balancing
TOR-based L2 logical-physical gateways (VTEP)
Software-based L2/L3 logical-physical gateways

OVN integration components exists for OpenStack Neutron, Kubernetes and others.

Problem Description

There is currently no charms available that lets you easily deploy the services required to make use of OVN.

This will also benefit OPNFV’s JOID installer in providing another scenario in its deployment.

Proposed Change

Implement new reactive charms that adds support for deploying OVN and integration with OpenStack.

Alternatives

None

Implementation

The implementation consists of the following charms:

ovn-central
- Principal charm that deploys ovn-northd, the OVN central control daemon, and ovsdb-server, the Open vSwitch Database (OVSDB).
- The ovn-northd daemon is responsible for translating the high-level OVN configuration into logical configuration consumable by daemons such as ovn-controller.
- The ovn-northd process talks to OVN Northbound- and Southbound- databases.
- The ovsdb-server exposes endpoints over relations implemented by the ovsdb interface.
- The charm supports clustering of the OVSDB, you must have a odd number of units for this to work. Note that write performance decreases as you increase the number of units.
- Running multiple ovn-northd daemons is supported and they will operate in active/passive mode. The daemon uses a locking feature in the OVSDB to automatically choose a single active instance.
- The charm MUST configure TLS transport between OVSDB and its clients supporting both charm static configuration or with help from vault and the tls-certificates interface.
ovn-chassis
- Subordinate charm that deploys the OVN local controller and Open vSwitch Database and Switch.
- It connects up to the OVN Southbound database over the OVSDB protocol, down to the unit local Open vSwitch database over the OVSDB protocol and to the unit local Open vSwitch daemon via OpenFlow.
- Each hypervisor, software gateway or other units in need of OVN networking runs its own independent copy of the ovn-controller daemon.
- Modules to integrate with specific software, such as OpenStack nova-compute and Kubernetes worker, should all be added to this subordinate charm. Which module(s) to activate should be determined at runtime through presence of optional relations.
- A LXD profile will be required for this charm to enable container placement of OVN/OVS components.
ovn-dedicated-chassis
- Principal charm for deployments that require a dedicated software gateway.
- This charm will be built from the ovn-chassis subordinate codebase.
neutron-api-plugin-ovn
- Subordinate charm that deploys the networking-ovn component on neutron-api units and augments Neutron’s configuration for use with the OVN ML2 plugin.
octavia-ovn-provider
- Subordinate charm that deploys the networking-ovn component on octavia units and augments Octavia’s configuration for use with the OVN provider driver.

The implementation consists of the following interfaces:

ovsdb
- Interface that facilitates a provider charm to publish connection properties of a OVSDB.
- Interface that facilitates a requirer charm to consume a remote OVSDB.
octavia-provider
- Interface that facilitates a provider subordinate charm to publish information about a Octavia provider driver to and to trigger restart of the Octavia service.
- Interface that facilitates the Octavia charm to add provider information and react to restart requests from a subordinate provider charm.

The charms should be implemented in such a way so that the non-OpenStack specific components can be deployed with other software components such as the Charmed Distribution of Kubernetes.

Wherever possible and relevant, existing interfaces or charm libraries should be consumed or extended to work with the new charms.

Assignee(s)

Primary assignee:: fnordahl

Gerrit Topic

Use Gerrit topic “ovn-charm” for all patches related to this spec.

git-review -t ovn-charm

Work Items

See Implementation section

Repositories

charm-ovn
charm-ovn-controller
charm-ovn-dedicated-controller
charm-neutron-api-plugin-ovn
charm-octavia-ovn-provider
charm-interface-ovsdb
charm-interface-octavia-provider

Documentation

Each charm should contain a README with instructions on deploying the charm.

In addition to that a appendix will be added to the deployment guide with details about how to deploy OpenStack with OVN.

Security

Communication between OVSDB and its clients is authenticated and secured by the use of TLS and PKI. This must be implemented using secure best practices.

TLS configuration should be possible both statically through configuration and through automatic certificate provisioning with vault and the tls-certificates interface.

Testing

Code written or changed will be covered by unit tests; functional testing will be implemented using the Zaza framework.

Dependencies

There may be need for changes to the layout of the openvswitch source package to cater for running the ovsdb-server process without having the package attempting to configure the switching components. This will help with container placement of the ovsdb charm without requiring a LXD profile.
There may be need for changes to the layout of the netwokring-ovn source package to cater for Octavia provider driver enablement.

Policy overrides using the policy.d directory

Fri, 20 Dec 2019 00:00:00

The objective is to provide a mechanism where operators can modify the policies of OpenStack services in order to manage the permissions of the tenants and users of the system. It is NOT intended that the facility described in this specification is used to manage the permissions of the services themselves. This must remain wholly the domain of the charms, and the facility described here should not be (ab)used.

Problem Description

The preferred approach has always been to use charm options to enable a more controlled approach to modifying the policy of a service by providing templates that the options modify - i.e. a controlled approach that always results in consistent policy files.

However, over time, it has become apparent that there will always be an aspect of the policy of a service that an operator wants to tweak or change that the charm authors hadn’t considered, and the cycle time to introduce the option exceeds the expectations of the delay that an operator is willing to tolerate.

Therefore, it has been recognised that there are aspects of the configuration files that the charms control need to be more fluidly modifiable by operators. These tend to be orthogonal to the act of deployment and instead are to do with operating the cloud.

So, the objective is to provide to operators with a mechanism to override the policy defaults without (hopefully) breaking the system functioning. The policy defaults are either coded in the service via “policy-in-code” and/or via a default policy YAML file provided by the charm/service.

Proposed Change

This functionality will be supported on Ubuntu Xenial and later.
This functionality will be supported on OpenStack queens and later.
Juju version 2.0 or later is required for resource support.

The addition of policy overrides needs to be provided in the set of target charms (see later). In order to do this efficiently most of the work needs to be performed in charm-helpers with minimal work done in each charm.

A config option boolean use-policyd-override (default False) will control whether the resource file is used. If False then the resource file is ignored - i.e. it doesn’t matter whether it is present/damaged or not.

If config option use-policyd-override is True then the Juju status will be prefixed with "PO:" indicating that the policies are overridden by the attached resource file. This is intentionally kept short so that other information can also be seen on the status line. A info log is also generated to the Juju log to indicate that policies are overridden.

Juju resources will be used to attach a zipped file (using pkzip or equivalent) to the juju application using the resource name policy-override. The charm(s) associated with the application will verify the contents of the zipped file and update the contents of the policy directory (default /etc/<service-name>/policy.d with the files in the zipped resource file.

The resource name will be policyd-override.

Conceptually, the charm owns the /etc/<service-name>/policy.d/. Subordinate charms will not have the facility to override policies; this should be performed in the principal charm. The consequence of this ‘ownership’ is that the charm will remove files from the directory if they are not put there by the charm itself (for its own override purposes) or via this policy override system. i.e. ad hoc dropping of files into the directory will most likely be removed during the next upgrade-charm hook leading to unexpected results. Changes to the /etc/<service-name>/policy.d/ directory are logged to the charm’s debug log.

Templates in the Resource File

The policy override feature essentially comes in two parts; charm-helpers (and charms.openstack) and calling the support functions from the charm. One slightly contentious issue is around using templates to allow context variables (built from the relations’ data and config in a charm) to be baked into a policy override file prior to being dropped into the policy.d directory.

This is not the author’s preferred mechanism which is to instead use the existing template facility to render rules into the charm’s default policy file (in the same was as admin_required is done in the keystone policy.json file) and then refer to these in the stack override files. That is, templates are still controlled in the charm and not in the drop-in overrides.

However, it is fairly trivial to offer a string substitution capability in the charm-helpers functions, even if this is not used when called by the charm. The author believes that this should be the default condition, and only allow the template facility to be enabled when a policy override requirement cannot be satisfied in any other way.

Template and substitution will offered so that files in the resource file with either .j2, .tmpl or .tpl extensions are processed by a charm provided string substitution function which may be a Jinja template function using the context, or just a simple text substitution with some defined variables. This will be under the control of the charm. The substituted string returned by the charm-supplied function is then processed as per a static yaml file with all the following validation.

Validation

The charm will attempt some validation of the attached resource file:

That the file is a valid zip file (note that any directories are ignored. The zip file is flattened to remove directories.
Files that don’t have a .yaml or .yml extension are ignored. This allows documentation to be added to the zip file.
The remaining, flattened, identified yaml files are checked to have unique names. If not, then the validation fails.
Each file is (attempted to be) loaded using the Python yaml safe_load function. If this fails, then validation fails.
Each of the policy overrides will be checked against a blacklist that the charm maintains. This blacklist will be on a per-charm basis, and will initially blacklist changing admin_required and cloud_admin. This is to ensure that the charms themselves do not get locked out of the OpenStack system that they maintain.
If validation fails then the whole policy override fails and the charm proceeds as if use-policyd-override is False. The application/charm(s) active status will show "PO (broken): ...". Note that there will not be a hook error. The rationale is that policy overrides are an operator facility and the charm attempts to apply them, but should only indicate that they have not been applied so that the operator can change the policy overrides. A message will be logged to charm’s log explaining the validation failure.

If charm validation succeeds then the /etc/<service-name>/policy.d/ directory is cleared (apart from other charm intrinsic policy overrides) and then new files written to the directory. The payload service will be restarted.

OpenStack Service Support

The policy override feature appeared in oslo.policy for the ocata release, and was picked up by keystone and nova initially. In order for it to be supported, the OpenStack service needs to support the Enforcer class to be able to make use of the policy.d override directory (which is the default).

The following service projects (that have Canonical charms) examined and have the Enforcer class at the queens release:

keystone
neutron
nova
glance
cinder
aodh
designate
heat
horizon (openstack-dashboard)
manila
octavia
panko
trove
zaqar

The following service projects do not have Enforcer class support at present (stein):

swift

Services where it doesn’t apply:

gnocchi
ceilometer

This work will affect a number of charms. Bug#1741723 records the charms that (most likely) need the policy override facility. The principal ones that are the scope of this work are:

cinder
designate (reactive charm)
glance
keystone
neutron-api
nova-cloud-controller

The nature of the implementation should make it relatively easy to implement this on further charms. Designate is included to provide a proof-of-concept for reactive charms.

Alternatives

N/A

Implementation

Assignee(s)

Primary assignee:: ajkavanagh (irc: tinwood)

Gerrit Topic

Use Gerrit topic “policy-overrides” for all patches related to this spec.

git-review -t policy-overrides

Work Items

The changes will be implemented in:

charm-helpers (contrib.openstack)
charms.openstack
individual charms (keystone, glance, etc.)

The following work items have been identified with this work:

charm-helpers:
- helpers to read zip file, perform validation
- helpers to update policy.d directory with validated files.
- helper to call from install and upgrade hooks
- helper to call from config-changed hook handler to determine whether to enable or disable the policy overrides.
- helper to merge into update-status to determine whether policy is overridden or not.
charms.openstack:
- Mix-in that calls out to charm-helpers to perform validation and update/control of policy.d directory.
- Method to integrate into install and upgrade hooks to call relevant functions in the mix-in.
- Method to integrate with config-changed hook to determine whether to enable or disable the policy overrides (probably call out to charm-helpers).
keystone charm (in the first instance):
- Link into the helper functions in charm-helpers.
- Functional test to verify that policy can be overridden and that files appear in the policy.d directory.
designate charm (reactive example)
- Add in mix-in to designate main class, and any required helper calls (it’s not clear yet how much additional support will be needed).
- Functional test to verify that policies can be overridden with a file.

Repositories

The following repositories will be affected:

And the testing frameworks:

Documentation

The following documents will require an update:

charm-guide
deployment-guide
README in charms affected by change

Security

Although a file is being uploaded to the controller (and thus as a resource to the charms), the Python zip module is used to examine the contents, and potential yaml files are read with the YAML module’s safe_load is used to verify that the contents are yaml. This is then written to relevant files in the /etc/<service-name>/policy.d/ directory. Thus the files in the zip file aren’t copied to the file directory. It is thus thought that, as no 3rd party module are used, and no files are directly copied, that there should not be a security issue with this approach.

Testing

Unit tests will be added to charm-helpers and charms.openstack
Functional tests will be added to those charms that support the zaza testing framework. This will verify that the overrides are put into the appropriate directory and that the service is restarted. A service action (e.g. list users) will be verified as working, then disabled, and then working again. A framework will be put into zaza-openstack-tests to support this.

Actual policy overrides are not checked. This is the domain of the payload (service) and outside of the scope of testing for this feature which is simply to drop appropriately formatted yaml files into the policy.d directory.

Dependencies

There are no dependencies

The Title of Your Specification

Fri, 20 Dec 2019 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

OpenStack with OVN

Fri, 27 Sep 2019 00:00:00

Openstack can be deployed with a number of SDN solutions (e.g. ODL). OVN provides virtual-networking for Open vSwitch (OVS). OVN has a lot of desirable features and is designed to be integrated into Openstack, among others.

Since there is already a networking-ovn project under openstack, it is the obvious next step to implement a Juju charm that provides this service.

Problem Description

Currently, Juju charms have support for deploying openstack, either with it’s default SDN solution (Neutron), or with others such as ODL. This project will expand the deployment scenarios under Juju for openstack by including OVN in the list of available SDN solutions.

This will also benefit OPNFV’s JOID installer in providing another scenario in its deployment.

Proposed Change

Charms implementing neutron-api, ovn-controller and neutron-ovn will need to be implemented. These will be written using the new reactive framework of Juju.

Charm : neutron-ovn

This charm will be deployed alongside nova-compute deployments. This will be a subordinate charm to nova-compute, that installs and runs openvswitch and the ovn-controller.

Charm : ovn-controller

This charm will deploy ovn itself. It will start the OVN services (ovsdb-server, ovn-northd). Since there can only be a single instance of ovsdb-server and ovn-northd in a deployment, we can also implement passive HA, but this can be included in further revisions of this charm.

Charm : neutron-api-ovn

This charm will provide the api only integration of neutron to OVN. This charm will need to be subordinate to the existing neutron-api charm. The main task of this charm is to setup the “neutron.conf” and “ml2_ini.conf” config files with the right parameters for OVN. The principal charm, neutron-api, handles the install and restart for neutron-server.

Refer for more information : https://docs.openstack.org/networking-ovn/latest/install/manual.html

Alternatives

N/A

Implementation

Assignee(s)

Aakash KT

Gerrit Topic

Use Gerrit topic “charm-os-ovn” for all patches related to this spec.

git-review -t charm-os-ovn

Work Items

Implement neutron-api-ovn charm
Implement ovn-controller charm
Implement neutron-ovn charm
Integration testing
Create a bundle to deploy OpenStack OVN
Create documentation for above three charms

Repositories

Yes, three new repositories will need to be created : * charm-neutron-api-ovn * charm-ovn-controller * charm-neutron-ovn

Documentation

This will require creation of new documentation for the covered scenario of openstack + ovn in Juju. A README file for the bundle needs to be written. Add documentation in charm-deployment guide to detail how to deploy OVN with OpenStack.

Security

Communications to and from these charms should be made secure. For eg. communication between ovn-central and ovn-edges to be made secure using self-signed certs.

Testing

For testing at Juju level, we can use the new “juju-matrix” tool. For testing functionality at OpenStack level, Mojo should be used. This will help validate the deployment.

Dependencies

This charm will support OpenStack Queens as its baseline.

OpenStack Endpoint Load Balancer

Wed, 25 Sep 2019 00:00:00

To enable Openstack services for a single cloud to be installed in a highly available configuration without requiring that each unit of a service is in the same broadcast domain.

Problem Description

As a cloud administrator I would like to simplify my deployment so that I don’t have to manage a corosync and pacemaker per OpenStack API service.
As a cloud architect I am designing a new cloud where all services will be in a single broadcast domain. I see no need to use the new central loadbalancer and would like to continue to have each service manage its own VIP.
As a cloud architect I would like to spread my control plane across N racks for redundancy. Each rack is in its own broadcast domain. I do not want the users of the cloud to require knowledge of this topology. I want the endpoints registered in Keystone to work regardless of a rack level failure. I am using network spaces to segregate traffic in my cloud and the OpenStack loadbalancer has access to all spaces so I only require one set of loadbalancers for the deployment.
As a cloud architect I would like to spread my control plane across N racks for redundancy. Each rack is in its own broadcast domain. I do not want the users of the cloud to require knowledge of this topology. I want the endpoints registered in Keystone to work regardless of a rack level failure. I am using network spaces to segregate traffic in my cloud. I want the segregation to extend to the load balancers and so will be requiring a set of load balancers per network space.
As a cloud architect I am designing a new internal cloud and have no interest in IPv6, I wish to deploy a pure IPv4 solution.
As a cloud architect I am designing a new cloud. I appreciate that it has been 18 years since the IETF brought us IPv6 and feel it maybe time to enable IPv6 within the cloud. I am happy to have some IPv4 where needed and am looking to deploy a dual stack IPv4 and IPv6.
As a cloud architect I am designing a new cloud. I appreciate that it has been 18 years since the IETF brought us IPv6 and wish to never see an IPv4 address again. I am looking to deploy a pure IPv6 cloud.
As a cloud architect I wish to use DNS HA in conjunction with the OpenStack loadbalancer so that loadbalancer units can be spread across different subnets within each network space.
As a cloud administrator I would like to have the OpenStack load balancers look after HA and so will be deploying in an Active/Passive deployment. I will need to use a VIP for the loadbalancer in this configuration.
As a cloud architect I have an existing hardware loadbalancers I wish to use. I do not want to have to update it with the location of each API service backend. Instead I would like to have the OpenStack load balancers in an Active/Active configuration and have the hardware loadbalancers manager traffic between haproxy instance in the OpenStack loadbalancer service. I do not need to use a VIP for the loadbalancer in this configuration. My hardware loadbalancers utilise vip(s) which will need to be registered as the endpoints for services in Keystone.
As a cloud administrator haproxy statistics are fascinating to me and I want the statistics from all haproxy instances to be aggregated.
As a cloud administrator I would like haproxy to be able to perform health checks on the backends which assert the health of a service more conclusively than simple open port checking.
As a cloud administrator I want to be able to configure max connections and timeouts as my cloud evolves.
As a charm author of a service which is behind the OpenStack load balancer I would like the ability to tell the loadbalancer to drain connection to a specific unit and take it out of service. This will allow the unit to go into maintenance mode.

Proposed Change

New interface: openstack-api-endpoints

Example - neutron-api (single API endpoint per unit):

endpoints:
  - service-type: network
    frontend-port: 9696
    backend-port: 9689
    backend-ip: 10.10.10.1
    check-type: http

Example - nova-cloud-controller (multiple API endpoints per unit):

endpoints:
  - service-type: nova
    frontend-port: 8774
    backend-port: 8764
    backend-ip: 10.10.10.2
    check-type: http
  - service-type: nova-placement
    frontend-port: 8778
    backend-port: 8768
    backend-ip: 10.10.10.2
    check-type: http

Having processed the requests from all backend units, the loadbalancer now needs to tell the backend application the external IP being used to listen for connections for each endpoint service type:

endpoints:
  - service-type: nova
    frontend-ip: 98.34.12.1
    frontend-port: 8774
  - service-type: nova-placement
    frontend-ip: 98.34.12.1
    frontend-port: 8778

The backend service now updates the endpoints in the Keystone registry to point at the IPs passed back by the loadbalancer.

provides:
  public-backend:
    interface: openstack-api-endpoints
  admin-backend:
    interface: openstack-api-endpoints
  internal-backend:
    interface: openstack-api-endpoints

Updates to keystone endpoint calculation code

Currently the following competing options are used to calculate which EP should be registered in Keystone:

os-*-network set do resolve_address old method
dnsha use dnsha
os-*-hostname set use hostname
juju network space binding via extra-bindings
prefer ipv6 via configuration option
presence of {public,internal,admin}-backend relations to openstack loadbalancers

OpenStack Loadbalancer charm

New charm - OpenStack Loadbalancer - with corresponding tests & QA CI/setup.

Alternatives

Extend existing HAProxy charm.
Use DNS HA.

Implementation

Assignee(s)

Primary assignee:: unknown

Gerrit Topic

Use Gerrit topic “osbalancer” for all patches related to this spec.

git-review -t osbalancer

Work Items

Provide OpenStack Loadbalancer Charm

Write draft interface for LB <-> Backend
Write unit tests for Keystone endpoint registration code
Write Keystone endpoint registration code

Mojo specification deploying and testing Mistral

Write Mojo spec for deploying LB in an HA configuration

Repositories

A new git repository will be required for the Mistral charm:

https://git.openstack.org/openstack/charm-openstack-loadbalancer

Documentation

The OpenStack Loadbalancer charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

None

Renaming the lxd charm to nova-lxd

Tue, 27 Aug 2019 00:00:00

The initial ‘push’ to rename the charm to nova-lxd is:

The name nova-lxd better reflects that the charm is used with nova. It is confusing for users that the (currently named) lxd charm must be used with the nova-compute charm, and can’t be used in a different environment.
It releases the name lxd for use as a general purpose charm for LXD, that could be created to allow clusters of LXD machines to configured with Juju and MaaS.

Problem Description

The name of the charm as lxd is causing some confusion in the community as potential users are unaware that it is an OpenStack specific charm for nova as a subordinate. Therefore, changing the name of the charm to nova-lxd will help mitigate this confusion.

Proposed Change

The lxd charm will be copied to nova-lxd with all of its existing functionality.
The existing lxd charm will be converted to a no-op charm with a blocked status message indicating that the user should use the nova-lxd charm instead. The lxd charm will then be removed entirely after another cycle.
Making the existing lxd charm a no-op also has the advantage that existing installations will be unable to upgrade to the newer ‘no-op’ lxd charm as the relations will not match, thus forcing the operator to investigate as well as not breaking existing installations.

The charm name will be switched with the juju upgrade-charm lxd --switch=nova-lxd method of changing a charm and this will be tested during upgrades.

Alternatives

No alternatives suggested.

Implementation

Assignee(s)

Primary assignee: ajkavanagh

Gerrit Topic

Use Gerrit topic “nova-lxd-charm” for all patches related to this spec.

git-review -t nova-lxd-charm

Work Items

There are the following, logically separate, pieces of work in this specification:

Turning the existing lxd charm into a ‘no-op’ charm with a blocked status message. For the avoidance of doubt, Juju will refuse to upgrade an existing lxd charm to this charm. The purpose of the blocked message is for new deployments where the deployment bundle should use nova-lxd, but instead refers to the old name. In this case, the charm will not do anything as a subordinate except to show the blocked message.
Creating the new nova-lxd charm by copying the existing lxd charm repository over.
Create project for nova-lxd charm on OpenStack on gerrit
Create launchpad project for nova-lxd charm

Repositories

New git repository: openstack/charm-nova-lxd

This will be created as part of creating the charm-nova-lxd project on openstack.

Documentation

Add notes the Charm Deployment Guide about the new charm.
Add notes to the upgrades section to discuss how to upgrade from lxd to nova-lxd, what issues may occur, and how to recover the situation.

Security

None noted.

Testing

As the charm is just being renamed, there should be no issues with unit and functional tests. They will not change as part of the specification.

However, the various bundles that use the lxd charm will need to be updated to refer to a nova-lxd charm for the 19.04 release. These are contained in:

github: openstack-charmers/openstack-bundles - openstack-lxd
github: openstack-charmers/openstack-charm-testing - README-lxd.md - README-multihypervisor.md - bundles/lxd/* - bundles/multi-hypervisor/*

Testing on a MaaS will be necessary for the modified openstack-lxd bundles.

Dependencies

None noted.

Ceph RBD Mirror Charm

Fri, 03 May 2019 00:00:00

Problem Description

RBD image mirroring can be used to provide a solution for Ceph cluster disaster recovery. Ceph has a daemon called rbd-mirror which can be placed on a primary and a backup cluster and provide asynchronous replication of RBD images in a given pool.

rbd-mirror can work in two modes:

pool (all images in a given pool are synchronized);
image (per-image synchronization).

The scenario targeted by this spec involves an operator when it comes to performing promote/demote actions and the DR procedure is operator-driven as opposed to a full automatic failover in the event of a outage at the primary site.

Note

Promotion/demotion of pools will be operator driven

Note

RADOS objects are not mirrored so for mirroring radosgw objects which use RADOS objects or gnocchi metrics stored in RADOS objects different backup mechanisms are required - this spec covers RBD images only.

Note

RBD mirroring relies on the use of the exclusive-lock and journaling features of RBD; these are only supported in the userspace integration libraries as used by libvirt and qemu for native KVM virtualization. This requirement excludes the use of this feature with LXD based clouds which disable the majority of RBD features for compatibility with the Linux kernel RBD driver.

Note

The initial RBD mirror charm will only support mirroring of whole pools.

Proposed Change

High Level Design

As rbd-mirror is a separate package and the service itself acts as an RBD client it makes sense to implement the target functionality in a separate principle charm (ceph-rbd-mirror). The charm will accept parameters on which pools to replicate and be able to be related to multiple ceph-mon applications in separate clusters.

The charm will relate to a local ceph cluster and a remote ceph cluster typically using a cross model relation.

A new interface type (‘rbd-mirror’) will be created to support this integration; this will be provided by the ceph-mon charm, and consumed by the new ceph-rbd-mirror charm for both local and remote cluster connections.

Each rbd-mirror daemon requires a key for connectivity to the local cluster (named uniquely for the daemon) and a key for connectivity to the remote cluster (named globally for all rbd-mirror daemons). Multiple ceph configurations will also be maintained on the ceph-rbd-mirror units - ‘ceph’ to reference the local cluster and ‘remote’ to reference the remote cluster. Configuration files and keys will be prefixed inline with this naming - for example:

$ ls /etc/ceph
    ceph.conf
    ceph.client.rbd-mirror.<hostname>.keyring
    remote.conf
    remote.client.rbd-mirror.keyring

In order to support resilience and scale-out of rbd mirroring, multiple units of the charm may be deployed; as a result this feature will only be supported with Ceph Luminous or later (which support multiple instances of the rbd-mirror service).

Deployment and Scalability Considerations

From the deployment perspective the charm units should have high-bandwidth and low-latency L3 connectivity to access and replication networks of both clusters to be able to keep up with the changes to Ceph pools it tries to replicate. At minimum, static routes will need to be configured on the node running rbd-mirror daemon but that is outside of the scope of this spec.

Multiple units of the ceph-rbd-mirror charm may be used to scale out replication traffic.

Alternatives

No alternative solutions have been considered.

Dependencies

This feature relies on use of a Juju version which supports cross model relations.

Implementation

Assignee(s)

Primary assignee:

<tbd>

Gerrit Topic

Use Gerrit topic “rbd-mirror” for all patches related to this spec.

git-review -t rbd-mirror

Work Items

Implement a new reactive charm called ceph-rbd-mirror.
Implement the following relation: * rbd-mirror - ceph-mon (local and cross model).
Add “cluster” endpoint to extra-bindings in metadata.yaml to allow binding the “cluster” endpoint to a ceph replication space.
ceph-mon relations should retrieve cluster details and cephx keys via the broker protocol implemented in Ceph charms (code reuse).
Add config options to specify pool names for replication.
Automate creation of pools on a backup cluster if they are not present.
Add actions to promote and demote pools.
Enable RBD journaling feature as documented in rbd-mirror docs.
Write unit tests.
Write functional tests via zaza framework.

Repositories

A new git repository will be required for the ceph-rbd-mirror charm:

https://git.openstack.org/openstack/charm-ceph-rbd-mirror

Documentation

The ceph-rbd-mirror charm should contain a README with instructions on deploying the charm and on limitations related to scalability and networking.

Security

Users created for replication must not have admin privileges - they only need to be able to write to the pools they require on the target cluster. This is supported through the existing group based permissions system in the ceph-mon broker using the ‘rbd’ profile for mon and osd permissions.

Testing

Code written or changed will be covered by unit tests; functional testing will be done using Zaza.

Keystone Federation

Fri, 03 May 2019 00:00:00

Keystone can be configured to integrate with a number of different identity providers in a number of different configurations. This spec attempts to discuss how to implement pluggable backends that utilise Keystone federations.

Problem Description

When deploying the OpenStack charms to an enterprise customer they are likely to want to integrate OpenStack authentication with an existing identity provider like AD, LDAP, Kerberos etc. There are two main ways that Keystone appears to achieve this integration: backends and federation. Although this spec is concerned with federation it is useful to go over backends to in an attempt to distinguish the two.

The following are not covered here:

Integration with Kerberos
Horizon SSO
Federated LDAP via SSSD and mod_lookup_identity
Keystone acting as an IdP in a federated environment.

Backends

When keystone uses a backend it is Keystone itself which knows how to manage that backend, how to talk to it and how to deal with operations on it. This limits the number of backends that keystone can support as each new backend needs new logic in Keystone itself. This approach also has negative security implications. Keystone may need an account with the backend (an LDAP username and password) to perform lookups, these account details will be in clear text in the keystone.conf. In addition, all users passwords with flow through keystone.

The keystone project highlights SQL and LDAP (inc AD) as their supported backends. The status of support for these is as follows:

SQL: Currently supported by the keystone charm.
LDAP: Currently supported by the keystone and keystone-ldap subordinate.

These backends are supported with the Keystone v2 API if they are used exclusively. To support multiple backends then Keystone v3 API needs to be used and each backend is associated with a particular Keystone domain. This allows for service users to be in SQL but users to be in ldap for example.

Adding a new backend is achieved by writing a keystone subordinate charm and relating it to keystone via the keystone-domain-backend interface.

Enabling a backend tends to be achieved via the keystone.conf

Federation

With federation Keystone trusts a remote Identity provider. Keystone communicates with that provider using a protocol like SAML or OpenID connect. Keystone relies on a local Apache to manage communication and Apache passes back to keystone environment variables like REMOTE_USER. Keystone is abstracted from the implementation details of talking to the identity provider and never sees the users password. When using Federation, LDAP may still be the ultimate backend but it is fronted by something providing SAML/OpenID connectivity like AD federation service or Shibboleth.

Each Identity provider must be associated with a different domain within keystone. The keystone v3 API is needed to support federation.

Compatible Identity Providers: (https://docs.openstack.org/ocata/config-reference/identity/federated-identity.html#supporting-keystone-as-a-sp ):

OpenID
SAML

Proposed Change

Both Keystone backends and federated backends may need to add config to the keystone.conf and/or the Apache WSGI vhost. As such, it makes sense for both types to share the existing interface particularly as the existing interface is called keystone-domain-backend which does not differentiate between the two.

This spec covers changes add support for federation using either SAML or OpenID, to the keystone charm. This will involve extending the keystone-domain-backend interface to support passing configuration snippets to the Apache vhosts and creating subordinate charms which implement OpenID and SAML.

Alternatives

Add support for federation via OpenID and SAML directly to the keystone charm.
Create a new interface for federation via OpenID and SAML

Implementation

Assignee(s)

Primary assignee:: None

Gerrit Topic

Use Gerrit topic “keystone_federation” for all patches related to this spec.

git-review -t <keystone_federation>

Work Items

Create deployment scripts to create test env for OpenID or SAML integration

Extend keystone-domain-backend

The keystone-domain-backend interface will need to provide the following:

Modules for Apache to enable
Configuration for principle to insert into Apache keystone wsgi vhosts
Subordinate triggered restart of Apache
Auth method(s) to be added to keystone’s [auth] methods list
Configuration for principle to insert into keystone.conf

Configure Keystone to consume new interface

Keystone charm will need to be updated to respond to events outlined in the interface description above

New keystone-openid and keystone-saml subordinates

The new subordinates will need to expose all the configuration options needed for connecting to the identity provider. It will then need to use the interface to pass any required config for Apache or Keystone up to the keystone principle.

Repositories

New projects for the interface and new subordinates will be needed.

Documentation

This will require documentation in the READMEs of both the subordinates and the keystone charm. A blog walking through the deployment and integration would be very useful.

Security

Although a Keystone back-end will determine who has access to the entire OpenStack deployment, this specific charm will only change Keystone and Apache parameters, avoiding default values and leave the configuration to the user should be enough.

Testing

The code must be covered by unit tests. Ideally amulet tests would be extended to cover this new functionality but deploying a functional openid server for keystone to use may not be practical. It must be covered by a Mojo spec though.

Dependencies

None

Keystone Charm Goal State Support

Fri, 03 May 2019 00:00:00

Charms install and configure a piece of software on a unit in a model. How the software should be set up is determined by a set of information sources.

Unit operating environment
Application level configuration options
- Set in the model and available to the charm at a early stage
Subordinate relations to charms installed on the same unit
Peer/Cluster relation to other units of same application in same model
Relations to units of other applications in the same model
Cross-model relations

In its simplest form a charm’s sole goal is to as quickly as possible install a piece of software, write its configuration and fire it up enabling the end user to make use of the software.

Problem Description

As the deployment progresses the charm learns more about its broader environment in pace with its relation-joined and relation-changed hooks being fired. Some of these relations may fundamentally change how the software should be set up or behave. Which dependencies and requirements a piece of software needs met before it can be brought to a operational state also relies heavily on which relations the charm has.

At present you may find that by the time a deployment is complete a charm has reconfigured and restarted it’s services several times, each time it may also have notified it’s peers and relations to do the same. These activities prolong the deployment time, and causes race conditions and transient errors on depending services.

Proposed Change

To address some of these issues Juju has introduced a goal-state command to the hook environment. This can be used by a charm to get a upfront view of what Juju is going to deploy.

Scope

Introduce goal-state support in Keystone charm
When available, make use of the early view of deployment to streamline charm operation

Limitations

The feature has dependency on support from Juju. At the time of this writing this feature is only supported by Juju 2.4.
- When charm is deployed with versions of Juju without goal state support it falls back to inferring information about the deployment from configuration options and as relations appear.
Goal state gives us a peek into a future, the reality of that future can change beneath us if modifications are applied to the model before deployment completes.
- In the event of goal state changing mid-deployment, the benefits of the goal state optimizations may be minimized or in worst case make deploy time longer.

Alternatives

Some of these issues can be alleviated by introducing configuration options that give the charm a hint of expected scale. But this approach is static and contradicts the dynamic use and ease of scale out properties of Juju.

Implementation

Assignee(s)

Primary assignee:: fnordahl

Gerrit Topic

Use Gerrit topic “goal-state” for all patches related to this spec.

git-review -t goal-state

Work Items

Write test charm and associated test specs to exercise and monitor how identity-* relations are exercised by the keystone charm in complex deployment scenarios.
Delay initial run of identity-admin, identity-service and identity-credentials relations until:
- all keystone units have joined and completed the charms peer/cluster relation
- shared-db relation is ready and settled
- If HA, make sure it is really up and ready
Limit the number of times identity-admin, identity-service and identity-credentials relations are fired after initial run
Limit the number of times credentials are added/updated to the database

Repositories

No new repositories required.

Documentation

This is mainly a under the hood change and documentation will be added as part of the code in docstrings.

Security

No changes affecting security.

Testing

Unit tests will be written alongside any code changes.

Functional tests will be expanded or written as necessary.

A test charm and associated test specs will be written to exercise and monitor how identity-* relations are exercised by the keystone charm in complex deployment scenarios.

Comparison of deployment times prior to and after the proposed change will be documented as part of the review process.

Dependencies

Juju 2.4 is required for goal-state support, charm will fall back to current behaviour when this is not available.

RadosGW Charm Multi-site Replication

Fri, 03 May 2019 00:00:00

Problem Description

RadosGW multi-site configuration can be set up to provide object sync for disaster recovery and other purposes such as using the same object data stored in a Ceph cluster local to a cloud region. A typical setup would look like this:

One zone per Zone Group (1 cluster per “region”);
Multiple Zone Groups (“regions”);
One Realm;
Mode of operation: active-active or active-passive.

Note

Ceph does support active-passive configurations, but to simplify deployment choice the charms will only support active-active.

There could also be more complex configurations with multiple zones (clusters) per zone group.

In order to set this up, independent radosgw application deployments in different Juju models have to be aware of each other and set up the necessary configuration:

Realm name for radosgw;
Master zone group and master zone configuration;
a system user for authentication between daemons;
Access key and secret key setup for master zone authentication;
A period needs to be updated after configuration changes to change an epoch.

Note

Migration of an existing single site ceph-radosgw deployment to a multi-zone deployment will not be supported by the charms.

Proposed Change

To be able to configure multi-site radosgw deployments it is necessary to modify the radosgw charm to support cross-model relations between multiple radosgw applications. This relation will be used to exchange endpoint and authentication information between the RADOS gateway deployment for configuration of replication.

The charms will target a fix topology with a single realm and zone group and two zones. Its assumed that zones will be supported by separate Ceph clusters but this is not a hard requirement (but is recommended).

Actions will be provided to promote and demote a RADOS gateway cluster to and from master status. No automatic failover will be provided and these operations must be performed by an operator in the event of site failover/failback.

Alternatives

As this is a RADOS gateway specific feature, no alternatives have been considered.

Implementation

Assignee(s)

Primary assignee:

Gerrit Topic

Use Gerrit topic “radosgw-multi-site” for all patches related to this spec.

git-review -t radosgw-multi-site

Work Items

Implement support for new (cross-model) relation ‘rgw-peer’ between radosgw applications associated with different Ceph clusters.
Add support for additional configuration keys to set up realm, zonegroup and zone for each ceph-radosgw deployment.
Implement functionality to set up a master zone and add secondary zones to it.
Write unit tests for newly added features.
Write functional tests that include the deployment of multiple clusters and verification of object synchronization.

Repositories

No new git repositories will be created.

Documentation

The radosgw charm README should contain instructions on deploying the charm with new functionality enabled.

Security

TLS termination can be enabled on any side and needs to be supported without manual steps of synchronizing CA certificates between sites. SSL CA certs will be shared between RADOS peers using the new rgw-peer relation.

Testing

Code written or changed will be covered by unit tests; functional testing will be done using Zaza.

Dependencies

The ceph-radosgw charm currently uses the old-style radosgw systemd unit and a global cephx key for access to the underlying Ceph cluster.

The charm should be migrated to use the new ceph-radosgw systemd units and switch to use of cephx keys which are specific to individual radosgw units.

The Title of Your Specification

Thu, 25 Apr 2019 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

The Title of Your Specification

Thu, 25 Apr 2019 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

The Title of Your Specification

Thu, 25 Apr 2019 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

The Title of Your Specification

Thu, 25 Apr 2019 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

The Title of Your Specification

Thu, 25 Apr 2019 00:00:00

Introduction paragraph – why are we doing anything?

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

What versions of the operating system are affected or required?

What versions of OpenStack are affected or required?

What version of Juju is required?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>

Can optionally list additional ids if they intend on doing substantial implementation work on this blueprint.

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t <topic_name>

Work Items

Repositories

Will any new git repositories need to be created?

Identify all new charm repos, interface repos, layer repos, whether new or existing, which will be affected or involved directly in the work which is defined by this spec.

Documentation

Will this require a documentation change? If so, which documents? Will it impact developer workflow? Will additional communication need to be made?

Identify the surface area of doc updates explicitly, ie. charm-guide, deployment-guide, charm README, wiki page links.

Security

Does this introduce any additional security risks, or are there security-related considerations which should be discussed?

Testing

What tests will be available or need to be constructed in order to validate this? Unit/functional tests, development environments/servers, etc.

Are there any special hardware requirements to test this?

Dependencies

Include specific references to specs and/or stories, or in other projects, that this one either depends on or is related to.
Does this feature require any new library or program dependencies not already in use?
What are the plans to package and distribute the payload and/or dependencies?

Aodh Charm

Sun, 24 Mar 2019 00:00:00

Aodh (alarms and notifications based on metrics) was split from Ceilometer during the Liberty cycle and equivalent function was removed from Ceilometer during the Mitaka cycle.

We need to charm Aodh to allow OpenStack Operators to deploy and use alarming and notification services as part of an OpenStack cloud.

Problem Description

Aodh provides a method generating alarms and notifications based on metrics.

Proposed Change

The new Aodh charm should include, as a minimum, the following features:

Deployable in a highly available configuration
Allow clients and services to interact using SSL encryption
Charm progress displayed via workload status

Alternatives

Manually review metrics and trigger alarms.

Implementation

Assignee(s)

Primary assignee:: jamespage

Gerrit Topic

Use Gerrit topic “aodh” for all patches related to this spec.

git-review -t aodh

Work Items

Provide Aodh charm

Create skeleton charm layer based on OpenStack base layer and available interface layers to deploy Aodh.
Add support for upgrading Aodh
Add config option and accompanying support for upgrades via action-managed-upgrade.
Add support for deploying Aodh in a highly available configuration
Add support for the Aodh to display workload status
Add support SSL endpoints
Charm should have unit and functional tests.

Mojo specification deploying and testing Aodh

Write Mojo spec for deploying Mojo in an HA configuration and testing automatic and manual creation of DNS records.

Repositories

A new git repository will be required for the Aodh charm:

https://git.openstack.org/openstack/charm-aodh

Documentation

The Aodh charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

Provide mongodb interface
Provide hacluster interface layer
Provide nrpe-external-master interface layer
Provide OpenStack base layer with all common hook code that is not already covered by an interface layer.
Provide OpenStack base layer with support for HA deployments
Provide OpenStack base layer with support for SSL communication
Provide OpenStack base layer with support for workload status

Barbican Charm

Sun, 24 Mar 2019 00:00:00

Provide a charm for deploying Barbican with support for associated HSM modules/devices.

Problem Description

OpenStack services and users often need a repository to store sensitive information like passwords, cryptographic keys etc. The Barbican service provides an interface on top of an HSM for doing that.

Proposed Change

One new charm - Barbican; Charm needs to take into account potential use of backed hardware security modules (HSM) - this might be nicely done using the cinder-backend approach as a subordinate charm to avoid polluting the main charm with details of every HSM possible.

The new charm, as a minimum, should include the following features:

Deployable in a highly available configuration
Allow clients and services to interact using SSL encryption
Charm progress displayed via workload status

Alternatives

Secrets stored via other means outside of OpenStack.

Implementation

Assignee(s)

Primary assignee:: ajkavanagh gnuoy

Gerrit Topic

Use Gerrit topic “barbican” for all patches related to this spec.

git-review -t barbican

Work Items

Provide base and interface layers required for OpenStack charms

Provide rabbitmq interface layer
Provide mysql-shared interface layer
Provide pgsql interface layer
Provide keystone interface layer
Provide hacluster interface layer
Provide nrpe-external-master interface layer
Provide OpenStack base layer with all common hook code that is not already covered by an interface layer.
Provide OpenStack base layer with support for HA deployments
Provide OpenStack base layer with support for SSL communication
Provide OpenStack base layer with support for workload status

Provide Barbican charm

Create skeleton charm layer based on OpenStack base layer and available interface layers to deploy Barbican.
Add support for upgrading Barbican
Add config option and accompanying support to enable barbicans use of configurable storage backends: ie. HSM (hardware security module) NOTE: configuration without HSM is not secure and is for testing purposes only.
Add config option and accompanying support for upgrades via action-managed-upgrade.
Add support for deploying Barbican in a highly available configuration
Add support for the Barbican to display workload status
Add support SSL endpoints
Charm should have unit and functional tests.

Mojo specification deploying and testing Barbican

Write Mojo spec for deploying Mojo in an HA configuration and testing storage and retrieval of secrets.

Repositories

A new git repository will be required for the Barbican charm:

https://git.openstack.org/openstack/charm-barbican

Documentation

The Barbican charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

Given the purpose of Barbican is to store and manage secrets a review of the charm by the security team may be appropriate.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

None

Designate Charm

Sun, 24 Mar 2019 00:00:00

Charm OpenStack DNS (Designate).

Problem Description

Designate (DNSaaS) provides a method for managing DNS records for addressing OpenStack guests and floating IPs.

Proposed Change

Two new charms - designate and designate-bind - designate provides the API/RPC services, and bind provides a flexible scale out service for managing BIND DNS servers; the interfaces between the two charms should be sufficiently flexible to allow different DNS backends to be plugged in at a later date if require (i.e. a PowerDNS charm instead of BIND).

The new Designate charm should include, as a minimum, the following features:

Deployable in a highly available configuration
Allow clients and services to interact using SSL encryption
Charm progress displayed via workload status

Alternatives

DNS records could be managed manually in an existing DNS server.

Implementation

Assignee(s)

Primary assignee:: gnuoy

Gerrit Topic

Use Gerrit topic “designate” for all patches related to this spec.

git-review -t designate

Work Items

Provide Designate charm

Create skeleton charm layer based on OpenStack base layer and available interface layers to deploy Designate.
Add support for upgrading Designate
Add config option and accompanying support for upgrades via action-managed-upgrade.
Add support for deploying Designate in a highly available configuration
Add support for the Designate to display workload status
Add support SSL endpoints
Charm should have unit and functional tests.

Provide Designate Bind charm

Create bind charm designed to integrate with designate.
Ensure charm meets basic non-functional requirements, such as HA and workload status

Extend Designate charm

Add support for Designate integration with Neutron to extend the automatically created record information.

Mojo specification deploying and testing Designate

Write Mojo spec for deploying Mojo in an HA configuration and testing automatic and manual creation of DNS records.

Repositories

New git repositories will be required for the Designate and Designate Bind charms:

https://git.openstack.org/openstack/charm-designate
https://git.openstack.org/openstack/charm-designate-bind

Documentation

The Designate charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

Provide rabbitmq interface layer
Provide mysql-shared interface layer
Provide pgsql interface layer
Provide keystone interface layer
Provide hacluster interface layer
Provide nrpe-external-master interface layer
Provide OpenStack base layer with all common hook code that is not already covered by an interface layer.
Provide OpenStack base layer with support for HA deployments
Provide OpenStack base layer with support for SSL communication
Provide OpenStack base layer with support for workload status

CephFS

Sun, 24 Mar 2019 00:00:00

CephFS has recently gone GA and we can now move forward with offering this to people as a charm. Up until this point it was considered too experimental to store production data on.

Problem Description

A new CephFS charm will be created. This charm will leverage the Ceph base layer.

Proposed Change

A new CephFS charm will be created. This charm will leverage the Ceph base layer. The effort required should be small once the Ceph base layer is ready to go.

Alternatives

GlusterFS is an alternative to CephFS and will probably fit many users needs. However there are users who don’t want to deploy additional hardware to create another cluster so this can be convenient in those cases.

Implementation

Assignee(s)

Primary assignee:: cholcombe973

Gerrit Topic

Use Gerrit topic “cephfs” for all patches related to this spec.

git-review -t cephfs

Work Items

ceph-fs charm

Create the ceph-fs charm utilizing the base Ceph layer
Expose interesting config items such as: mds cache size, mds bal mode
Create actions to allow blacklisting of misbehaving clients, breaking of locks, creating new filesystems, add_data_pool, remove_data_pool, set quotas, etc.

cephfs-interface

Create an interface to allow other charms to mount the filesytem.

Repositories

A new git repository will be required to host the ceph-fs charm:

https://git.openstack.org/openstack/charm-ceph-fs

Documentation

A README.md will be created for the charm as part of the normal development workflow.

Security

No additional security concerns.

Testing

A mojo spec will be developed to exercise this charm along with amulet tests if needed:

Deploy ceph-mon

Deploy ceph-osd

Deploy cephfs

Relate the three

Verify that CephFS can be mounted and responds to reads/writes

Dependencies

This project depends on the Ceph Layering project being successful.

Gnocchi Charm

Sun, 24 Mar 2019 00:00:00

Storage of metrics in an OpenStack Cloud is a big data problem.

Ceilometer has used MongoDB since its initiation for storage of measures; however this has not proven scalable, and the telemetry project have moved to a default of using Gnocchi (a spin out of the telemetry project) for storage of measures.

We need to charm Gnocchi to allow OpenStack Operators to deploy and use telemetry services as part of an OpenStack cloud.

Problem Description

Gnocchi is a multi-tenant timeseries, metrics and resources database. It provides an HTTP REST interface to create and manipulate the data. It is designed to store metrics at a very large scale while providing access to metrics and resources information and history.

Proposed Change

The new Gnocchi charm should include, as a minimum, the following features:

Deployable in a highly available configuration
Allow clients and services to interact using SSL encryption
Charm progress displayed via workload status

The charm will use the default storage options:

Storage driver: Ceph (preferred)
Indexing driver: MySQL (not preferred but default for rest of OpenStack)
Coordination: Memcache (zookeeper and redis are alternative options)

The choice of coordination driver is limited to memcache, zookeeper or redis as these drivers support the required features for Gnocchi.

The Gnocchi charm will support OpenStack Mitaka and later on Ubuntu 16.04.

Graphical reporting for Ceilometer was dropped from Horizon during Ocata; the telemetry stack will make use of the Grafana Charm + the Gnocchi Plugin for graphical reporting on Gnocchi metrics.

Alternatives

None.

Implementation

Assignee(s)

Primary assignee:: jamespage

Gerrit Topic

Use Gerrit topic “gnocchi-charm” for all patches related to this spec.

git-review -t gnocchi-charm

Work Items

Reactive Interfaces

interface: gnocchi

Provide Gnocchi charm

Create skeleton charm layer based on OpenStack base layer and available interface layers to deploy Gnocchi.
Add support for upgrading Gnocchi
Add config option and accompanying support for upgrades via action-managed-upgrade.
Add support for deploying Gnocchi in a highly available configuration
Add support for the Gnocchi to display workload status
Add support SSL endpoints
Charm should have unit and functional tests.

Update Ceilometer and Aodh charms

Support for deployment with Gnocchi

Mojo specification deploying and testing Gnocchi

Update HA Mojo spec for deploying Gnocchi in an HA configuration.

Update telemetry bundle

Update telemetry bundle to deploy Gnocchi and Grafana.

Repositories

A new git repository will be required for the Gnocchi charm:

https://git.openstack.org/openstack/charm-gnocchi

Documentation

The Gnocchi charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

No dependencies outside of this specification.

Designate - Neutron integration

Sun, 24 Mar 2019 00:00:00

Users of an OpenStack cloud would like to use Designate DNS records auto-generation feature which is already available in the upstream:

https://docs.openstack.org/ocata/networking-guide/config-dns-int.html

Because of the reasons described below, the existing solutions based on integration of Designate with Nova via notification_topics does not satisfy their requirements.

Problem Description

With OpenStack Mitaka release there has been a new feature introduced which is integration of the Compute service and the Networking service with an external DNSaaS (DNS-as-a-Service) via extension_drivers. This feature allows DNS records to be automatically generated in an external DNSaaS (i.e. Designate) when Neutron ports / floating IPs are created.

The feature has become an official standard for Designate - Neutron integration and it seems to replace the old approach based on Designate - Nova integration via notificiation_topics. It also solves the key limitation of the old approach which was lack of support for multiple DNS domains within the tenant.

In order to use this feature, the following configuration parameters need to be set on the Neutron Server side:

/etc/neutron/neutron.conf

[default] external_dns_driver = designate

[designate] url = <designate public endpoint> admin_auth_url = <keystone admin endpoint> admin_username = <admin user username> admin_password = <admin user password> admin_tenant_name = <admin user tenant> allow_reverse_dns_lookup = <allow creation of PTR records> ipv4_ptr_zone_prefix_size = <IPv4 PTR zone prefix size> # optional ipv6_ptr_zone_prefix_size = <IPv6 PTR zone prefix size> # optional cafile = <path to CA certificate> # optional
/etc/neutron/plugins/ml2/ml2_conf.ini

[ml2] extension_drivers = dns

However, currently the neutron-api charm does not support any way of setting up these parameters, except of ml2 extension.

Proposed Change

The proposed change can be split to the following parts:

Create new interface (designate) to expose public API endpoint of Designate service.
Create new relation (“external-dns”) between designate and neutron-api charms to send Designate public endpoint through relation data via the “designate” interface. Implement reactive handlers / hooks on both designate and neutron-api charms size to support the new relation.
Add new config options to neutron-api charm to handle the following parameters:

“allow_reverse_dns_lookup”:

purpose: specifies whether to enable or not the creation of PTR records data type: boolean valid values: True / False
“ipv4_ptr_zone_prefix_size”:

purpose: the size in bits of the prefix for the IPv4 reverse lookup zones data type: int valid values: 0 - 32
“ipv6_ptr_zone_prefix_size”:

purpose: the size in bits of the prefix for the IPv6 reverse lookup zones data type: int valid values: 0 - 128

Implement new class (“DesignateContext”) on neutron-api charm to generate the following context based on relation data and config options:

ctxt = {‘enable_designate’: <True if “external-dns” relation established>,
‘designate_endpoint’: <consumed from “designate” interface>, ‘allow_reverse_dns_lookup’: <based on config option>, ‘ipv4_ptr_zone_prefix_size’: <based on config option>, ‘ipv6_ptr_zone_prefix_size’: <based on config option>}
Update any other context classes which may be affected by the change.
Update templates to enable the feature if “external-dns” relation is established. Remaining parameters will be retrieved from the existing contexts.

Alternatives

The designate charm currently supports integration with nova-compute charm for DNS records auto-generation via notification_topics. This approach, however, has some limitations which can be summarized as follows:

It seems to be obsolete starting from OpenStack Mitaka release in favor of Designate - Neutron integration via extension_drivers,
It lacks support for multiple DNS domains within the tenant.

Implementation

Assignee(s)

Primary assignee:: tkurek

Gerrit Topic

Use Gerrit topic designate-neutron for all patches related to this spec.

git-review -t charm-designate-neutron

Work Items

Update designate charm

Update metadata.yaml and layer.yaml files to support new relation and interface
Add new reactive handler to support new relation and interface
Update config.yaml file to indicate that old options are obsolete
Update README.md to describe new feature
Update unit tests

Update neutron-api charm

Update metadata.yaml file to support new relation and interface
Update neutron-api charm to consume “designate_endpoint” from relation data
Add new config options to the neutron-api charm
Update template files to implement upstream feature
Update README.md to describe new feature
Update unit tests

Provide designate interface

Create new interface to expose public API endpoint of Designate service

Repositories

A new git repository will be required for the designate interface:

https://git.openstack.org/openstack/charm-interface-designate

Documentation

README files of designate and neutron-api charms will need to be updated to describe the new feature.

Security

No security implications for this change.

Testing

Unit and functional tests of designate and neutron-api charms will need to be updated to support implementation of the new feature.

Dependencies

No external dependencies.

Panko Charm

Sun, 24 Mar 2019 00:00:00

Ceilometer used to provide an event API to query and store events from different OpenStack services. However, this functionality was deprecated in Newton and removed in Ocata. Event storage and querying functionality is now provided by a service called Panko. Use-cases of historical event data storage include audit logging, debugging and billing.

Problem Description

Panko is an event storage service that provides an ability to store and query event data generated by Ceilometer with potentially other sources. Panko includes support for several storage options (sqlalchemy-compatible databases, mongodb, elasticsearch) which differ in their level of maturity.

At its core Panko is a regular API service with a database backend.

Events are published to Panko via Direct publisher in Ocata while in Pike Direct publisher was deprecated and will be removed. For that reason Panko publisher was added.

Direct publisher deprecation (ceilometer/publisher/direct.py) was done under this commit.

Another mechanism that was deprecated in Pike is dispatchers which were used to send data specified by publishers. So were {event,meter}_dispatchers options in ceilometer.conf

Panko dispatcher deprecation.
Notes on unneeded duplication of publishers and dispatchers.
A discussion on dispatchers vs publishers.

This is instead done directly by publishers in Pike and Panko publisher is present in Panko’s repository itself, not ceilometer repository.

Panko first appeared in Ocata Ubuntu Cloud Archive.

Ceilometer is able to query Panko’s presence via Keystone catalog but does not define a publisher for sending event data to Panko by default.

Proposed Change

The new charm should include the following features:

Support SQLAlchemy-compatible databases as storage backends;
HA support;
TLS support;
integration with Ceilometer charm.

Alternatives

None for historical event data within OpenStack.

Implementation

Assignee(s)

Primary assignee:: dmitriis

Gerrit Topic

Use Gerrit topic “panko-charm” for all patches related to this spec.

git-review -t panko-charm

Work Items

Reactive Interfaces

interface: panko

Provide Panko charm

Create a charm layer based on openstack-api layer;
Add support for upgrading Panko (schema changes);
Add support for deploying Panko in a highly available configuration;
Add support for the Panko to display workload status;
Add support TLS endpoints;
Charm should have unit and functional tests.

Update Ceilometer Charm

Support for deployment with Panko (by specifying publishers correctly in event_pipeline.yaml for both Ocata and Pike+).

Mojo specification deploying and testing Panko

Update HA Mojo spec for deploying Panko in an HA configuration.

Update telemetry bundle

Update telemetry bundle to deploy Panko

Repositories

A new git repository will be required for the Panko charm:

https://git.openstack.org/openstack/charm-panko

Documentation

The Panko charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

No dependencies outside of this specification.

Mistral Charm

Sun, 24 Mar 2019 00:00:00

To add a service to Openstack to set up and manage on-schedule jobs for multiple machine.

Problem Description

As a cloud administrator I have maintenance jobs that I’d like to be run against the cloud. I want to ble to set up and manage on-schedule jobs for multiple machines. I’d like a single point of control over their schedule.

As a devops engineer I’d like to be able to specify workflows needed for deploying environments consisting of multiple VMs and applications.

As a business process enforcer I’d like to be able to make a request to run a complex multi-step business process and have it be fault-tolerant so that if the execution crashes at some point on one node then another active node of the system can automatically take on and continue from the exact same point where it stopped.

As a data analyst I need a tool for data crawling. Eg I’d like to be able to express as a graph the related tasks I need in order to prepare a financial report.

Proposed Change

One new charm - Mistral with corresponding tests and QA CI/setup.

The new Mistral charm should include, as a minimum, the following features:

Deployable in a highly available configuration
Allow clients and services to interact using SSL encryption
Charm progress displayed via workload status

Alternatives

Jobs could scheduled manually via cron on each machine.

Implementation

Assignee(s)

Primary assignee:: unknown

Gerrit Topic

Use Gerrit topic “mistral” for all patches related to this spec.

git-review -t mistral

Work Items

Provide Mistral charm

Create skeleton charm layer based on OpenStack base layer and available interface layers to deploy Mistral.
Add support for upgrading Mistral
Add config option and accompanying support for upgrades via action-managed-upgrade.
Add support for deploying Mistral in a highly available configuration
Add support for the Mistral to display workload status
Add support SSL endpoints
Charm should have unit and functional tests.

Mojo specification deploying and testing Mistral

Write Mojo spec for deploying Mojo in an HA configuration and testing creation of jobs.

Repositories

A new git repository will be required for the Mistral charm:

https://git.openstack.org/openstack/charm-mistral

Documentation

The Mistral charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

Provide rabbitmq interface layer
Provide mysql-shared interface layer
Provide pgsql interface layer
Provide keystone interface layer
Provide hacluster interface layer
Provide nrpe-external-master interface layer
Provide OpenStack base layer with all common hook code that is not already covered by an interface layer.
Provide OpenStack base layer with support for HA deployments
Provide OpenStack base layer with support for SSL communication
Provide OpenStack base layer with support for workload status

Murano Charm

Sun, 24 Mar 2019 00:00:00

To add a service to Openstack provide a catalogue of applications deployable on Openstack.

Problem Description

To provide UI and API which allows to compose and deploy composite environments on the Application abstraction level and then manage their lifecycle. The Service should be able to orchestrate complex circular dependent cases in order to setup complete environments with many dependent applications and services. However, the actual deployment itself will be done by the existing software orchestration tools (such as Heat), while the Murano project will become an integration point for various applications and services.

Proposed Change

One new charm - Murano with corresponding tests and QA CI/setup.

The new Murano charm should include, as a minimum, the following features:

Deployable in a highly available configuration
Allow clients and services to interact using SSL encryption
Charm progress displayed via workload status

Alternatives

Jobs could scheduled manually via cron on each machine.

Implementation

Assignee(s)

Primary assignee:: unknown

Gerrit Topic

Use Gerrit topic “murano” for all patches related to this spec.

git-review -t murano

Work Items

Provide Murano charm

Create skeleton charm layer based on OpenStack base layer and available interface layers to deploy Murano.
Add support for upgrading Murano
Add config option and accompanying support for upgrades via action-managed-upgrade.
Add support for deploying Murano in a highly available configuration
Add support for the Murano to display workload status
Add support SSL endpoints
Charm should have unit and functional tests.

Mojo specification deploying and testing Murano

Write Mojo spec for deploying murano in an HA configuration and testing creation of jobs.

Repositories

A new git repository will be required for the Murano charm:

https://git.openstack.org/openstack/charm-murano

Documentation

The Murano charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

Provide rabbitmq interface layer
Provide mysql-shared interface layer
Provide pgsql interface layer
Provide keystone interface layer
Provide horizon interface layer
Provide heat interface layer
Provide hacluster interface layer
Provide nrpe-external-master interface layer
Provide OpenStack base layer with all common hook code that is not already covered by an interface layer.
Provide OpenStack base layer with support for HA deployments
Provide OpenStack base layer with support for SSL communication
Provide OpenStack base layer with support for workload status

Trove Charm

Sun, 24 Mar 2019 00:00:00

To add a service to Openstack to provide on-demand databases.

Problem Description

As a cloud user I need be able to deploy a database that is scalable and reliable quickly and easily without the burden of handling complex administrative tasks.

Proposed Change

One new charm - Trove with corresponding tests and QA CI/setup.

The new Trove charm should include, as a minimum, the following features:

Deployable in a highly available configuration
Allow clients and services to interact using SSL encryption
Charm progress displayed via workload status

Alternatives

juju deploy {mysql,percona-cluster,cassandra,etc}

Implementation

Assignee(s)

Primary assignee:: unknown

Gerrit Topic

Use Gerrit topic “trove” for all patches related to this spec.

git-review -t trove

Work Items

Provide Trove charm

Create skeleton charm layer based on OpenStack base layer and available interface layers to deploy Trove.
Add support for upgrading Trove
Add config option and accompanying support for upgrades via action-managed-upgrade.
Add support for deploying Trove in a highly available configuration
Add support for the Trove to display workload status
Add support SSL endpoints
Charm should have unit and functional tests.

Mojo specification deploying and testing Trove

Write Mojo spec for deploying Trove in an HA configuration and testing creation of databases.

Repositories

A new git repository will be required for the Trove charm:

https://git.openstack.org/openstack/charm-trove

Documentation

The Trove charm should contain a README with instructions on deploying the charm. A blog post is optional but would be a useful addition.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a combination of Amulet, Bundle tester and Mojo specification.

Dependencies

Provide rabbitmq interface layer
Provide mysql-shared interface layer
Provide pgsql interface layer
Provide keystone interface layer
Provide hacluster interface layer
Provide nrpe-external-master interface layer
Provide OpenStack base layer with all common hook code that is not already covered by an interface layer.
Provide OpenStack base layer with support for HA deployments
Provide OpenStack base layer with support for SSL communication
Provide OpenStack base layer with support for workload status

Octavia Charm

Sun, 24 Mar 2019 00:00:00

Problem Description

At present our support for LBaaS makes use of the neutron-lbaas service_plugin and service_provider drivers. In the existing implementation the load balancer code, which is a part of the datapath, is run in namespaces directly on the neutron gateway units.

The neutron-lbaas and neutron-lbaas-dashboard projects are marked as deprecated as of the Queens OpenStack release cycle and has stopped receiving updates. They will be removed entirely at some point in the future.

Our usage of neutron-lbaas driver also gives a sub-optimal configuration for users in need of TLS.

Proposed Change

The neutron-lbaas projects have been replaced by octavia, octavia-dashboard, and python-octaviaclient.

octavia has a separate API endpoint and implements a superset of the LBaaS v2 API. For migration of existing users there is a lbaasv2-proxy service plugin which allows neutron API endpoint to forward LBaaS v2 API calls to the octavia endpoint in a migration period.

Contrary to the neutron-lbaas implementation, octavia runs the in-datapath load balancer code as instances in your cloud. This gives richer flexibility both in terms of freedom of choice of provider of the load balancer software itself and dynamic scaling of the service.

On the scaling side, most if not all, of the octavia services benefit from being scaled out proportional to the number of running load balancers and load balancer life cycle events. Thus it makes sense to co-locate the API, Worker, Health Manager and Housekeeping Manager daemons in the same charm unit, and scale by increasing the number of charm units deployed.

A reference implementation load balancer based on Ubuntu cloud images running HAProxy called amphorae is available.

Alternatives

As our current dependencies for providing LBaaS are deprecated and set to be removed at a future date there are no real alternatives to doing this work.

Implementation

Assignee(s)

Primary assignee:: fnordahl

Gerrit Topic

Use Gerrit topic “octavia-charm” for all patches related to this spec.

git-review -t octavia-charm

Work Items

Implement a new principal octavia charm which will deploy the octavia API, Worker, Health Manager and Housekeeping Manager daemons. The charm MUST have all the characteristics you expect from a OpenStack API charm such as TLS, HA, pause / resume actions, upgrades etc.
Make necessary changes to neutron charms for operation with octavia.
Make necessary changes to openstack-dashboard charm for replacing the neutron-lbaas-dashboard with the octavia-dashboard.
Implement support for storing TLS private key secrets in Vault either through relation to barbican or by utilizing the castellan library directly.
Implement support for doing post-deployment configuration and plumbing of the private administration network between octavia and the load balancer instances it deploys. This will be done by interacting with the neutron API and deploying the neutron-openvswitch subordinate with the charm. The neutron-openvswitch subordinate will automatically take care of creating necessary tunnels for access to the overlay network and we can subsequently create OpenvSwitch ports on the units where octavia services are deployed.
Provide documentation and/or any actions necessary for obtaining, provisioning or indeed building (if necessary) the amphorae load balancer software images.
Provide documentation and/or any actions necessary for migrating existing load balancer workloads to octavia.

Repositories

A new git repository will be required for the octavia charm:

https://git.openstack.org/openstack/charm-octavia

Documentation

The octavia charm should contain a README with instructions on deploying the charm.

Security

Serving TLS terminated traffic is a basic feature of a load balancer service. With that comes responsibility for secure handling and storage of sensitive information such as private keys.

Best practices for handling and storing these keys will be implemented.
If we are to provide our user with a mechanism to build the amphorae images, it must be made clear that the responsibility of keeping the built image secure and up to date rests solemnly with our user.
The presence of a direct network tunnel between the octavia unit placed in the undercloud control plane and the load balancer instances in the overcloud is a construct we have not encountered previously. We must consider all aspects of this carefully with security and integrity in mind.

Testing

Code written or changed will be covered by unit tests; functional testing will be done using Zaza.

Dependencies

To be able to support deployment of octavia charm in LXD containers we depend on Juju implementation of charm controlled LXD profile updates specification.

SR-IOV networking support

Thu, 27 Dec 2018 00:00:00

SR-IOV is a mechanism for passing hardware virtualized network functions (VF) or full physical function (PF) directly to KVM instances; the OpenStack charms should be updated to support this capability.

Problem Description

Using Open vSwitch and the chain of bridges, veth pairs and tap devices incurs an overhead on network throughput and latency that is not acceptable in some use cases for OpenStack.

SR-IOV allows part of (a VF) or a full (a PF) SR-IOV enabled network card to be connected directly to a KVM instance via the virtualization layer provided by libvirt.

Support for both of these types of passthrough was implemented in full in the Mitaka release.

Proposed Change

Supporting SR-IOV will require changes in three charms; specifically:

neutron-api: Enablement of the ML2 mechanism driver for SR-IOV
nova-cloud-controller: Enablement of appropriate scheduler filters for management of PCI passthrough devices
nova-compute: White listing of PCI devices for use by compute instances.

SR-IOV will be enabled using a configuration option on the neutron-api charm ‘enable-sriov’; The nova-cloud-controller charm will be notified of the state of this option via the relation between the nova-cloud-controller charm and the neutron-api charm.

In the first implementation, a pci-device-whitelist configuration option will be provided by the nova-compute charm to allow allocation of specific PCI devices to compute instances. This is a direct pass through to a Nova configuration option. Later revisions of this feature may use as-yet unimplemented features in Juju to manage PCI device allocation at the unit level.

Initial SR-IOV support will be agent-less (i.e. not using the neutron-sriov-agent).

Flat and VLAN networking modes will be supported.

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: james-page

Gerrit Topic

Use Gerrit topic “sriov-support” for all patches related to this spec.

git-review -t sriov-support

Work Items

SR-IOV VF/PF scheduling testing

Focussed testing on ensuring that scheduling behaviour is tuned appropriately when SR-IOV is in use within a cloud.

Juju device binding support

Stretch objective for Newton cycle; superceeds pci-device-whitelist option on nova-compute charm

Repositories

No new git repositories required.

Documentation

Updates to the README’s in the impacted charms will be made as part of this change.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a Mojo specification.

Dependencies

OpenStack Mitaka.
SR-IOV enabled hardware in the Ubuntu OpenStack QA lab for functional testing
Juju device management support

Cells V2

Fri, 07 Dec 2018 00:00:00

Nova cells v2 has been introduced over the Ocata and Pike cycles. In fact, all Pike deployments are now deployments using nova cells v2 usually using a single compute cell.

Problem Description

Nova cells v2 allows for a group of compute nodes to have their own dedicated database, message queue and conductor while still being administered through a central API service. This has the following benefits:

Reduced pressure on Rabbit and MySQL in large deployments

In even moderately sized clouds the database and message broker can quickly become a bottle neck. Cells can be used to alleviate that pressure by having a database and message queue per cell of compute nodes. It is worth noting that the charms already support having traffic for neutron etc in a separate rabbit instance.

Create multiple failure domains

Grouping compute cells with their local services allows the creation of discrete failure domains (from a nova POV at least).

Remote Compute cells (Edge computing)

In some deployments a group of compute nodes maybe far removed (from a networking pov) from the central services. In this case it maybe useful to have the compute nodes act as a largely independent group.

Different SLAs per cell

Different groups of compute nodes can have different levels of performance, HA. etc. A cell could have no local HA for the database, message queue and conductor for a development cell but the production cell could have significantly higher specification servers running clustered services.

(These use cases were paraphrased from *4.)

Proposed Change

To facilitate a cells v2 deployment a few relatively simple interfaces and relations need to be added. From a nova perspective the topology looks like this. This spec proposes mapping that to this charm topology.

Superconductor access to Child cells

The superconductor needs to be able to query the databases of the compute cells and to send and receive messages on the compute_cells message bus. The cleanest way to model this would be to have a direct Juju relation between the superconductor and the compute cells database and message bus. To facilitate this the following relations will be added to the nova-cloud-controller charm:

requires:
  shared-db-cell:
    interface: mysql-shared
  amqp-cell:
    interface: rabbitmq

Superconductor configuring child cells

With the above change the superconductor has access to the child db and mq but does not know which compute cell name to associate with them. To solve this the nova-cloud-controller charm will have the following new relations:

provides:
  nova-cell-api:
    interface: cell
requires:
  nova-cell:
    interface: cell

The new cell relation will be used to pass the cell name, db service name and message queue service name.

{
  'amqp-service': 'rabbitmq-server-cell1',
  'db-service': 'mysql-cell1',
  'cell-name': 'cell1',
}

Given this information the superconductor can examine the service names that are attached to its shared-db-cell and amqp-cell relations and construct urls for them. The superconductor is then able to create the cell mapping in the api database by running:

nova-manage cell_v2 create_cell \
    --name <cell_name> \
    --transport-url <transport_url> \
    --database_connection <database_connection>

The superconductor needs five relations to be in place and their corresponding contexts to be complete before the cell can be mapped. Given the nova-cloud-controller is a non-reactive charm special care will be needed to ensure that the cell mapping happens irrespective of the order in which those relations are completed.

Compute conductor no longer registering with keystone

The compute conductor does not need to register an endpoint with keystone nor does it need service credentials. As such the identity-service relation should not be used for compute cells. A guard should be put in place in the nova-cloud-controller charm to prevent a compute cells nova-cloud-controller from registering an incorrect endpoint in keystone.

Compute conductor cell name config option

The compute conductor needs to know its own cell name so that it can pass this information up to the superconductor. To allow this a new configuration option will be added to the nova-compute charm:

options:
  cell-name:
    type: string
    default:
    description: |
      Name of the compute cell this controller is associated with. If this is
      left unset or set to api then it is assumed that this controller will
      be the top level api and cell0 controller.

Leaving the cell name unset assumes the current behaviour of associating the nova-cloud-controller with the api service, cell0 and cell1.

nova-compute service credentials

The nova-compute charm needs service credentials for RPC calls to the Nova Placement API and the Neutron API service. It currently gets these credentials via its cloud-compute relation which is ugly at best. However, given that the compute cells nova-cloud-controller will no longer have a relation with keystone it will not have any credentials to pass on to nova-compute. This is overcome by adding a cloud-credentials relation to the nova-compute charm.

requires:
  cloud-credentials:
    interface: keystone-credentials

nova-compute will request a username based on its service name so that users for different cells can be distinguished from one another.

Bespoke vhosts and db names

The ability to specify a nova db name and a rabbitmq vhost name should either be removed from the nova-cloud-controller charm or the new cell interface needs to support passing those up to the superconductor so that the superconductor can request access to the correct resources from the compute nodes database and message queue.

Disabling unused services

The compute cells nova-cloud-controller only needs to run the conductor service and possible the console services. Unused services should be disabled by the charm.

New cell conductor charm?

The nova cloud controller in a compute node only runs a small subset of the nova services and does not require a lot of the complexity that is baked into the current nova-cloud-controller charm. This begs the question of whether a new cut-down reactive charm that just runs the conductor would make sense. Most of the changes outlined above actually impact the superconductor rather than the compute conductor. However, looking at this the other way around the changes needed to allow the nova-cloud-controller charm to act as a child conductor are actually quite small and so probably do not warrant the creation of a new charm. It is probably worth noting some historical context here too, every time the decision has been made to create a charm which can operate in multiple modes that decision has been reversed at some cost at a later data ( ceph being a prime example).

Taking all that into consideration a new charm will not be written and the existing nova-cloud-controller charm will be extended to add support for running as a compute conductor.

Message Queues

There is flexibility around which message queue the non-nova services use. A dedicated rabbit instance could be created for them or they could reuse the rabbit instance the nova api service is using.

Telemetry etc

This spec does not touch on integration with telemetry. However, this does require further investigation to ensure that message data can be collected.

Juju service names

It will be useful, but not required, to embed the cell name in the service name of each component that is cell specific. Eg deploying services for cellN may look like this:

juju deploy nova-compute nova-compute-cellN
juju deploy nova-cloud-controller nova-cloud-controller-cellN
juju deploy mysql mysql-cellN
juju deploy rabbitmq-server rabbitmq-server-cellN

Alternatives

Do nothing and do not support additional nova v2 cells.
Resurrect support for the deprecated and bug ridden cells v1

Implementation

Assignee(s)

Primary assignee:: Unknown

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t cellsv2

Existing Work

As part of writing the spec prototype charms and a bundle were created for reference: Bundle and charm diffs

Work Items

Remove support for cells v1 from nova-compute and nova-cloud-controller charms
Add identity-context relation to nova-compute and ensure the supplied credentials are used when rendering placement and keystone sections in nova.conf
Add shared-db-cell relation to nova-cloud-controller assuming ‘nova’ database name when requesting access.
Add amqp-cell relations to nova-cloud-controller assuming ‘openstack’ vhost name when requesting access.
Add code for registering a cell to nova-cloud-controller. This will use the AMQ and SharedDB contexts from the shared-db-cell and amqp-cell relation to create the cell mapping.
Update nova.conf templates in nova-cloud-controller to only render api db url if the nova-cloud-controller is a superconductor.
Update db initialisation code to only run the relevant cell migration if not a superconductor.
Add nova-cell and nova-cell-api relations and ensure that the shared-db, amqp shared-db-cell, amqp-cell and nova-api-cell relations all attempt to register compute cells.
Write bundles to use cells topology
Check integration with other services (designate and telemetry in particular)

Repositories

No new repositories needed.

Documentation

READMEs of nova-cloud-controller and nova-compute will need updating to explain new relations and config options.
Blog with deployment walkthrough and explanation.
Update Openstack Charm documentation to explain how to do a multi-cell deployment
Add bundle to charm store.

Security

No new security risks that I am aware of

Testing

A multi-cell topology is probably beyond the scope of amulet tests
Bundles added to openstack-charm-testing
Mojo specs

Dependencies

None that I can think of

Credits

Much of the benefit of cells etc was lifted from *4

*1 https://docs.openstack.org/nova/pike/cli/nova-manage.html *2 https://docs.openstack.org/nova/latest/user/cellsv2-layout.html *3 https://bugs.launchpad.net/nova/+bug/1742421 *4 https://www.openstack.org/videos/sydney-2017/adding-cellsv2-to-your-existing-nova-deployment

External Networking REDUX

Tue, 13 Nov 2018 00:00:00

Refactor neutron-gateway and neutron-openvswitch charms to support more flexible configuration of ‘external’ north/south traffic routing to tenant networks.

Problem Description

The neutron-gateway and neutron-openvswitch charms currently support the ‘ext-port’ configuration option to plumb the Open vSwitch br-ex bridge into the external/public network.

OpenStack switched to a more flexible way of doing this a few cycles back, and the use of the default ‘br-ex’ == external network approach is officially deprecated.

We need to refactor the charms to support the reference way of configuring external networking, which provides more flexibility than the current approach but does require more knowledge of Neutron.

Use Cases

Using the reference approach to external networking, its possible to have multiple external networks served from the same set of neutron gateway units, potentially on the same physical network with VLAN segmentation:

neutron net-create --provider:network_type vlan \
                   --provider:segmentation_id 400 \
                   --provider:physical_network physnet1 --shared external

neutron net-create --provider:network_type vlan \
                   --provider:segmentation_id 401 \
                   --provider:physical_network physnet1 --shared \
                   --router:external=true floating

neutron router-gateway-set provider floating

with the associated neutron-gateway configuration:

neutron-gateway:
    bridge-mappings:         physnet1:br-data
    data-port:               br-data:eth1

Proposed Change

Both the neutron-gateway and neutron-openvswitch charms will need to be updated to support this change; for now the ‘ext-port’ configuration will be deprecated for backwards compatibility with existing installations preserving current behaviour.

The existing ‘data-port’ configuration option will be used instead to configure bridge-mappings and connectivity to physical network ports on the units deployed.

Alternatives

No alternatives.

Implementation

Assignee(s)

Primary assignee:: james-page
Additional contributors:: lathiat (Trent Lloyd)

Gerrit Topic

Use Gerrit topic “ext-net-redux” for all patches related to this spec.

git-review -t ext-net-redux

Work Items

Updates for neutron-gateway charm
Updates for neutron-openvswitch charm
Updates for openstack-charm-testing to configure external networking in the new style.
Updates to Mojo specifications for new style configuration.
Updates to official bundles with details on new configuration for external networking.

Repositories

None required.

Documentation

README’s in impacted charms will be updated with details on how to configure external networking.

Security

No additional security requirements.

Testing

Unit tests will be used to validate the code; Mojo specifications will be updated to use the new style of external networking configuration along with openstack-charm-testing.

Dependencies

No additional dependencies for this specification.

References

Ceph Broker

Tue, 13 Nov 2018 00:00:00

Connect an existing Ceph to a juju deployed Openstack

Problem Description

Some customers have asked how to connect a Juju deployed Ceph to an existing OpenStack cluster that was deployed with home grown or someone else’s tooling. This causes a problem in that it is not possible to relate Ceph to the OpenStack cluster.

Proposed Change

A new charm will be created that acts a broker between an existing Ceph deployment, and a Juju deployed OpenStack Cloud; The charm will provide the same relations as the existing ceph-mon charm.

Alternatives

The alternative is manually connecting the Ceph and OpenStack together. This is fine for some customers but not acceptable for bootstack. This kind of manual configuration isn’t particularly manageable.

Implementation

Assignee(s)

Primary assignee:: ChrisMacNaughton

Gerrit Topic

git-review -t ceph-broker

Work Items

Decide on which relations the OpenStack charm requires

Expose all relations needed by way of config.yaml options.

For every relation that OpenStack expects just return the config.yaml values.

Repositories

Yes a new repo will be needed for this. https://github.com/openstack/charm-ceph-broker

Documentation

Documentation will be added to the README.md as part of the normal workflow.

Security

No additional security concerns.

Testing

Deploy OpenStack using juju.
Deploy Ceph using juju.
Deploy the Ceph-broker to a lxd container or a vm after filling out the config.yaml
Relate Ceph-broker to OpenStack and verify that OpenStack can talk to Ceph
Mojo bundle tests will be used to show this works functionally.

Dependencies

None

Service Restart Control In Charms

Thu, 13 Sep 2018 00:00:00

Problem Description

For example, let’s say an application unit knows it has id 215 and the user has provided two options via config; a modulo value of 2 and an offset of 10. We could then do the following:

time.sleep((215 % 2) * 10)

As mentioned above we will require two new config options to any charm for which this logic is supported:

service-restart-offset (default to 10)
service-restart-modulo (default to 1 so that default behaviour is same as
before)

The restart logic will skip for any charms not implementing these options.

Proposed Change

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: hopem

Gerrit Topic

Use Gerrit topic “controlled-service-restarts” for all patches related to this spec.

git-review -t controlled-service-restarts

Work Items

implement changes to charmhelpers
sync into openstack charms and add new config opts

Repositories

None

Documentation

These new settings will be properly documented in the charm config.yaml as well as in the charm deployment guide.

Security

None

Testing

Dependencies

None

Keystone Federation

Thu, 13 Sep 2018 00:00:00

Problem Description

The following are not covered here:

Integration with Kerberos
Horizon SSO
Federated LDAP via SSSD and mod_lookup_identity
Keystone acting as an IdP in a federated environment.

Backends

The keystone project highlights SQL and LDAP (inc AD) as their supported backends. The status of support for these is as follows:

SQL: Currently supported by the keystone charm.
LDAP: Currently supported by the keystone and keystone-ldap subordinate.

Adding a new backend is achieved by writing a keystone subordinate charm and relating it to keystone via the keystone-domain-backend interface.

Enabling a backend tends to be achieved via the keystone.conf

Federation

Each Identity provider must be associated with a different domain within keystone. The keystone v3 API is needed to support federation.

Compatible Identity Providers: (https://docs.openstack.org/ocata/config-reference/identity/federated-identity.html#supporting-keystone-as-a-sp ):

OpenID
SAML

Proposed Change

Alternatives

Add support for federation via OpenID and SAML directly to the keystone charm.
Create a new interface for federation via OpenID and SAML

Implementation

Assignee(s)

Primary assignee:: None

Gerrit Topic

Use Gerrit topic “keystone_federation” for all patches related to this spec.

git-review -t <keystone_federation>

Work Items

Create deployment scripts to create test env for OpenID or SAML integration

Extend keystone-domain-backend

The keystone-domain-backend interface will need to provide the following:

Modules for Apache to enable
Configuration for principle to insert into Apache keystone wsgi vhosts
Subordinate triggered restart of Apache
Auth method(s) to be added to keystone’s [auth] methods list
Configuration for principle to insert into keystone.conf

Configure Keystone to consume new interface

Keystone charm will need to be updated to respond to events outlined in the interface description above

New keystone-openid and keystone-saml subordinates

Repositories

New projects for the interface and new subordinates will be needed.

Documentation

This will require documentation in the READMEs of both the subordinates and the keystone charm. A blog walking through the deployment and integration would be very useful.

Security

Testing

Dependencies

None

Extended Swift Cluster Operations

Thu, 13 Sep 2018 00:00:00

Problem Description

As they stand, these charms currently support a base set of what is required to effectively maintain a Swift cluster over time:

deploy swift cluster with configurable min-part-hours, replica count, partition-power and block storage devices.
once deployed the only changes that can be made are the addition of block devices and modification of min-part-hours. Changes to partition-power or replicas are ignored by the charm once the rings have already been initialised.

This forces operators to manually apply changes like adjusting the partition-power to accomodate for additional storage added to the cluster. This poses great risk since manually editing the rings/builders and syncing them across the cluster could easily conflict with the swift-proxy charm’s native support for doing this resulting in a broken cluster.

Proposed Change

Currently we check whether the rings are already initialised and if they are we ignore the partition power and replica count configured in the charm i.e. changes are not applied. To make it possible to do this we will need to remove these blocks and implement the steps documented in [0] and [1]. I also propose that charm impose a cluster size limit (number of devices) above which we refuse to make changes until the operator has paused the swift-proxy units i.e. placed them into “maintenance mode” which will shutdown the api services and block any restarts until the units are resumed. The user will also have the option to set disable-ring-balance=true if they want check that their changes have been applied successully (to the builder files) prior to having the rings rebuilt and sycned across the cluster.

[0] https://docs.openstack.org/swift/latest/ring_partpower.html [1] https://docs.openstack.org/swift/latest/admin/objectstorage-ringbuilder.html#replica-counts

Alternatives

Implementation

Assignee(s)

Primary assignee:: hopem

Gerrit Topic

Use Gerrit topic swift-charm-extended-operations for all patches related to this spec.

git-review -t swift-charm-extended-operations

Work Items

add support for modifying partition-power
add support for modifying replicas
add support for removing devices
add support for removing an entire swift-storage host

Repositories

None

Documentation

All of the above additions will need to be properly documented in the charm deployment guide.

Security

None

Testing

Dependencies

None

Keystone - Change default preferred API version

Thu, 13 Sep 2018 00:00:00

The current default preferred API version used by the Keystone Charm is v2.0.

There exists multiple use cases where the use of the v3 API is required. To use features such as domains, domain specific drivers, LDAP authentication, Federation (OpenID Connect, SAML) and others not mentioned, the Keystone V3 API endpoints must be exposed in the catalog and all services in a deployment must leverage the V3 API when authenticating with Keystone. Increasingly so these use cases would be the desired mode of operation for new OpenStack deployments.

Additionally the v2.0 API has been marked as DEPRECATED since the OpenStack Mitaka release and will be REMOVED in the Queens release cycle.

Problem Description

The Keystone charm currently has two separate code paths for managing the Keystone service. Which path is exercised depends on the value of the ‘preferred-api-version’ setting. This leads to duplication of both code, and potential bugs.

The majority of our functional tests are currently run with services configured to talk to Keystone using the v2.0 API.

Change of endpoints in the Keystone catalog and reconfiguration of services to use the v3 API automatically in already running deployments may lead to unforeseen and undesirable side effects. We must find a way to have the change of default affect new deployments only, and leave existing deployments untouched on charm upgrade.

Proposed Change

Remove API v2.0 specific code from ‘hooks/manager.py’ and make use of a combination of the ‘keystone-manage’ CLI tool and the v3 API to manage Keystone service regardless of the value of ‘preferred-api-version’ configuration option.

Re-wire existing unit- and functional- tests to be performed on deployments configured to use both v2.0 and v3 API. This induces change in both the keystone charm itself as well as any other charm with specific tests for how it relates to the keystone charm.

Remove default value for ‘preferred-api-version’ in config.yaml and let charm decide which default to use programatically. The upgrade function should be made to retain ‘preferred-api-version’ as 2 for existing deployments that makes use of the current default.

Version fence will be implemented in charm to have the new default apply for OpenStack versions ‘Queens’ and onwward.

Alternatives

There are no real alternatives. Not changing the default would make our test coverage poorer for the real world use cases that our software is currently exposed to.

Implementation

Assignee(s)

Primary assignee:: fnordahl

Gerrit Topic

Use Gerrit topic “keystone-v3” for all patches related to this spec.

git-review -t keystone-v3

Work Items

Update ‘hooks/manager.py’ to use a combination of keystone-manage CLI and v3 API to manage the Keystone deployment. Remove any v2.0 API specifc code. This work item should include removal of dependency on having admin-token in Keystone configuration file and to switch charm over to use keystoneauth1 with sessions.
Update relevant functions in charmhelpers.contrib.openstack.amulet.utils
Make all existing unit- and functional- tests run on both v2.0 API and v3 API configured deployments in the gate.
Implement version fence that applies the new default for OpenStack release ‘Queens’ and onward.
Remove default value for ‘preferred-api-version’, make decision in charm programatically.
Make sure simplestreams with Keystone v3 API support is backported to our stable releases.

Repositories

No new git repositories required.

Documentation

This change calls for an update of the documentation in README.md with information about how the charm sets up domains and projects in a v3 API enabled Keystone deployment. The documentation should also include examples of openrc files and simple openstack client usage.

Security

We are exercising existing code paths and are changing the default value of how the Keystone deployment is configured to match real world usage, we are not introducing new changes that have security implications. Revisiting and re-validating the security of the existing implementation is in place as a part of this work.

Testing

Implementation will update existing unit- and functional- tests to leverage both v2.0 and v3 API configurations. Scripts, bundles and specs used in periodic and pre-release testing should also be updated to handle and excercise the new default.

Dependencies

No external dependencies.

Cells V2

Wed, 18 Jul 2018 00:00:00

Nova cells v2 has been introduced over the Ocata and Pike cycles. In fact, all Pike deployments are now deployments using nova cells v2 usually using a single compute cell.

Problem Description

Reduced pressure on Rabbit and MySQL in large deployments

Create multiple failure domains

Grouping compute cells with their local services allows the creation of discrete failure domains (from a nova POV at least).

Remote Compute cells (Edge computing)

Different SLAs per cell

(These use cases were paraphrased from *4.)

Proposed Change

Superconductor access to Child cells

requires:
  shared-db-cell:
    interface: mysql-shared
  amqp-cell:
    interface: rabbitmq

Superconductor configuring child cells

provides:
  nova-cell-api:
    interface: cell
requires:
  nova-cell:
    interface: cell

The new cell relation will be used to pass the cell name, db service name and message queue service name.

{
  'amqp-service': 'rabbitmq-server-cell1',
  'db-service': 'mysql-cell1',
  'cell-name': 'cell1',
}

nova-manage cell_v2 create_cell \
    --name <cell_name> \
    --transport-url <transport_url> \
    --database_connection <database_connection>

Compute conductor no longer registering with keystone

Compute conductor cell name config option

The compute conductor needs to know its own cell name so that it can pass this information up to the superconductor. To allow this a new configuration option will be added to the nova-compute charm:

options:
  cell-name:
    type: string
    default:
    description: |
      Name of the compute cell this controller is associated with. If this is
      left unset or set to api then it is assumed that this controller will
      be the top level api and cell0 controller.

Leaving the cell name unset assumes the current behaviour of associating the nova-cloud-controller with the api service, cell0 and cell1.

nova-compute service credentials

requires:
  cloud-credentials:
    interface: keystone-credentials

nova-compute will request a username based on its service name so that users for different cells can be distinguished from one another.

Bespoke vhosts and db names

Disabling unused services

The compute cells nova-cloud-controller only needs to run the conductor service and possible the console services. Unused services should be disabled by the charm.

New cell conductor charm?

Taking all that into consideration a new charm will not be written and the existing nova-cloud-controller charm will be extended to add support for running as a compute conductor.

Message Queues

There is flexibility around which message queue the non-nova services use. A dedicated rabbit instance could be created for them or they could reuse the rabbit instance the nova api service is using.

Telemetry etc

This spec does not touch on integration with telemetry. However, this does require further investigation to ensure that message data can be collected.

Juju service names

It will be useful, but not required, to embed the cell name in the service name of each component that is cell specific. Eg deploying services for cellN may look like this:

juju deploy nova-compute nova-compute-cellN
juju deploy nova-cloud-controller nova-cloud-controller-cellN
juju deploy mysql mysql-cellN
juju deploy rabbitmq-server rabbitmq-server-cellN

Alternatives

Do nothing and do not support additional nova v2 cells.
Resurrect support for the deprecated and bug ridden cells v1

Implementation

Assignee(s)

Primary assignee:: Unknown

Gerrit Topic

Use Gerrit topic “<topic_name>” for all patches related to this spec.

git-review -t cellsv2

Existing Work

As part of writing the spec prototype charms and a bundle were created for reference: Bundle and charm diffs

Work Items

Remove support for cells v1 from nova-compute and nova-cloud-controller charms
Add identity-context relation to nova-compute and ensure the supplied credentials are used when rendering placement and keystone sections in nova.conf
Add shared-db-cell relation to nova-cloud-controller assuming ‘nova’ database name when requesting access.
Add amqp-cell relations to nova-cloud-controller assuming ‘openstack’ vhost name when requesting access.
Add code for registering a cell to nova-cloud-controller. This will use the AMQ and SharedDB contexts from the shared-db-cell and amqp-cell relation to create the cell mapping.
Update nova.conf templates in nova-cloud-controller to only render api db url if the nova-cloud-controller is a superconductor.
Update db initialisation code to only run the relevant cell migration if not a superconductor.
Add nova-cell and nova-cell-api relations and ensure that the shared-db, amqp shared-db-cell, amqp-cell and nova-api-cell relations all attempt to register compute cells.
Write bundles to use cells topology
Check integration with other services (designate and telemetry in particular)

Repositories

No new repositories needed.

Documentation

READMEs of nova-cloud-controller and nova-compute will need updating to explain new relations and config options.
Blog with deployment walkthrough and explanation.
Update Openstack Charm documentation to explain how to do a multi-cell deployment
Add bundle to charm store.

Security

No new security risks that I am aware of

Testing

A multi-cell topology is probably beyond the scope of amulet tests
Bundles added to openstack-charm-testing
Mojo specs

Dependencies

None that I can think of

Credits

Much of the benefit of cells etc was lifted from *4

Encrypted Data at Rest

Wed, 18 Jul 2018 00:00:00

Problem Description

OpenStack Clouds provide a number of different types of storage to Cloud users, including instance ephemeral disks (typically attached to hypervisors), Cinder block devices (typically backed by some sort of storage solution such as Ceph) and Swift object storage.

By default the data residing on such virtual devices is unencrypted; regulation such as PCI-DSS and GPDR+ require that data at rest is stored encrypted, so that if devices are removed from the data center, the data on them cannot be recovered without access to the appropriate encryption keys.

Proposed Change

Underlying storage devices will be protected using dm-crypt/LUKS with encryption keys stored directly in Hashicorp Vault. No local copy of the key is made during the encryption process or the decryption process on boot.

A new tool ‘vaultlocker’ will be used to LUKS format block devices, directly storing encryption keys in Vault. Keys are referenced using the UUID of the underlying block device (which is generated as the disk is prepared for use).

On (re)boot, a vaultlocker-decrypt systemd unit will execute for each encrypted block device, retrieving the encryption key from Vault and opening the LUKS formatted block device ready for use.

vaultlocker will access vault over https using an approle issued as part of the deployment process; the approle will be passed from vault to the consuming service via a charm relation and will be scoped for access from units participating in the relation to vault.

The approle will be specific to each unit participating in the relation, with a policy that only permits read/write/delete/update/list to:

<kv-backend>/<hostname>/*

from the provided network address of the unit. Approles for other units will be visible in the relation data, but will not be usable as the CIDR ACL will not permit access.

In addition to the unit specific approle, and limitation of access to the /32 of the unit, a secret_id will also be used to authenticate use of the approle.

The secret_id will not be passed over the relation from the vault charm to the consuming application. Instead the vault charm will generate a secret_id and wrap it using Vault’s response wrapping feature. The resulting one-shot token will be passed over the relation to the consuming application unit, which can then use the token to pull the secret_id directly from Vault. This ensures that the secret_id is only known to Vault and the consuming application unit.

The one-shot token has a ttl of 1h (allowing for complex deployment with large numbers of hook executions to complete on converged hypervisor/storage machines).

The initial scope of support will include:

ceph-osd: OSD device encryption.
swift-storage: Block device encryption.
nova-compute: Ephemeral storage block device encryption; note that this requires that hypervisors are configured with a specific set of storage devices for use by Nova for ephemeral block devices for instances.

block device preparation

The encrypted block device will be labelled with a UUID generated by the charmhelper. This UUID will be used during encryption and during decryption during server reboots.

The device will be encrypted with key storage in Vault using:

vaultlocker encrypt --uuid $UUID $BLOCK_DEVICE

The resulting dm-crypt block device will be opened ready for use.

swift-storage and nova-compute

Block devices will be prepared inline with “block device preparation”; existing fstab management by the charm will be updated to use /dev/mapper/crypt-<UUID> entries with a x-systemd.requires option - for example:

/dev/mapper/crypt-$UUID  /mnt auto defaults,x-systemd.requires=vaultlocker-decrypt@$UUID.service,comment=vaultlocker 0 2

This ensures that the vaultlocker-decrypt task has completed prior to the mount of the mapper device being attempted.

ceph-osd/ceph-volume

Integration into the ceph-osd charm requires the charm to switch to using the new ceph-volume tool to manage the creation and activation of OSD’s. This requires that the block device be prepared with LVM volumes before passing to ceph-volume; to mirror existing ceph-disk functionality:

filestore

Use block device for journal and data; journal lv (osd-journal-<OSD-FSID>) created on vg ceph-<OSD-FSID> using configured journal size, data lv (osd-data-<OSD-FSID) created on vg ceph-<OSD-FSID> using remaining capacity.

pv /dev/sdb
  vg /dev/ceph-<OSD-FSID>
    lv /dev/ceph-<OSD-FSID/osd-journal-<OSD-FSID>
    lv /dev/ceph-<OSD-FSID/osd-data-<OSD-FSID>

Use separate device for journal; journal lv (osd-journal-<OSD-FSID>) created on vg ceph-journal-<UUID> of a journal device using configured journal size; data lv (osd-data-<OSD-FSID) created on ceph-<OSD-FSID> of data device using 100% of capacity.

pv /dev/sdb
  vg /dev/ceph-<OSD-FSID>
    lv /dev/ceph-<OSD-FSID/osd-data-<OSD-FSID>
pv /dev/sdg
  vg /dev/ceph-journal-<UUID>
    lv /dev/ceph-journal-<UUID>/osd-journal-<OSD-FSID>

bluestore

Bluestore is simpler in that there is no journal so a single logical volume will be created on ceph-<OSD-FSID> of the provided disk:

pv /dev/sdb
  vg /dev/ceph-<OSD-FSID>
    lv /dev/ceph-<OSD-FSID/osd-block-<OSD-FSID>

The Bluestore DB and WAL volumes may be optionally stored on separate devices again using a logical volume of the configured/default size on vg ceph-{db,wal}-<UUID>.

pv /dev/sdb
  vg /dev/ceph-<OSD-FSID>
    lv /dev/ceph-<OSD-FSID/osd-block-<OSD-FSID>
pv /dev/sdg
  vg /dev/ceph-db-<UUID>
    lv /dev/ceph-db-<UUID>/osd-db-<OSD-FSID>
pv /dev/sdh
  vg /dev/ceph-wal-<UUID>
    lv /dev/ceph-wal-<UUID>/osd-wal-<OSD-FSID>

Note that ceph-volume is only provided with Ceph Luminous or later releases; as a result encryption under Ceph Jewel is explicitly excluded from the scope of this specification.

Alternatives

ceph

Use of native suppport in Ceph for OSD encryption; discounted as it makes use of the ceph-mon cluster for key storage - keys are not sharded and deployments typically place ceph-mon units alongside ceph-osd units so its possible that the encryption keys might directly reside on the same server as encrypted Ceph OSD block devices.

Note

The ceph-osd charm already supports native Ceph block device encryption using ceph-disk/ceph-volume via the osd-encrypt option.

Support for use of vault could be added to ceph-volume; however due to the requirement to support existing Ceph releases (>= Luminous) this option is discounted in the short term but may be considered in the long term if support lands into Ceph upstream.

cinder

Cinder has native support for block device encryption using LUKS; keys are stored using Barbican which relies on HSM’s implementing PKCS#11 of KMIP to be considered secure. This would provide the required level of encryption support for Cinder block devices however does require use of a hardware based security module (Barbican does not have Vault support).

nova

Nova has native support for encryption of ephemeral disks if using an LVM backend for storage; again keys are stored in barbican, requiring use of a HSM or implementation of support for a suitable Software Security Module in Barbican. Use of this option is also limited to LVM storage only.

swift

Swift has no native encryption support so no alternatives considered for this part of the problem domain.

Implementation

Assignee(s)

Primary assignee:

james-page

Gerrit Topic

Use Gerrit topic “vaultlocker” for all patches related to this spec.

git-review -t vaultlocker

Work Items

vaultlocker

base codebase (support for encrypt/decrypt)
unit tests
functional tests

QA

mojo specification to validate encryption-at-rest support

Docs

example bundle + documentation for encryption-at-rest
appendix for deployment guide on usage and security considerations

charmhelpers

block device encryption helper

ceph-osd

add support for use of ceph-volume >= Luminous
enable support for block device encryption using vaultlocker
add relation to vault

swift-storage

enable support for block device encryption using vaultlocker
add relation to vault

nova-compute

enable support for block device encryption using vaultlocker
add relation to vault

Repositories

A new repository will be required for vaultlocker.

Documentation

Documentation will be provided as part of the ceph-osd, swift-storage and nova-compute charms.

An additional appendix will be added to the charm deployment guide to cover encryption at rest.

Security

As this solution covers the security of encryption keys used to secure block devices from unauthorized removal there are multiple security concerns to address.

Communication with Vault will be done over a TLS encrypted connection using an AppRole (without a secret_id) for authentication which will be delivered to the consuming charm over a charm relation; connectivity with Juju also TLS encrypted so the potential for interception of the AppRole is limited.

The secret_id for the unit to use with the AppRole is passed out-of-band of Juju - a one-shot token is passed over the vault-kv relation, which can only be used by the consuming unit to retrieve the generated secret_id for the AppRole. The token has a 1hr TTL and is CIDR limited in the same way as the AppRole.

Encryption keys will be stored under a Vault path specific to the AppRole. The Vault AppRole will limit access to the secrets backend based on the CIDR of the accessing servers.

Testing

Functionality will be validated by unit and functional tests within each component.

Overall solution function will be validated using a Mojo spec.

Dependencies

Production grade vault charm.
AppRole interface to vault charm.

Migrate Ceph Deployment Architecture

Mon, 09 Jul 2018 00:00:00

The all-in-one Ceph charm has been deprecated in favor of splitting out the distinct functions of the Ceph monitor cluster and the Ceph OSDs. The Ceph charm itself is receiving limited maintenance and all new features are being added to the ceph-osd or ceph-mon charms. Unfortunately, there currently exists no way of migrating existing deployments using the legacy architecture to the preferred architecture.

Problem Description

Deploying the all-in-one ceph charm was difficult to deploy properly and often resulted in too many monitors deployed. This (and other) problems eventually lead to splitting up the all-in-one ceph deployment into its different compositional parts - ceph-mon and ceph-osd. New deployments are encouraged to use the new architecture, but users which have deployed the previous architecture have no way to migrate to the new architecture.

In order to migrate to the new architecture, users will need to:

Deploy the ceph-mon charm into the environment without bootstrapping a new monitor cluster.
Relate the ceph application to the new ceph-mon application. The relation between these two applications is defined in the new ceph-bootstrap interface, documented herein.
Update the deployments using the ceph-client relation to point to the new ceph-mon application. This should include:
- ceph-radosgw
- nova-compute
- cinder-ceph
- cinder
- cinder-backup
- glance
Deploy the ceph-osd charm alongside the current ceph charms. The ceph-osd charm will not reformat any of the currently running OSD devices.
Remove the all-in-one ceph application.

Note: the configuration values will not be validated in any part of this migration scenario. If the user deploys the ceph-osd application or the ceph-mon application with different config values than that of the all-in-one ceph charms, then there may be unexpected changes to the environment.

Proposed Change

Most of the basic plumbing already exists to support the basic parts of this migration. There are some parts that are actually missing an need attention to detail.

Config Changes to charm-ceph-mon

A new config option will be added to the ceph-mon charm indicating that the charm should not do an initial bootstrap of the monitor cluster. When this value is set to true, the charm will not:

bootstrap a new monitor cluster
automatically generate an fsid

This new configuration option will be defined as follows:

no-bootstrap:
  type: boolean
  default: False
  description: |
    Causes the charm to not do any of the initial bootstrapping of the
    Ceph monitor cluster. This is only intended to be used when migrating
    from the ceph all-in-one charm to a ceph-mon / ceph-osd deployment.
    Refer to the Charm Deployment guide at https://docs.openstack.org/charm-deployment-guide/latest/
    for more information.

New ceph-bootstrap Interface

Notably, there exists no way of adding a relation between the ceph-mon charm and the all-in-one ceph charm. To address this, a new interface will be added to the ceph and ceph-mon charms called ceph-bootstrap, enabling the necessary information for joining the cluster to be shared. This is essentially the same information that’s shared on the peer mon interface for the ceph charm itself.

Since the charms are not reactive, no new interface repository is required. The information exchanged will contain the following:

ceph-bootstrap:
  - name: fsid
    type: string
    desc: |
      The fsid of the already bootstrapped monitor cluster

  - name: ceph-public-address
    type: string
    desc: |
      The public address that should be used by the charm for each of the
      units in the relation.

  - name: mon-key
    type: string
    desc: |
      The key used to authorize a monitor node for joining a ceph mon
      cluster.

In order to join this relation, the ceph-mon charm will require that the no-bootstrap config option be set to True and the monitor cluster is not already bootstrapped locally. If either of these is not valid, the ceph-mon charm will fail to join the relation.

Changes to charm-ceph

In addition to implementing the ceph-bootstrap interface, the all-in-one ceph charm needs to properly clean up when removing itself. It should not remove any OSDs or Ceph packages as this would interrupt the operational ceph cluster. However, the charm needs to remove its ceph.conf file as a registered alternative during its stop hook.

Alternatives

One alternative would be to not offer a migration to the new architecture and leave the deployments stuck. This has rather unfortunate side effect with respect to user happiness and has implications regarding the overall lifetime of the ceph charm.

Implementation

Assignee(s)

Primary assignee:: james-page
Additional assignee(s):: billy-olsen

Gerrit Topic

Use Gerrit topic charm-ceph-migration for all patches related to this spec.

git-review -t charm-ceph-migration

Work Items

charm-ceph-mon

Add new config flag
Implement new bootstrap interface
Remove additional monmap entries when removing the bootstrap relation

charm-ceph-osd

Ensure that the charm supports multiple monitor relations

charm-ceph

Add support for new bootstrap interface
Remove registered alternative for ceph.conf in the stop hook
Update charm README

charm-ceph-radosgw

Ensure that the charm supports multiple monitor relations

charm-helpers

Ensure that CephContext supports multiple monitor relations (consumed by charm-glance, charm-inder-ceph, charm-nova, etc)

openstack-charm-testing

Update all next.yaml and stable.yaml bundles to use split architecture
Add legacy bundle which deploys an all-in-one ceph bundle

docs

Update charm deployment guide to clearly describe this process
Update release notes with implementation note and reference to the deployment guide.

Repositories

None

Documentation

This will need to be carefully documented in the following places:

Charm Deployment Guide
The Ceph Charm’s README needs to refer to the migration process documentation and officially marked as deprecated.

Security

No new security implications or this change.

Testing

Create an appropriate functional test that can be run at release gating (mojo, bundle tester, etc)
Test the impact of running clients on the migration. This is important because the monitor information is exposed via the libvirt domain XML files which are created and may have an impact on running clients. Specifically, need to verify the impact to: - nova-compute w/ rbd backed instances - nova-compute w/ rbd cinder attached volumes - glance images w/ rbd backing

Dependencies

None

Ceph Storage Action

Wed, 21 Mar 2018 00:00:00

We should allow the user to specify the class of storage that a given storage device should be added to. This action would allow adding a list of osd-devices into specified Ceph buckets.

Problem Description

Proposed Change

Alternatives

User could specify a pre-build usage profile and we could map device types onto that profile by detecting the device type and deciding, based on the configured profile, what bucket the device should go into.

This solution is being discarded because of the difficulty of designing and implementing usage profiles to match any use case automigically. It will also require a lot of work to correctly identify the device type and decide where to place it in the desired profile.

In addition, it would require that we only support a single “profile” within a deployed ceph-osd cluster.
Charm could define additional storage attach points in addition to osd-devices that would allow the user to specify what bucket they should add devices to.

The reason for discarding this solution is that it was determined to be too limiting because of only supporting a fixed number of bindings, in addition to making changes harder because of backwards compatibility requirements.

Implementation

Assignee(s)

Primary assignee:: Chris MacNaughton <chris.macnaughton@canonical.com>
Contact:: Chris Holcombe <chris.holcombe@canonical.com>

Gerrit Topic

Use Gerrit topic “storage-action” for all patches related to this spec.

git-review -t storage-action

Work Items

Create a bucket with the required name. We do this so that we can create pool in the specified bucket type. This would be handled within the ceph-mon charm.
Create an action that returns a list of unused storage devices along with their device types.
Create an action that takes osd-devices and storage-type, that would be an enum into the buckets that we create. Rather than be a user specified string, this would be an enumerated list in the ceph charms provided through the shared charms_ceph library.
Add the ability for other charms to request the bucket that should back their created pools when creating pools through the broker.

Documentation

This will require additional documentation be added around how to use the action correctly and what to expect from it. This documentation will be added to the charm README

Security

This should have no security impact.

Testing

There will need to be new unit tests and functional tests to ensure that the necessary buckets are created and that the disks are added to them.

Repositories

None

Dependencies

None

Service Discovery

Wed, 21 Mar 2018 00:00:00

Many optional services may now be deployed as part of an OpenStack Cloud, with each service having a different optional features that may or may not be enabled as part of a deployment.

Charms need a way to discover this information so that services can be correctly configured for the options chosen by the charm user.

Problem Description

Charms need to be able to determine what other services are deployed within an OpenStack Cloud so that features can be enabled/disabled as appropriate.

Examples include:

notifications for ceilometer (really don’t want notifications enabled when ceilometer is not deployed).
misc panels within the openstack dashboard (fwaas, lbaas, l3ha, dvr etc… for neutron).
notifications for designate (disable when designate is not deployed).

Services and features of services are determined by the API endpoint charms that register them into the service catalog via the keystone charm.

Proposed Change

The keystone charm will expose a new provides interface ‘cloud-services’ which is a rich-ish description of the services deployed with registered endpoints.

By default, a registered endpoint of type ‘A’ will result in service type ‘A’ being listed as part of the deployed cloud on this interface.

Services may also enrich this data by providing ‘features’ (optional) alongside their endpoint registration - these will be exposed on the cloud-service and identity-service relations.

Data will look something like (populated with real examples - key and proposed values):

services: ['object-store', 'network', 'volumev2', 'compute',
           'metering', 'image', 'orchestration', 'dns']

Example - advertise features supported by the networking service, allowing features to be enabled automatically in the dashboard:

network: ['dvr', 'fwaas', 'lbaasv2', 'l3ha']

Example - allow ceilometer to know that the deployed object-store is radosgw rather than swift:

object-store: ['radosgw']

Values will be parseable as a json/yaml formatted list.

By using the basic primitive of tags, we get alot of flexibility with type/feature being easily express-able.

Interface will be eventually consistent in clustered deployments - all keystone units will present the same data.

Alternatives

Implementation

Assignee(s)

Primary assignee:: james-page

Gerrit Topic

Use Gerrit topic “service-discovery” for all patches related to this spec.

git-review -t service-discovery

Work Items

Core (keystone charm):

Add cloud-services relation to keystone charm
Add service and feature discover handler to keystone charm
Update keystone interface to advertise services and features in keystone charm.
Create cloud-services reactive interface

Enablement:

Update ceilometer charm for radosgw discovery.
Update openstack-dashboard charm to automatically enable panels for deployed services and features.
Update neutron-api charm for designate discovery.
Update cinder charm for ceilometer discovery.
Update glance charm for ceilometer discovery.
Update neutron-api charm for ceilometer discovery.
Update radosgw charm to advertise ‘radosgw’ feature.
Update neutron-api charm to advertise networking features.

Repositories

No new git repositories required.

Documentation

This change is internal for use across the OpenStack charms, no documentation updates are required for end-users.

Security

No security implications for this change.

Testing

Implementation will include unit tests for all new code written; amulet function tests will be updated to ensure that feature is being implemented correctly across the charm set.

Dependencies

No external dependencies

Internal DNS Resolution

Thu, 07 Sep 2017 00:00:00

Users of an OpenStack cloud would like to look up their instances by name in an intuitive way using the Domain Name System (DNS). When booting an instance using Nova, a name is provided which is then used as the hostname for the system. Many programs in the booted instance utilize DNS to resolve the hostname and function less than ideally when the hostname cannot be resolved.

Problem Description

The lack of sane internal DNS resolution has plagued users of OpenStack deployed instances since before the ice(-house) age. Neutron has long provided DNS services for tenant networks, but the default names provided by Neutron are defined as ‘host-%(octet0)s-%(octet1)s-%(octet2)s-%(octet3)s.openstacklocal’. Since this name does not match the instance name/hostname, the DNS queries fail when attempting to resolve the hostname.

Some examples where this becomes problematic for tenants are as follows:

Sudo attempts to resolve the hostname each time it runs. Sudo continues to work, but complains about the failed name resolution:
```
ubuntu@john-oliver:~$ sudo -s
sudo: unable to resolve host john-oliver
```
Users are unable to logically refer to other instances by name in normal network operations as they would in a bare metal lab environment:
```
ubuntu@jon-stewart:~$ ping john-oliver
ping: unknown host john-oliver
```
Applications such as rabbitmq-server require a resolvable hostname in order to function properly.

OpenStack Neutron and Nova projects have previously addressed this in the Liberty and Mitaka releases respectively and the charms should enable this functionality for the end users.

Proposed Change

The Liberty release of Neutron introduced the DNS extension driver [1]. This extension driver, when enabled, adds the dns_domain attribute to the Neutron Network API and added the dns_name and dns_fqdn (read-only) attributes to the Neutron Port API. These new attributes are used by the Neutron dhcp-agent when creating the host file for the dnsmasq process.

In the subsequent Mitaka release, the Nova project leveraged the newly minted DNS extension in Neutron to supply DNS-sanitized versions of the instance name [2]. When booting new instances, Nova will provide the dns_name attribute for ports it creates via the Neutron API.

OpenStack documentation [3] in the Mitaka release provides details for how to enable this DNS integration between the two projects. The configuration pieces entirely belong to the Neutron project as Nova will Just Do the Right Thing (TM).

This spec proposes to enable internal DNS resolution for deployed clouds and does not attempt to cover integration with an external DNS service such as Designate.

It is important to note that the internal DNS resolution provided by this feature is network isolated rather than tenant isolated. As such, instances launched in network A will not be able to resolve instances launched in network B. For this type of name resolution, an external DNS service such as those provided by Designate should be used instead and is beyond the scope of this spec.

To enable the internal DNS resolution, the neutron-server service must be configured with the dns_domain config option set to a value other than openstacklocal in neutron.conf file and the dns extension driver must be enabled in the ml2_conf.ini file.

This change will add a two new config options as defined below:

enable-ml2-dns:
  type: boolean
  default: False
  description: |
    Enables the Neutron DNS extension driver. When enabled, ports attached
    to Nova instances will have DNS names assigned based on the instance
    name.
dns-domain:
  type: string
  default: openstack.example.
  description: |
    Specifies the dns domain name that should be used for building instance
    hostnames. An empty option or the value of 'openstacklocal' will cause
    the dhcp agents to broadcast the default domain of openstacklocal and
    will not enable internal cloud dns resolution. This value should end
    with a '.', e.g. 'cloud.example.org.'.

In order to enable internal DNS resolution, the user must set the enable-ml2-dns to True. The default value is False in order to provide backwards compatibility with existing deployments.

DNS Forwarding Servers

The dns-domain alone is not enough to provide all the necessary configuration options for the neutron networking. In most instances, the administrator will need to be able to specify a dns fowarding server as well. In order to do this, a new config option will be provided allowing the user to set configure the nameservers to use as forwarding servers.

Per [4] there are three ways of configuring DNS nameservers for instances launched in the cloud. Tenant subnets can have their own nameservers identified and requires ano additional work in order to enable that. Default nameserver information is provided by the DHCP agents to point to the dhcp port address but contains no additional forwarding servers. By default, this only allows instances to be able to resolve other instances in the subnet. To amend this, the neutron-openvswitch and neutron-gateway charms will be amended to allow the user to specify the DNS forwarding servers. The charms will not include any options to allow the use of the DNS resolvers configured on the DHCP agent’s host (the dnsmasq_local_resolv option) as it poses a risk of leaking internal infrastructure level resources to the instances.

As such, the neutron-openvswitch and neutron-gateway charms will add an option dns-servers, which will configure the dnsmasq_dns_servers option in the dhcp_agent.ini file. This option is defined as follows:

dns-servers:
  type: string
  default:
  description: |
    A comma-separated list of DNS servers which will be used by dnsmasq as
    forwarders.

The dns-servers option will only apply for the neutron-openvswitch charm when the enable-local-dhcp-and-metadata option is set to True.

Relation Implications

The neutron-dhcp-agent does not run on the same node as the neutron-server and should have the dns_domain specified in the dhcp_conf.ini file. Specifying this value in the dhcp_conf.ini file is not strictly necessary, as the hostnames will properly resolve without it. However, the default search list advertised to hosts will be the default openstacklocal and may cause confusion to users. To allow the neutron-api charm to share this configuration information with interested parties, the neutron-plugin-api relation-data will be updated to contain the dns-domain name:

'dns-domain': 'domain.tld.'

The value will be a string containing the domain name.

Alternatives

Designate could be setup to provide the DNS service entries for the tenant. This option is valid, but requires additional components to be setup and deployed into the environment. Additionally, there are some limitations which are not well documented in the upstream documentation for configuring DNS integration. For example, the Neutron port API will not call the Designate API for ports on tunnelled tenant networks (e.g. GRE).
An out-of-band solution such as that provided by the serverstack-dns tool could be deployed to provide DNS based upon OpenStack events. This is less than desirable as it must be installed per tenant, each tenant must have access credentials to access the resources of the underlying cloud, and the tool itself is not intended for a production environment.

Implementation

Assignee(s)

Primary assignee:: billy-olsen

Gerrit Topic

Use Gerrit topic “charms-internal-dns” for all patches related to this spec.

git-review -t charms-internal-dns

Work Items

charm-neutron-api

Add new config option to the neutron api charm
Add dns-domain to the neutron-plugin-api interface
Update README.md to reflect new behavior

charm-neutron-gateway

Update neutron-gatway to consume dns-domain from relation data
Add dns-servers config option to charm

charm-neutron-openvswitch

Update neutron-openvswitch charm to consume dns-domain from relation data
Add dns-servers config option to charm

Repositories

No new git repositories required.

Supported Releases

This feature will be available on deployed clouds running Mitaka or newer. Attempting to enable this feature on earlier versions will have no effect.

Documentation

The neutron-api charm README will need to be updated to reflect the new feature and how to enable internal DNS.

Security

No security implications for this change.

Testing

Implementation will include unit tests for all new code written; amulet function tests will be updated to ensure that feature is being implemented correctly across the charm set.

Dependencies

No external dependencies.

Ceph Broker CephX Support

Wed, 08 Mar 2017 00:00:00

Problem Description

Currently the ceph/ceph-mon charm provides cephx keys to clients which have rw permissions to all pools in the ceph cluster; this is problematic because it means that any client can read/write/delete data in any pool so in the event of a compromise of a service (which might be directly accessible by end users such as cinder, glance or the ceph-radosgw), the entire cluster is compromised until the compromised key is revoked.

Proposed Change

Ceph supports fine grained access control on keys, so we need to leverage this functionality to improve the general security of a deployment, for example (taken directly from the Ceph documentation):

ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'

The ceph/ceph-mon will provide new broker methods to allow

Client requested pools to be placed into ‘groups’ at point of creation:

For example, multiple cinder backend pools would be placed into the ‘volumes’ group.
Clients can also request access to a group of pools:

For example, the cinder charm will gain ‘rwx’ for pools in the ‘volumes’ group, and will request ‘rx’ permission for pools in the ‘images’ group.
Clients can optional request additional permissions:

This supports the ‘allow class-read object_prefix rbd_children’ use-case where a key needs to be able to read from copy-on-write clones in different pools.

Alternatives

Security could be left open and secured post deployment by the operator, but this is neither repeatable or desirable from an operations perspective.

Implementation

Assignee(s)

Primary assignee:: xfactor973 (Chris Holcombe)

Gerrit Topic

Use Gerrit topic “cephx-keys” for all patches related to this spec.

git-review -t cephx-keys

Work Items

Implementation of group->pool mapping and maintenance in ceph-broker
Implementation of cephx key->group ACL mapping in ceph-broker
Implementation of methods on relation API to support mapping pools to groups.
Implementation of methods on broker relation API to support granting access to pools to cephx keys with correct permissions.
Implementation of cephx key update mechanism when membership of a pool group changes.
Updates to cinder, glance, nova-compute and ceph-radosgw charms to make appropriate pool access requests to the ceph/ceph-mon charm.

Repositories

No new repositories are required for this work.

Documentation

As the broker relation interface is self documenting, no additional documentation is required.

Security

This work mitigates an existing security risk.

Testing

The existing charm functional tests will automatically cover this work as they should be exercising the functionality of services consuming ceph resources; incorrect key permissions will result in broken function.

Unit tests should be written to validate the key and group management functions in charms.ceph; additionally the ceph/ceph-mon charm function tests should be updated to verify the permissions on created keys in the current test cases.

Dependencies

No external dependencies for this work.

Ceph Autotune

Mon, 17 Oct 2016 00:00:00

Ceph isn’t as fast as it could be out of the box.

Problem Description

Currently we don’t do any tuning of Ceph after installing it. There’s many little things that have to be tweaked to get ceph to be performant.

There are many mailing list messages of users becoming frustrated with the slow performance of Ceph. The goal of this epic is to capture the low hanging fruit so everyone can have a better experience with Ceph.

Proposed Change

I’m proposing several areas where easy improvements can be made.

There’s several areas that can be improved such as HDD read ahead and max hardware sector settings. We can also tune the sysctl network settings on 10 and 40Gb networks.

Finally I’ve noticed that IRQ alignment on multisocket motherboard isn’t ideal. We can use numactl to pin the IRQs for all the components in the IO path from network to osd to disk.

Of all the changes probably the IRQ pinning will be the most difficult.

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: cholcombe973

Gerrit Topic

Use Gerrit topic “performance-optimizations” for all patches related to this spec.

git-review -t performance-optimizations

Work Items

HDD Read ahead settings: Often times the read ahead settings for hdd’s are too small for storage servers.
10/40Gb sysctl settings: 10Gb and 40Gb networks need special sysctl settings to improve their performance.
HDD max hardware sectors: Making the max hardware sectors as large as possible allows bigger sequential writes and therefore better performance.
IRQ alignment for multisocket motherboards: It has been shown on multi socket motherboards that Linux does a bad job of lining up IRQ’s and results in lots of swapping of cache memory between CPU’s. This significantly degrades Ceph’s performance.

Repositories

No new git repositories required for this work.

Documentation

Updates to the README’s in the impacted charms will be made as part of this change.

Security

No additional security concerns.

Testing

Code changes will be covered by unit tests; functional testing will be done using a Mojo specification.

Dependencies

charm-layering

Team and repository tags

Thu, 26 May 2016 00:00:00

OpenStack Charm Specifications

This git repository is used to hold approved design specifications for additions to the Charm project. Reviews of the specs are done in gerrit, using a similar workflow to how we review and merge changes to the code itself. For specific policies around specification review, refer to the end of this document.

The layout of this repository is:

specs/<release>/

Where there are two sub-directories:

specs/<release>/approved: specifications approved but not yet implemented specs/<release>/implemented: implemented specifications

Example specifications

You can find an example spec in specs/template.rst.