Cinder Specs

Standardize Image Encryption and Decryption

Fri, 24 Oct 2025 00:00:00

OpenStack already supports encrypted volumes and ephemeral storage to ensure the confidentiality of block data. Even though it is already possible to store encrypted images, there is only one service (Cinder) that utilizes this option, but it is only indirectly usable by Nova (a user must create a volume from the image first), and thus users do not have an intuitive way to create and upload encrypted images. In addition, all metadata needed to detect and use encrypted images is either not present or specifically scoped for Cinder right now. In conclusion, support for encrypted images does exist to some extent but only in a non-explicit and non-standardized way. To establish a consistent approach to image encryption for all OpenStack services as well as users, several adjustments need to be implemented in Glance, Cinder, and OSC.

Problem description

An image, when uploaded to Glance or created through Nova from an existing server (VM), may contain sensitive information. The already provided signature functionality only protects images against alteration. Images may be stored on several hosts over long periods of time. First and foremost this includes the image storage hosts of Glance itself. Furthermore it might also involve caches on systems like compute hosts. In conclusion, the images are exposed to a multitude of potential scenarios involving different hosts with different access patterns and attack surfaces. The OpenStack components involved in those scenarios do not protect the confidentiality of image data.

As stated in the introduction above, some disk image encryption implementations for ephemeral disks in Nova and volumes in Cinder already touch on this topic but not always in a standardized and interoperable way. For example, the way of handling image metadata and encryption keys can differ. Furthermore, users are not easily able to make use of these implementations when supplying their own images in a way that encryption can work the same across services.

That’s why we propose the introduction of a streamlined encrypted image format along with well-defined metadata specifications which will be supported across OpenStack services for the existing encryption implementations and increase interoperability as well as usability for users. Additionally we require, that encrypted images will always result in encrypted volumes to avoid decryption.

Use Cases

1. A user wants to create a new volume based on an encrypted image. The corresponding volume host has to be enabled to detect that the image is encrypted.

If an encrypted image is the base for a new volume the used volume type should always have an encryption type. If the given volume type or default volume type does not have an encryption type the operation should result in an ERROR.

There are two possible types of encrypted images: qcow2 images with encryption (qcow2+LUKS) and raw LUKS images. Cinder can already handle raw LUKS-encrypted images. Encrypted qcow2+LUKS images may need to be converted to raw LUKS before transforming them into volumes.

2. Whenever an encrypted image is converted to an encrypted volume the secret should be copied to give Cinder full control over the lifecycle of the secret.

There is only one secret per image and it can be either a key or a passphrase. The secret type classification in the Key Manager will determine the key handling (“symmetric” vs “passphrase”). In case of “passphrase”, the secret is passed directly as the passphrase to the encryption layer. In any othercase, it is interpreted as binary and converted to a suitable string representation before being passed to the encryption. Currently, Cinder is only able to handle “symmetric” keys, i.e., the latter binary case. Support for “passphrase” (like used in Nova) has to be added.

3. A user wants to create an image from a volume with an encrypted volume type The target image will reuse the LUKS encryption and key. This use case is already implemented as part of the default behavior of Cinder.

Creating an encrypted image from an unencrypted volume will not be part of this spec, but may be implemented later on.

Creating an unencrypted image from an encrypted volume is not possible right now. The volume encryption is transparently also used for the image. This behavior will stay in place to optimize resource usage and avoid costly conversion of the whole volume data on volume hosts.

Proposed change

In Glance we propose the following additional metadata properties that should be carried by encrypted images:

‘os_encrypt_format’ - the specific mechanism used, e.g. ‘LUKSv1’
‘os_encrypt_key_id’ - reference to key in the key manager
‘os_encrypt_key_deletion_policy’ - on image deletion indicates whether the key should be deleted too
‘os_decrypt_size’ - size after payload decryption

We propose to align the encryption with Nova and Cinder and use LUKS, which will be allowed in combination with qcow2 and raw images. We use these two variations for the following reasons:

Nova might be able to directly use qcow2+LUKS encrypted images when creating a server. The qcow2 format is already familiar to Nova and used by it to directly boot instances from. Currently, support for encrypted ephemeral storage is not yet implemented in Nova, so qcow2+LUKS cannot be used by it yet but since the LUKS encryption is a native feature of qcow2, this will provide a good starting point for a lightweight future extension.
Cinder allows the creation of images from encrypted volumes. These will always result in LUKS-encrypted raw images. Those images can be converted directly to volumes again. This behavior is already implemented and requires no format conversion as the LUKS encryption is native to Cinder. The intention is to keep this functionality and make the format usable outside of Cinder and provide interoperability for it. To better identify such an image, we propose the new ‘container_format’ ‘luks’ to be set for these images.

In the latter case it is already possible to upload such an encrypted image to another OpenStack deployment, upload the key as well and set the corresponding metadata. After doing so the image can be used in the second deployment to create an encrypted volume.

We want to align the existing implementations between Nova and Cinder by standardizing the used metadata parameters and adding interoperability where applicable. This would in the case of Cinder mainly be a change in the naming of the following image properties:

‘cinder_encryption_key_deletion_policy’ to ‘os_encrypt_key_deletion_policy’
‘cinder_encryption_key_id’ to ‘os_encrypt_key_id’

A check in the volume creation flow will be added to look for encrypted images proposed as a volume source. If an image is encrypted, another check is added to determine, whether the volume type used to create the volume has an encryption type. If that is not the case the volume creation will be aborted early in the API. Since the encrypted data is always directly transferred over, the volume would end up as unusable otherwise.

The conversion of a qcow2+LUKS image should be handled when downloading the image to a volume. The required volume size should be determined based on the ‘os_decrypt_size’ property of the image.

The key management for creating an encrypted volume from an encrypted image must include the cloning of the secret in Barbican. This way Cinder always has the full control over the lifecycle of the secret and is similar to the original creation of an encrypted volume. As secrets used for encrypted images may be user-defined, Cinder cannot rely on the existence of the original secret over time.

In all places that implement image decryption within Cinder, an additional check for the type of the secret needs to be added. Different key handling needs to be triggered if the secret is a “passphrase”, because the way Cinder currently treats keys to create a passphrase for the LUKS header of a volume in way that always attempts to convert binary keys to an ASCII encoding first and differs from Nova’s handling of images, that directly passes passphrases only.

The creation of an image from a volume just needs to be adjusted to use the new properties.

Alternatives

We also evaluated an image encryption implementation based on GPG. The downside with such an implementation is, that every time such an image is used to create a server or a volume the image has to be decrypted and maybe re-encrypted for another encryption format as both Nova and Cinder use LUKS as an encryption mechanism. This would not only have an impact on the performance of the operation but it also would need free space for the encrypted image file, the decrypted parts and the encrypted volume or server that is created.

Data model impact

None

REST API impact

When creating a volume from an encrypted image there might occur a new ERROR that is triggered when an image is encrypted but no encrypted volume type is given.

Security impact

There are impacts on the security of OpenStack:

confidentiality of data in images will be addressed in this spec
image encryption is introduced formally
cryptographic algorithms will be used in all involved components (Nova, Cinder, OSC) that work with encrypted images

Active/Active HA impact

None

Notifications impact

None

Other end user impact

Users should be able to use encrypted images to create volumes in a consistent way

Performance Impact

The proposed checks for the Cinder API may have minimal impact on performance.

When creating a volume or server from an encrypted image the only operation that may be triggered is the conversion between qcow2+LUKS and raw LUKS blocks.

Thus, any performance impact is only applicable to the newly introduced encrypted image type where the processing of the image will have increased computational costs and longer processing times than regular images. Impact will vary depending on the individual host performance and supported CPU extensions for cipher algorithms.

Other deployer impact

For interoperability between the OpenStack services only the presence of a key manager should decide, whether encryption can be used or not.
A key manager - like Barbican - is required, if encrypted images are to be used.

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Markus Hentsch (IRC: mhen)

Other contributors: Josephine Seifert (IRC: Luzi)

Work Items

Add checks in the create volume API
Add cloning the secret for image encryption keys
Add dedicated handling of “passphrase” type secrets via os-brick
Add conversion of qcow2+LUKS images to raw LUKS encrypted block data
In the image creation from volume: change the ‘cinder_encryption_key_deletion_policy’ to ‘os_encrypt_key_deletion_policy’ and ‘cinder_encryption_key_id’ to ‘os_encrypt_key_id’
Add setting the ‘os_encrypt_format’ property when creating images from encrypted volumes

Dependencies

Handling of the new image encryption parameters and secret consumer usage in Glance has to be implemented
os-brick implements a shared utility function for converting encryption keys of different types, including ‘passphrase’ and ‘symmetric’

Testing

Tempest tests will be added to the barbican-tempest-plugin in addition to its existing scenario tests revolving around usage of secrets. These scenario tests will create encrypted images in various permutations, including the different image formats (qcow2+LUKS, raw LUKS) as well as the different secret types influencing the key conversion. Each of the created images will be used to create a volume from which a Nova instance will be booted and health-checked.

Another scenario will be added that specifically tests the creation of encrypted images based on an encrypted volume by Cinder and the subsequent use for a new encrypted volume to verify the correct production and consumption of encrypted images as implemented by Cinder itself. This will act as a regression test to make sure that the existing functionality of Cinder being able to backup and restore encrypted volumes to/from images while keeping the encryption intact through the whole chain stays functional.

Documentation Impact

It should be documented for deployers, how to enable this feature in the OpenStack configuration.

References

[1] Barbican Secret Consumer Spec: https://review.opendev.org/#/c/662013/

[2] Glance Image Encryption Spec: https://review.opendev.org/c/openstack/glance-specs/+/964755

[3] Extended key to passphrase conversion outsourced to os-brick: https://review.opendev.org/c/openstack/os-brick/+/926293

OpenAPI Schemas

Tue, 14 Oct 2025 00:00:00

https://blueprints.launchpad.net/cinder/+spec/openapi

We would like to start documenting our APIs in an industry-standard, machine-readable manner. Doing so opens up many opportunities for both OpenStack developer and OpenStack users alike, notably the ability to both auto-generate and auto-validate both client tooling and documentation alike. Of the many API description languages available, OpenAPI (fka “Swagger”) appears to be the one with both the largest developer mind share and the one that would be the best fit for OpenStack due to the existing tooling used in many OpenStack services, thus we would opt to use this format.

Problem description

The history of API description languages has been mainly a history of half-baked ideas, unnecessary complication, and in general lots of failure. This history has been reflected in OpenStack’s own history of attempting to document APIs, starting with our early use of WADL through to our experiments with Swagger 2.0 and RAML, leading to today’s use of our custom os_api_ref project, built on reStructuredText and Sphinx.

It is only in recent years that things have started to stabilise somewhat, with the development of widely used API description languages like OpenAPI, RAML and API Blueprint, as well as supporting SaaS tools such as Postman and Apigee. OpenAPI in particular has seen broad adoption across multiple sectors, with sites as varied as CloudFlare and GitHub providing OpenAPI schemas for their APIs. OpenAPI has evolved significantly in recent years and now supports a wide variety of API patterns including things like webhooks. Even more beneficial for OpenStack, OpenAPI 3.1 is a full superset of JSON Schema meaning we have the ability to re-use much of the validation we already have.

Use Cases

As an end user, I would like to have access to machine-readable, fully validated documentation for the APIs I will be interacting with.

As an end user, I want statically viewable documentation hosted as part of the existing docs site without requiring a running instance of Cinder.

As an SDK/client developer, I would like to be able to auto-generate bindings and clients, promoting consistency and minimising the amount of manual work needed to develop and maintain these.

As a Cinder developer, I would like to have a verified API specification that I can use should I need to replace the web framework/libraries we use in the event they are no longer maintained.

Proposed change

This effort can be broken into a number of distinct steps:

Add missing request body and query string schemas

These schemas will merely validate what is already allowed, which means extensive use of "additionalProperties": true or empty schemas.

Once these are added, tests will be added to ensure all API resource have appropriate schemas.
Add response body schemas

These will be sourced from both existing OpenAPI schemas, currently published at opendev.org/openstack/codegenerator, and from new schemas auto-generated from JSON response bodies generated in tests and manually modified handle things like enum values.

Once these are added, tests will be added to ensure all API resource have appropriate response body schemas. In addition, we will add a new configuration option that will control how we do verification at the API layer, [api] response_validation. This will be an enum value with three options:

error
Raise a HTTP 500 (Server Error) in the event that an API returns an “invalid” response.

This will be the default in CI i.e. for our unit, functional and integration tests. Eventually this could be the default for production also (but probably not).

warn
Log a warning about an “invalid” response, prompting operations to file a bug report against Cinder.

This will be initial (and likely forever) default in production.

ignore
Disable API response body validation entirely. This is an escape hatch in case we mess up.

Note

The development of tooling required to gather these JSON Schema schemas and generate an OpenAPI schema will not be developed inside Cinder and is therefore not covered by this spec. Cinder will merely consume the resulting tooling for use in documentation.

Alternatives

Use a different tool

OpenAPI has been chosen because it is the most widely used API description language available and matches well with our existing use of JSON Schema for API validation.
Maintain these specs out-of-tree

This prevents us testing these on each commit to Cinder and means work that could be spread across multiple teams is instead focused on one small team.

Data model impact

None.

REST API impact

There will be no direct REST API impact. Users will see HTTP 500 error if they set [api] response_validation = error and encounter an invalid response, however, we will not encourage use of this option in production and will instead focus on validating this ourselves in CI.

We may wish to address issues that are uncovered as we add schemas, but this work is considered secondary to this effort and can be tackled separately.

Security impact

None.

Active/Active HA impact

None.

Notifications impact

None.

Other end user impact

This should be very beneficial for users who are interested in developing client and bindings for OpenStack. In particular, this should (after an initial effort in code generation) reduce the workload of the SDK team as well as teams outside of OpenStack that work on client tooling such as the Gophercloud team.

Performance Impact

There will be a minimal impact on API performance when validation is enabled as we will now verify both requests and responses for all API resources. Given our existing extensive use of JSON Schema for API validation, it is expected that this should not be a significant issue. In addition, we will not recommend enabling this option in production.

Other deployer impact

As noted previously, there will be one new config option, [api] response_validation. Operators may see increased warnings in their logs due to incomplete schemas, but most if not all of these issues should be ironed out by our CI coverage.

Developer impact

Developers working on the API microversions will now be encouraged to provide JSON Schema schemas for both requests and responses.

Implementation

Assignee(s)

Primary assignee:: stephenfinucane
Other contributors:: gtema

Work Items

Add missing request body schemas
Add tests to validate existence of request body schemas
Add missing query string schemas
Add tests to validate existence of query string schemas
Add response body schemas
Add decorator to validate response body schemas against response
Add tests to validate existence of response body schemas
Update doc/source/contributor/api_microversion_dev.rst to include the requirement that anyone contributing a change that requires a new microversion is expected to provide request and response schemas

Dependencies

The actual generation of an OpenAPI documentation will be achieved via a separate tool. It is not yet determined if this tool will live inside an existing project, such as os_api_ref or openstacksdk, or inside a wholly new project. In any case, it is envisaged that this tool will handle OpenStack-specific nuances like microversions that don’t map 1:1 to OpenAPI concepts in a consistent and documented fashion.

Testing

Unit tests will ensure that schemas eventually exist for request bodies, query strings, and response bodies.

Unit, functional and integration tests will all work together to ensure that response body schemas match real responses by setting [api] response_validation to error.

Documentation Impact

Initially there should be no impact as we will continue to use os_api_ref as-is for our api-ref docs. Eventually we will replace or extend this extension to generate documentation from our OpenAPI schema.

References

None.

History

Revisions
Release Name	Description
2025.01	Introduced
2026.01	Re-proposed

No specs have yet been approved.

Fri, 26 Sep 2025 00:00:00

Restrict new container creation for Cinder Backup

Fri, 29 Aug 2025 00:00:00

The URL of launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/restrict-backup-container-creation

This spec is designed to allow operators to restrict new container creation by users during backup creation, when a non-existent container is passed as container argument with a POST /v3/{project_id}/backups request.

Problem description

While creating Cinder backups, users are able to supply an arbitrary container name, where the backup will be stored. While this behavior makes total sense with the Swift driver and backup_swift_auth=per_user, as this will utilize tenant quota and data will be accounted properly, using arbitrary containers might be unwanted by operators while using other drivers.

Current behavior may cause various negative side-effects when users use arbitrary container names, for example: being unable to enforce storage policies, going out of designed quota for backups, complications for accounting of consumed disk space and billing, etc.

At the moment container creation is supported by the following drivers:

Swift
S3
GCS
POSIX

Use Cases

For instance, with the S3/GCS driver, as an operator, I want to pre-create a series of buckets with different policies like object locking with various retention rules or different storage classes. While current behavior might work, ability for users to create new buckets will result in an approach that is not error-prone, as due to typos backups may end in unexpected locations.

With the POSIX driver as example, as operator I want to have different mounts and underlying storages per “container”. With current behavior this approach is impossible as users may bypass the defined mount boundaries by creating a new path with an arbitrary container name.

Proposed change

It is proposed to introduce a new configuration option backup_create_containers, which will control whether Cinder Backup will attempt to create a new container when a user has supplied a non-existing container name in request or fail such request.

When this option is set to False, duty of creating all containers, including the default one, falls under operator responsibility. This means that all containers, including the default one, must be pre-created before any backup can be done.

Alternatives

Alternative approach would be to introduce a separate policy, which may allow users with required privileges to create containers with backups. However, implementation of this approach is more complex while not providing much more benefits for the use case, given that operators are usually not executing backup creation on their own.

Data model impact

No data model impact

REST API impact

No direct impact on REST API.

POST requests to /v3/{project_id}/backups supplying container parameter that is not found on backends with backup_create_containers=False will result in backup creation failures, with the corresponding reason being recorded in fail_reason column of backups table.

Security impact

This change should positively impact overall security of Cinder Backup, as it allows restricting potentially too open possibilities of container creation, which depending on deployment design and driver may lead to unexpected behavior or even denial of service.

Active/Active HA impact

Not applicable for Cinder Backup service

Notifications impact

No impact

Other end user impact

When backup_create_containers=False users will be able to create backups only in a predefined set of container names provided by the cloud provider and will not be able to use arbitrary container names.

A disadvantage of the approach, is that check for container existence is performed asynchronously by cinder-backup service, so user will not receive an error on issuing API request, but will see a 404 error as a reason for backup creation failure instead. The error will be shown in backup failure reason and not in the API response to the request, as API does not wait for RPC messaging with the backup service to happen.

There is also no discoverability mechanism if this property is enabled or not, so user is not able to know in advance if backup_create_containers is enabled or not. It is the cloud provider responsibility to inform users about cases, where backup_create_containers is set to False.

Performance Impact

No performance impact

Other deployer impact

Default value of backup_create_containers is proposed to be set to True, to preserve existing behavior. Therefore, no deployer impact is expected.

Developer impact

No impact

Implementation

Assignee(s)

Primary assignee:: Dmitriy Rabotyagov <noonedeadpunk>

Work Items

Introduce new configuration parameter for cinder-backup globally
Add support to S3 backup driver
Add support to GCS backup driver
Add support to Swift backup driver
Add support to POSIX backup driver

Dependencies

No dependencies

Testing

Unit testing of this option will be introduced, to ensure that cinder-backup will not call for new container creation to the backend, when backup_create_containers is set to False.

Documentation Impact

New option will be documented as part of the Cinder Backup configuration guide.

References

Reference implementation: https://review.opendev.org/c/openstack/cinder/+/959425

Show migration_progress in volume details

Mon, 16 Dec 2024 00:00:00

https://blueprints.launchpad.net/cinder/+spec/migration-progress-in-volume-details

This spec proposes adding a migration_progress field to the volume details API response when a volume is migrating.

Problem description

Volume migration might be facilitated by the backend driver, but often requires cinder or nova to copy data from the source to the new volume. For large volumes this can take a long time to complete (multiple hours, even days). Currently there’s no way to track the copy operation’s progress, which makes it difficult to predict when the migration will complete.

Use Cases

As a cloud admin who is migrating a volume, I need to know it isn’t taking so much time to complete that I fear the operation is stuck.
As a cloud admin who is migrating a volume, I would like to track the migration progress in order to gauge how much longer it might take for the operation to complete.

Proposed change

The API code for showing a volume’s details would use a microversion to optionally include a migration_progress field. The field would be included in the API response only when the volume is migrating, and when permitted by the microversion. As the existing migration_status field is admin-only, the new field would also be admin-only.

The value would be obtained via an RPC that fetches a response from either the volume service associated with the volume, or from nova when the volume is attached and nova is copying the data. See the references for the proposed nova feature. The RPC to the volume service or nova would be executed only when the following criteria are met:

The context is admin
The volume’s migration_status is “migrating”
The API request satisfies the new microversion

The volume service handling the RPC will determine how the volume is migrating in order to provide the proper response.

When a migration requires cinder to copy the data then the RPC response will be a string representing a percent completion (e.g. “42” for 42% complete). This is the same response that nova returns when it provides a swap_progress response when its swapping a volume attachment.
When a migration is handled by the volume’s driver then the RPC response will be a TBD string to indicate the migration is driver-assisted. The subset of cinder drivers that support driver-assisted migration do not expose an API for reporting migration progress, and the expectation is drivers complete the migration fast enough to provide a satisfactory user experience (i.e. the use cases are covered).
If the migration has completed by the time the RPC is received, the response will be None.

At the API layer, the migration_progress field will be included in the volume details response only when the RPC response is a non-empty string. For attached volumes, nova’s swap_progress will be reported as the volume’s migration_status.

Alternatives

An earlier idea was to enhance cinder so it tried to use “driver assisted volume migration” on attached volumes. The hope was that cinder drivers would be capable of migrating data fast enough to alleviate concerns about the process taking too long. This idea was rejected for multiple reasons:

Not all cinder drivers support driver-assisted migration, and most of them explicitly disallow migrating attached volumes.
Interactions between cinder and nova would be complicated. Cinder would need to tell nova to pause IO prior to migrating the volume, and then unpause after migration completes.

Data model impact

TBD, but hopefully none. It may be necessary to add a DB field to facilitate obtaining the proper RPC response. This will be carefully considered during the design phase, with a goal of minimizing impact on the DB.

REST API impact

Add a new microversion to the GET /v3/volumes/{volume_id} API to include an optional migration_progress field in the response when the migration_status field is migrating, and the API context is for an admin.

{
    "volume": {
        "attachments": [],
        "availability_zone": "nova",
        "bootable": "false",
        "consistencygroup_id": null,
        "created_at": "2018-11-29T06:50:07.770785",
        "description": null,
        "encrypted": false,
        "id": "f7223234-1afc-4d19-bfa3-d19deb6235ef",
        "links": [
            {
                "href": "http://127.0.0.1:45839/v3/89afd400-b646-4bbc-b12b-c0a4d63e5bd3/volumes/f7223234-1afc-4d19-bfa3-d19deb6235ef",
                "rel": "self"
            },
            {
                "href": "http://127.0.0.1:45839/89afd400-b646-4bbc-b12b-c0a4d63e5bd3/volumes/f7223234-1afc-4d19-bfa3-d19deb6235ef",
                "rel": "bookmark"
            }
        ],
        "metadata": {},
        "migration_status": "migrating",
        "multiattach": false,
        "name": null,
        "os-vol-host-attr:host": null,
        "os-vol-mig-status-attr:migstat": null,
        "os-vol-mig-status-attr:name_id": null,
        "os-vol-tenant-attr:tenant_id": "89afd400-b646-4bbc-b12b-c0a4d63e5bd3",
        "replication_status": null,
        "size": 10,
        "snapshot_id": null,
        "source_volid": null,
        "status": "creating",
        "updated_at": null,
        "user_id": "c853ca26-e8ea-4797-8a52-ee124a013d0e",
        "volume_type": "__DEFAULT__",
        "provider_id": null,
        "group_id": null,
        "service_uuid": null,
        "shared_targets": true,
        "cluster_name": null,
        "volume_type_id": "5fed9d7c-401d-46e2-8e80-f30c70cb7e1d",
        "consumes_quota": true,
        "migration_progress": "42"
   }
}

Security impact

None. The migration_progress will be admin-only.

Active/Active HA impact

None, I presume.

Notifications impact

None.

Other end user impact

Both the openstack and cinder cli will automatically show the new migration_progress field if it’s present in the API response.

Performance Impact

There will be no performance impact unless the volume is migrating. When it is migrating, an RPC is requred to fetch the migration progress data, either from nova (when the volume is attached) or from the volume service.

Other deployer impact

None.

Developer impact

None.

Upgrade impact

None.

Implementation

Assignee(s)

Primary assignee:: <abishop@redhat.com>

Work Items

Add a new microversion in cinder and bump the MAX in python-cinderclient
Add a new field to the volume details response in a new microversion
Add an RPC to fetch the migration progress from the volume service
Update the API code to invoke the appropriate (cinder or nova) RPC
Extend existing unit and functional tests
Update API documentation

Dependencies

Tracking the migration progress of an attached volume requires a nova feature. See the references for the nova feature spec.

Testing

TBD. The tricky part for incorporating something into a tempest test is catching a migration operation during the time it’s copying the volume data, which is the only time when the migration_progress field will be present in the volume details response.

Documentation Impact

The Block Storage API reference documentation will need to be updated, but there should be no impact on other user or admin facing documentation.

References

https://review.opendev.org/c/openstack/nova-specs/+/937485

OpenAPI Schemas

Fri, 15 Nov 2024 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/openapi

Problem description

Use Cases

As an end user, I would like to have access to machine-readable, fully validated documentation for the APIs I will be interacting with.

As an end user, I want statically viewable documentation hosted as part of the existing docs site without requiring a running instance of Cinder.

As an SDK/client developer, I would like to be able to auto-generate bindings and clients, promoting consistency and minimising the amount of manual work needed to develop and maintain these.

As a Cinder developer, I would like to have a verified API specification that I can use should I need to replace the web framework/libraries we use in the event they are no longer maintained.

Proposed change

This effort can be broken into a number of distinct steps:

Add missing request body and query string schemas

These schemas will merely validate what is already allowed, which means extensive use of "additionalProperties": true or empty schemas.

Once these are added, tests will be added to ensure all API resource have appropriate schemas.
Add response body schemas

These will be sourced from both existing OpenAPI schemas, currently published at opendev.org/openstack/codegenerator, and from new schemas auto-generated from JSON response bodies generated in tests and manually modified handle things like enum values.

Once these are added, tests will be added to ensure all API resource have appropriate response body schemas. In addition, we will add a new configuration option that will control how we do verification at the API layer, [api] response_validation. This will be an enum value with three options:

error
Raise a HTTP 500 (Server Error) in the event that an API returns an “invalid” response.

This will be the default in CI i.e. for our unit, functional and integration tests. Eventually this could be the default for production also (but probably not).

warn
Log a warning about an “invalid” response, prompting operations to file a bug report against Cinder.

This will be initial (and likely forever) default in production.

ignore
Disable API response body validation entirely. This is an escape hatch in case we mess up.

Note

Alternatives

Use a different tool

OpenAPI has been chosen because it is the most widely used API description language available and matches well with our existing use of JSON Schema for API validation.
Maintain these specs out-of-tree

This prevents us testing these on each commit to Cinder and means work that could be spread across multiple teams is instead focused on one small team.

Data model impact

None.

REST API impact

We may wish to address issues that are uncovered as we add schemas, but this work is considered secondary to this effort and can be tackled separately.

Security impact

None.

Active/Active HA impact

None.

Notifications impact

None.

Other end user impact

Performance Impact

Other deployer impact

Developer impact

Developers working on the API microversions will now be encouraged to provide JSON Schema schemas for both requests and responses.

Implementation

Assignee(s)

Primary assignee:: stephenfinucane
Other contributors:: gtema

Work Items

Add missing request body schemas
Add tests to validate existence of request body schemas
Add missing query string schemas
Add tests to validate existence of query string schemas
Add response body schemas
Add decorator to validate response body schemas against response
Add tests to validate existence of response body schemas
Update doc/source/contributor/api_microversion_dev.rst to include the requirement that anyone contributing a change that requires a new microversion is expected to provide request and response schemas

Dependencies

Testing

Unit tests will ensure that schemas eventually exist for request bodies, query strings, and response bodies.

Unit, functional and integration tests will all work together to ensure that response body schemas match real responses by setting [api] response_validation to error.

Documentation Impact

References

None.

History

Revisions
Release Name	Description
2025.01	Introduced

Rootwrap daemon mode

Mon, 04 Nov 2024 00:00:00

https://blueprints.launchpad.net/cinder/+spec/rootwrap-daemon-mode

Cinder is one of projects that require root privileges. Currently this is achieved with oslo.rootwrap that has to be run with sudo. Both sudo and rootwrap produce significant performance overhead. This blueprint is one of the series of blueprints that would cover mitigating rootwrap part of the overhead using new mode of operations for rootwrap - daemon mode. These blueprints will be created in several projects starting with oslo.rootwrap [1].

Problem description

As you can see in [2] rootwrap presents big performance overhead for Neutron. Impact on Cinder is not as significant but it is still there. Details of the overhead are covered in [1].

Use Cases

This will eliminate bottleneck in large number of concurrent executed operations.

Proposed change

This blueprint proposes implement changes that allow to run oslo.rootwrap daemon. The daemon works just as a usual rootwrap but accepts commands to be run over authenticated UNIX domain socket instead of command line and run continuously in background.

Note that this is not usual RPC over some message queue. It uses UNIX socket, so no remote connections are available. It also uses digest authentication with key shared over stdout (pipe) with parent process, so no other processes will have access to the daemon. Further details of rootwrap daemon are covered in [1].

use_rootwrap_daemon configuration option should be added that will make utils.execute use daemon instead of usual rootwrap.

Alternatives

Alternative approaches have been discussed in [5].

Data model impact

None

REST API impact

None

Security impact

This change requires additional endpoint to be available to run as root - cinder-rootwrap-daemon.

All security issues with using client+daemon instead of plain rootwrap are covered in [1].

Notifications impact

None

Other end user impact

None

Performance Impact

This change introduces performance boost for disk operations that are required to be run with root privileges. Current state of rootwrap daemon in Neutron shows over 10x speedup comparing to usual sudo rootwrap call. Total speedup for Cinder shows impressive results too [4]: test scenario CinderVolumes.create_and_delete_volume Current performance :

action	min (sec)	avg (sec)	max (sec)	count
cinder.create_volume cinder.delete_volume total	2.779 13.535 16.314	5.76 24.958 30.718	14.375 32.959 35.96	8 8 8

Load duration: 131.423681974 Full duration: 135.794852018

With use_rootwrap_daemon enabled:

action	min (sec)	avg (sec)	max (sec)	count
cinder.create_volume cinder.delete_volume total	2.49 2.183 4.673	2.619 2.226 4.845	3.086 2.353 5.3	8 8 8

Load duration: 19.7548749447 Full duration: 22.2729279995

Other deployer impact

This change introduces new config variable use_rootwrap_daemon that switches on new behavior. Note that by default use_rootwrap_daemon will be turned off so to get the speedup one will have to turn it on. With it turned on cinder-rootwrap-daemon is used to run commands that require root privileges.

This change also introduces new binary cinder-rootwrap-daemon that should be deployed beside cinder-rootwrap.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Anton Arefiev(aarefiev)

Work Items

The only work item here is to implement new config variable and run rootwrap in daemon mode with it.

Dependencies

rootwrap-daemon-mode blueprint in oslo.rootwrap [1].

Testing

Rootwrap has it’s own functional testing for the rootwrap client/daemon pieces [3], Cinder part will be covered by unit tests.

Cinder has an unusual usecase where tens/hundreds of mb are passed over stdin/out (sheepdog backup) - that test case should be covered in the functional tests.

Also we can add new Tempest job with turned on use rootwrap daemon flag.

Documentation Impact

Set use_rootwrap_daemon=True configuration option in cinder.conf to make utils.execute use daemon instead of usual rootwrap.

References

No specs have yet been approved.

Wed, 28 Aug 2024 00:00:00

rbd: Add snapshot mirroring support

Wed, 14 Aug 2024 00:00:00

https://blueprints.launchpad.net/cinder/+spec/rbd-snapshot-mirroring

This would be a good small project for someone interested in learning about rbd and setting up an rbd test environment. The blueprint was proposed back in 2021 and has code proposed, but the original author decided not to pursue it. You can make yourself a co-author and complete the patch.

This was a small change, so it doesn’t have a full spec, just a blueprint in Launchpad outlining the change.

References

Add extend volume completion action

Thu, 20 Jun 2024 00:00:00

https://blueprints.launchpad.net/cinder/+spec/extend-volume-completion-action

This blueprint proposes a new volume action that can be used by Nova to notify Cinder on success or failure when handling volume-extended external server events. The new volume action is used to add support for extending attached volumes to the NFS, NetApp NFS, Powerstore NFS, and Quobyte volume drivers.

Problem description

Many remotefs-based volume drivers in Cinder use the qemu-img resize command to extend volume files. However, when the volume is attached to a guest, QEMU will lock the file and qemu-img will be unable to resize it.

In this case, only the QEMU process holding the lock can resize the volume, which can be triggered through the QEMU monitor command block-resize.

There is currently no adequate way for Cinder to use this feature, so the NFS, NetApp NFS, Powerstore NFS, and Quobyte volume drivers all disable extending attached volumes.

Use Cases

As a user, I want to extend a NFS/NetApp NFS/Powerstore NFS/Quobyte volume while it is attached to an instance and I want the volume size and status to reflect the success or failure of the operation.

Proposed change

Nova’s libvirt driver uses the block-resize command when handling the volume-extended external server event, to inform QEMU that the size of an attached volume has changed. It is in principle also capable of extending a volume file, but is currently unable to provide feedback to Cinder on the success of the operation.

Currently, Cinder will send the volume-extended external server event to Nova only after it has finalized the extend operation and reset the volume status from extending back to in-use.

This spec proposes to give volume drivers a mechanism to hold off finalizing the extend operation until after the volume-extended event has been sent and Cinder has received feedback from Nova that it was handled successfully.

This spec also proposes a new volume action that Nova will use to provide this feedback to Cinder.

API

A new API microversion is introduced, adding the new os-extend_volume_completion volume action.

The volume action takes a boolean error argument, indicating success or failure to extend the attached volume. It is intended to be used exclusively by Nova to notify Cinder, and an appropriate policy will be added to enforce this.

API.extend_volume_completion

The new volume action will be handled by a new method in the volume API:

def extend_volume_completion(self,
                             context: context.RequestContext,
                             volume: objects.Volume,
                             error: bool) -> None:

The new method expects the volume to have status extending, and to have the keys extend_reservations and extend_new_size in its admin metadata. The first should hold a list of quota reservations, and the second should contain an integer larger than the volume’s current size, representing the new size after extending.

If these conditions are not met, then an InvalidVolume exception will be raised, resulting in an HTTP response of 400 Bad Request.

If the conditions are met, it will remove size and reservations from the admin metadata and call VolumeManager.extend_volume_completion() via RPC, passing both as arguments.

VolumeManager.extend_volume_completion

def extend_volume_completion(self,
                             context: context.RequestContext,
                             volume: objects.Volume,
                             new_size: int,
                             reservations: list[str],
                             error: bool) -> None:

The behavior of this method depends heavily on the error argument:

If error is True, the method will roll back the quota reservations, set the volume status to error_extending, and log the error.
If error is False, it will finalize the quota reservation, update the size field of the volume to the new size, and reset the volume status to available or in-use, depending on the presence of attachments. It will also update the pool stats and send a resize.end notification with the new volume size.

This is identical to how VolumeManager.extend_volume() currently handles success and failure of the volume driver’s extend_volume() method, except that this method will not notify Nova with the volume-extended external server event.

VolumeDriver.extend_volume

A mechanism will be introduced by which the driver’s extend_volume() method can signal to the volume manager that it has to wait for a response from Nova before finishing the extend operation. This could take the form of a return value or a new exception that the volume manager will have to catch.

The NFS, NetApp NFS, Powerstore NFS, and Quobyte volume drivers currently have checks in their respective extend_volume methods, that will raise an exception if the volume to be resized is attached, causing the operation to fail. Those checks will be removed.

Instead, the drivers will catch any exceptions resulting from the volume files being locked (see the proposed change to nfs.py in [1] for an example on how to do that), and notify the volume manager that feedback from Nova is required.

VolumeManager.extend_volume

The call to the volume driver’s extend_volume() method will be handled as follows:

If the call fails, extend_volume_completion will be called with error=True.
If the call succeeds, but the volume is not attached, extend_volume_completion will be called with error=False.
If the call succeeds, and the volume is attached, extend_volume_completion will be called with error=False and Nova will be notified with the external server event.

This matches the current inline behavior of the method, and covers offline extend for all drivers, as well as online extend for the drivers that previously supported it.

To support remotefs-based drivers that have to rely on Nova for online extend, two aditional cases will be handled:

If the driver notifies the volume manager that a response from Nova is required, but the volume is not attached, or the volume is attached to more than one instance, it will be handled as failure and extend_volume_completion will be called with error=True.

QEMU can not resize shared volume files, because they are locked read-only, so adding multi-attach support for this feature is currently not worthwhile. However, support may be added later if other drivers require it, e.g. by enabling Cinder to handle multiple completion actions for the same volume.
If the driver notifies the volume manager that a response from Nova is required, and the volume is attached to exactly one instance, then Cinder will store the quota reservations and the target size in the in the admin metadata with the keys extend_reservations and extend_new_size.

It will then attempt to send the volume-extended external server event with the new Nova API microversion proposed in [4], making sure that Nova supports using the os-extend_volume_completion action.
- If the volume-extended event has been submitted to Nova successfully, this method will just return normally. The volume will now be left in status extending, which will signal to Nova that it should respond with the os-extend_volume_completion action, as described in the Nova subsection.
- If the volume-extended event could not be submitted, the operation will be rolled back by calling extend_volume_completion with error=True.
  
  This can happen if Nova doesn’t support the required microversion yet, or if the external event API responded with an error code such as 403 or 404.

Visible Admin Metadata

extend_new_size has to be stored in the admin metadata, because the regular volume metadata is editable by users. A malicious user could otherwise edit the target size during the operation to bypass their quota.

Admin metadata of volumes is not visible to clients, but Cinder supports mapping select keys to the regular metadata, shadowing any user-set values of the same key.

The key extend_new_size will be added to the list of visible admin metadata in cinder/api/api_utils.py, so that Nova is able to read the target size of the extend operation.

OpenStack SDK

Support for the new volume action will be added to the OpenStack SDK, which Nova will use to call it.

Nova

When the Nova API receives a volume-extended external server event, and the call used the new microversion proposed in [4], it will check the target compute service version. If a target compute agent is too old to support the feature, the API will discard the event and call the os-extend_volume_completion volume action with "error": true.

Otherwise, the event will be forwarded to the compute agent. When handling the volume-extended external server event, compute will check the volume status:

If the volume status is extending, then compute will attempt to read extend_new_size from the volume’s metadata and use this value as the new size of the volume, instead of the volume size field.

After successfully extending the volume, it will call the extend volume completion action of the volume, with "error": false.

If anything goes wrong, including extend_new_size being missing from the metadata, or being smaller than the current size of the volume, compute will log the error and call the extend volume completion action with "error": true.
For any other volume status the event will be handled as before.

The changes in Nova are detailed in the current version of the Nova spec at [4].

os-reset_status

When resetting from status extending, the os-reset_status volume action will check for the extend_reservations key in the admin metadata. If it finds quota reservation keys, it will try to roll them back.

This is done to avoid a pile up of quota reservations in case communication between Cinder and Nova was lost and the status has to be reset to retry the resize.

The keys extend_reservations and extend_new_size will then be removed from the admin metadata.

Alternatives

A previous change tried to use the volume-extended external server event to support online extend for the NFS driver [1], but did not rely on feedback from Nova to Cinder at all. Instead, it would just set the new size of the volume, change the status back to in-use, notify Nova, and hope for the best.

If anything went wrong on Nova’s side, this would still result in a volume state indicating that the operation was successful, which is not acceptable.
The specs at [2] and [3] proposed a new synchronous API in Nova that can be used to trigger an assisted resize operation. This API would provide a single mechanism to trigger the resize operation, communicate the new size to Nova, and get feedback on the success of the operation.

The problem with a synchronous API is, that RPC and API timeouts limit the maximum time an extend operation can take. For QEMU, this seemed to be acceptable, because storage preallocation is hard disabled for the block-resize command, and because all currently plausible file systems support sparse file operations.

However, as reviewers in [2] have pointed out, this may not be true for other volume or virt drivers that might require this API in the future. It would also break with the established pattern of asynchronous coordination between Nova and Cinder, which includes the assisted snapshot and volume migration features.
Following this pattern, we could make the proposed API asynchronous and use a new callback in Cinder, similar to Nova’s os-assisted-volume-snapshots API, which uses the os-update_snapshot_status snapshot action to provide feedback to Cinder.

The function of the new Nova API would then just be to trigger the operation and to communicate the new size. The question is then, whether that warrants adding a new API to Nova, since there are existing mechanisms that could be used for either.
The existing mechanism for triggering the extend operation in Nova is, of course, the volume-extended external server event. Using it for this purpose, as this spec proposes, requires the target size to be transferred separately, because external server events only have a single text field that is freely usable, which for volume-extended is already used for the volume ID.

Besides storing it in the admin metadata, as this spec proposes, there is also the option of updating the size field of the volume, as [1] was essentially doing.

This would require the volume size field to be reset on a failure. If an error response from Nova was lost, the volume would just keep the new size. We would need to extend os-reset_status to allow a size reset, or something similar to clean up volumes like this. This would be possible, but updating the size field only after the volume was successfully extended seems like a cleaner solution.
We could also extend the external server event API to accept additional data for events, and use this to communicate the new size to Nova.

This option was judged favorably by reviewers on the previous version of this spec, [2], but it would be a more complex change to the Nova API.

However, if additional data fields become available in a future version of the external server event API, it would be a relatively minor change to use those instead of the volume metadata.

Data model impact

None

REST API impact

Starting with the new microversion, the POST /v3/{project_id}/volumes/{volume_id}/action API will accept request bodies of the following form:

{
    "os-extend_volume_completion": {
        "error": false
    }
}

with error indicating success or failure of the resize operation.

If the volume does not exist, the return code will be 404 Not Found.

If the volume status and admin metadata do not indicate that Cinder was waiting for an extend volume completion action, the return code will be 400 Bad Request.

Otherwise the return code will be 202 Accepted.

The new volume action is intended to only be used by Nova and will require the caller to have admin permissions.

Security impact

None

Active/Active HA impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: kgube

Work Items

Move extend completion code from VolumeManager.extend_volume to new method and add tests.
Create new volume action and add unit tests.
Add a new microversion for the new os-extend_volume_completion action.
Add OpenStack SDK support.
Add Nova support.
Update drivers to use the feature.
Adapt the devstack-plugin-nfs-tempest CI-jobs to also test online volume extend.

Dependencies

Nova support of the callback [4].

Testing

Unit tests for the volume action will test the conditions all possible API responses.
Unit tests for VolumeManager.extend_volume will test all the code paths described in VolumeManager.extend_volume.
The new volume action cannot be independently tested by Tempest, because it requires the volume to be in a state that cannot be reproduced externally. It is, however, covered by the existing tests for online volume extend when they are run with one of the volume drivers that use this feature. The devstack-plugin-nfs-tempest jobs that run as part of the Cinder and Nova CI gates will be configured to enable online volume extend tests.

Documentation Impact

The Block Storage API reference will be updated to include the new volume action.

The volume driver support matrix will be updated to show online resize support for the affected drivers.

References

Standardize Image Encryption and Decryption

Mon, 13 May 2024 00:00:00

OpenStack already has the ability to create encrypted volumes and ephemeral storage to ensure the confidentiality of block data. Even though it is also already possible to store encrypted images, there is only one service (Cinder) that utilizes this option, but it is only indirectly usable by Nova (a user must create a volume from the image first), and thus users don’t have an intuitive way to create and upload encrypted images. In addition, all metadata needed to detect and use encrypted images is either not present or specifically scoped for Cinder right now. In conclusion, support for encrypted images does exist to some extent but only in a non-explicit and non-standardized way. To establish a consistent approach to image encryption for all OpenStack services as well as users, several adjustments need to be implemented in Glance, Cinder and OSC.

Problem description

An image, when uploaded to Glance or being created through Nova from an existing server (VM), may contain sensitive information. The already provided signature functionality only protects images against alteration. Images may be stored on several hosts over long periods of time. First and foremost this includes the image storage hosts of Glance itself. Furthermore it might also involve caches on systems like compute hosts. In conclusion they are exposed to a multitude of potential scenarios involving different hosts with different access patterns and attack surfaces. The OpenStack components involved in those scenarios do not protect the confidentiality of image data.

Use Cases

1. A user wants to create a new volume based on an encrypted image. The corresponding volume host has to be enabled to detect, that the image is encrypted. Additionally encrypted images should always result in encrypted volumes to avoid decryption.

1.1 If an encrypted image is the base for a new volume the used volume type
should always have an encryption type. If the given volume type or default
volume type does not have an encryption type the operation should result
in an ERROR.

1.2 There are two possible types of encrypted images: qcow2 and raw images.
Cinder can already handle encrypted raw images. Encrypted qcow2 images
may need to be flattend to raw before transfering them into volumes.

1.3 Encrypted images, that were created from encrypted volumes, may be
compressed depending on Cinder’s allow_compression_on_image_upload
option. This also needs to be handled when creating an encrypted volume
from such an image.

2. Whenever an encrypted image is converted to an encrypted volume the secret should be copied to give Cinder full control over the life-cycle of the secret.

2.1. The secret can be a key or a passphrase. The secret type classification in the Key-Manager will determine the key handling (“symmetric” vs “passphrase”). Currently, Cinder is only able to handle “symmetric”. Support for “passphrase” (like used in Nova) has to be added.

3. A user wants to create an image from a volume with an encrypted volume type. The target image will reuse the LUKS encryption and key. This use case is already implemented as part of the default behavior of Cinder.

3.1. Creating an encrypted image from an unencrypted volume will not be
part of this spec, but may be implemented later on.

3.2. Creating an unencrypted image from an encrypted volume is not possible
right now. The volume encryption is transparently also used for the image.
This behavior will stay in place to optimize resource usage and avoid costly
conversion of the whole volume data on volume hosts.

Proposed change

In Glance we propose the following additional metadata properties that should be carried by encrypted images:

‘os_encrypt_format’ - the main mechanism used, e.g. ‘LUKS’
‘os_encrypt_cipher’ - the cipher algorithm, e.g. ‘AES256’
‘os_encrypt_key_id’ - reference to secret in the key manager
‘os_encrypt_key_deletion_policy’ - on image deletion indicates whether the key should be deleted too
‘os_decrypt_container_format’ - format change, e.g. from ‘compressed’ to ‘bare’
‘os_decrypt_size’ - size after payload decryption

We propose to align the encryption with Nova and Cinder and use LUKS, which will be allowed in combination with qcow and raw images. We use this two versions for the following reasons:

Nova can directly use qcow-LUKS encrypted when creating a server. This is the standard procedure of Nova.
Cinder allows the creation of Images from encrypted volumes. These will always result in LUKS-encrypted raw images. Those images can be converted directly to volumes again.

In the latter case it is already possible to upload such an encrpyted image to another OpenStack infrastructure, upload the key as well and set the corresponding metadata. After doing so the image can be used in the second infrastructure to create an encrypted volume.

We want to align the existing implementations between Nova and Cinder by standardizing the used metadata parameters and adding interoperability where applicable. This would in the case of Cinder mainly be a rename: - ‘cinder_encryption_key_deletion_policy’ to ‘os_encrypt_key_deletion_policy’ - ‘cinder_encryption_key_id’ to ‘os_encrypt_key_id’

In the Volume creation there will be a check added, to look for encrypted images proposed as a volume source. If an image is encrypted another check is added to determine, whether the volume type to create the volume has an encryption type. If that is not the case the volume create will be aborted in the API still. Otherwise there will be an unusable volume created.

The flattening of a qcow2 image should be handled when uploading the image to the volume. The volume size should be resulting from the ‘os_decrypt_size’ parameter. If compression is enabled through Cinder’s allow_compression_on_image_upload option Cinders implementation to handle this should be re-used.

The key management for creating an encrypted volume from an encrypted image must include the copying of the secret in Barbican. On this way Cinder always has the full control over the life-cycle of the secret, because this way is similar to the original creation of an encrypted volume.

In all places that use decryption within Cinder, there need to be an additional check for the type of the secret. And a different handling, if the secret is a “passphrase”, because the way Cinder treats keys to create a passphrase for the LUKS header of a volume is quite unique and differs from Nova’s handling of images, that have passphrases only.

The creation of an image from an volume just need to be adjusted to use the new parameters.

Alternatives

We also evaluated an image encryption implementation based on GPG. The downside with such an implementation is, that everytime such an image is used to create a server or a volume the image has to be decrypted and maybe re-encrypted for another encryption format as both Nova and Cinder use LUKS as an encryption mechanism. This would not only have impact on the performance of the operation but it also would need free space for the encrypted image file, the decrypted parts and the encrypted volume or server that is created.

Data model impact

None

REST API impact

When creating a volume from an encrypted image there might occure a new ERROR that is triggered, when an image is encrypted but no encrypted volume type is given.

Security impact

There are impacts on the security of OpenStack:

confidentiality of data in images will be addressed in this spec
image encryption is introduced formally, thus cryptographic algorithms will be used in all involved components (Nova, Cinder, OSC)

Active/Active HA impact

None

Notifications impact

None

Other end user impact

Users should be able to use encrypted images to create volumes in a consistant way

Performance Impact

The proposed checks for the Cinder API may have minimal impact on performance.

When creating a volume or server from an encrypted image the only operation that may be triggerd is the conversion between qcow-LUKS and raw LUKS blocks.

Other deployer impact

For interoperability between the OpenStack services only the presence of a key manager should decide, whether encryption can be used or not.
A key manager - like Barbican - is required, if encrypted images are to be used.

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Markus Hentsch (IRC: mhen)

Other contributors: Josephine Seifert (IRC: Luzi)

Work Items

Add checks in the create volume API
Add copying the secret and registering as a consumer in Barbican
Add flattening of qcow2 to raw encrypted images
In the image create from volume: change the ‘cinder_encryption_key_deletion_policy’ to ‘os_encrypt_key_deletion_policy’ and ‘cinder_encryption_key_id’ to ‘os_encrypt_key_id’

Dependencies

Presence of the image encryption parameters in Glance has to be implemented

Testing

Tempest tests would require access to encrypted images for testing. This means that Tempest either needs to be provided with an image file that is already encrypted and its corresponding key or needs to be able to encrypt images itself. This point is still open for discussion.

Documentation Impact

It should be documented for deployers, how to enable this feature in the OpenStack configuration.

References

[1] Barbican Secret Consumer Spec: https://review.opendev.org/#/c/662013/

[2] Glance Image Encryption Spec: https://review.opendev.org/c/openstack/glance-specs/+/915726

History

Revisions
Release Name	Description
Dalmatian	Introduced

User visible information in volume types

Fri, 16 Feb 2024 00:00:00

https://blueprints.launchpad.net/cinder/+spec/user-visible-information-in-volume-types

Volume types are used to find a fitting backend when creating a volume. But they also imply a few configurations to the volume to be created, that are good to know for anyone who has to choose from various volume types.

Problem description

When a non-admin user is creating a volume they would like to see various information about the volume type. Some of those information are already visible for users, when looking into a volume type (e.g. multiattach). Other information are either not available at all or not visible for non-admin users. This is a problem when trying to choose a fitting volume type when creating a volume.

Use Cases

The information users would like to see are first and foremost: 1. Whether a volume type can be used to create encrypted volumes 2. Whether a volume type creates replicated volumes, either through Cinder or through a configured backend like ceph

The first information comes from the encryption type within the volume, which is only accessible for an admin. The second information can either be set and directly seen in the volume type extra_specs or is indirectly a part of the configuration of the backend.

There might be more use cases, that should benefit from a solution allowing an admin to add certain user facing information into the volume type in a uniform way.

As there are more and more tools that automatically create IaaS resources, that information should be accessible in a uniform way and could best be used to filter for fitting volume types in the client.

Proposed change

To let an operator add information that is visible for users and also machine- readable, we introduce a new field in the volume type object called “metadata”. This will include API changes to address the new view of the volume type as well as a new API endpoint to set information as key value pairs for this metadata field. A new database table for those metadata is also necessary.

The table will consist of columns for the volume type id, the key, the value, an id as the primary key and metadata for creation, updates and deletion. The upgrade path for the database may look like this:

def upgrade():
  op.create_table(
      'volume_type_metadata',
      sa.Column('created_at', sa.DateTime),
      sa.Column('updated_at', sa.DateTime),
      sa.Column('deleted_at', sa.DateTime),
      sa.Column('deleted', sa.Boolean),
      sa.Column('id', sa.Integer, primary_key=True, nullable=False),
      sa.Column(
          'volume_type_id',
          sa.String(36),
          sa.ForeignKey(
              'volume_types.id',
              name='volume_type_metadata_ibfk_1',
          ),
          nullable=False,
          index=True,
      ),
      sa.Column('key', sa.String(255)),
      sa.Column('value', sa.String(255)),
      mysql_engine='InnoDB',
      mysql_charset='utf8',
  )

In API there will be two fields: “extra_specs” and “metadata” and in CLI those fields will be “properties” and “metadata”. The latter might confuse users, so documentation is needed for both API and CLI to clarify, what to expect of each field. The limit for this change is, that key-value pairs not meant for scheduling purposes can still be put into the properties/extra_specs and will lead to Errors in the scheduling process later on.

In a second phase metadefs can be added to standardize certain keys like “encryption_type” or “replicated”.

Alternatives

We evaluated using the already existing user facing “extra_specs” field. Administrators can already set key-value pairs here and user visible extra_specs can be seen in this field. But this would lead to a problem in the volume scheduler, as EVERY input in the extra_specs table will be evaluated by the scheduler when looking for a fitting backend for the volume. While this problem could be solved through a whitelisting or blacklisting approach, maintaining and evaluating such lists is very opaque for users. Overall this approach may lead to even more confusion in users.

Another option is to face use cases individually:

1. Information about whether a volume type creates encrypted volumes or not could be calculated in the API calls for list/show volume types from the presence of an encryption type. The information would be shown in the “properties” field. This would be a very minimal patch, but will not solve other use cases, that would benefit from user facing information. 2. Creating an extra field for the encryption in the volume type table, that is automatically set when creating or deleting an encryption type. This would need a database change and a change of the view of the volume_types. 3. Looking into the different drivers and how they handle internal configurable replication and whether there are ways to let OpenStack know this and propagate it to users. This is very hard to achieve, maybe even impossible without input from an operator, who configured the backends and volume types. 4. The policy for the encryption type could be loosend to let users see not only whether a volume type has an encryption type, but also what algorithm and provider is used for it. This may have a security impact.

All these options are not able to solve the general issue of having a reliable way to provide user visible information in volume types. Neither will they solve the problem, that operators are able to add key-value pairs not meant for scheduling purposes to the volume type extra specs.

Data model impact

It will be necessary to create a new table metadata that works like the extra_specs table. As there are currently no user visible information in volume types in place, an initially empty table would be sufficient.

To adhere the case of an upgrade from a previous OpenStack version, all volume types need to be checked and for volume types with associated encryption types, there need to be an encryption parameter set in the database according to the chosen option. This should not be doable through an API call but will need direct DB access, as changing the metadata of volume types in use should be prohibited.

REST API impact

A new field needs to be added to the view of volume types. For that field new API calls will be needed:

a POST method to set new key-value pairs
a DELETE method to delete a key-value pair.
optionally a GET method to show a key-value pair.
optionally add filtering for certain metadata for the GET volume types method

The volume type metadata will be handled like the volume type description, that is, we will allow it to be modified even if the volume-type is in use. We can do this because the volume type metadata is only descriptive, as opposed to the volume-type extra_specs that affect the scheduler.

Additionally the create volume type API call should be able to handle key-value pairs for the new metadata field.

Security impact

This change will expose a boolean value, that shows, whether a volume type will encrypt volumes at creation or not.

Active/Active HA impact

There should not be any impact upon Cinder’s Active/Active HA support.

Notifications impact

There will be additional notifications when database entries are made or removed. They will behave the same way as the notifications of extra_specs. Those will be sent, whenever a key-value pair is created or deleted from the new metadata field/database.

Other end user impact

This change may lead to users wanting to use the filtering mechanism in openstack volume type list –metadata key=value for both explained use cases.

Performance Impact

The API call for getting volume types and their details will have an additional query for the new metadata table.

Other deployer impact

Old volume types with associated encryption types will need to have the potentially new database entry set. This cannot be done via volume type set command, but has to be written into the database either manually or with a script, that sets the correct entry for all affected volume types.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Josephine Seifert (josei)
Other contributors:: None

Work Items

Add a “volume_type_metadata” table to the Cinder database
Add new API-Endpoints for the metadata field of volume types and add new possibility to create new key-value pairs in the metadata field when creating a volume type
Add support in CLI/SDK for the new API calls

Dependencies

There are no dependencies.

Testing

The test case for creating a volume type may be updated to integrate the addition of a metadata key-value pair.

Documentation Impact

There will be a documentation needed to separate between the extra specs and the handling of user visible information.

References

No specs were approved for the 2023.1 release.

Thu, 07 Dec 2023 00:00:00

New Quota System

Sat, 21 Oct 2023 00:00:00

https://blueprints.launchpad.net/cinder/+spec/count-resource-to-check-quota

Cinder quotas have been a constant pain for operators and cloud users. This spec proposes a new quota system to resolve inconsistencies in tracking quota usage.

Problem description

Cinder’s current quota system is based on reservations and commits/rollbacks. When the API receives an operation that consumes quota, the request is validated and Cinder checks that there’s enough quota to perform the operation, and finally creates reservations to ensure that this quota is not consumed by other operations. Then when the operation finishes these reservations are committed as used resources or rolled back if the operation failed.

The current usage and reservations for resources, e.g. number of volumes or gigabytes, are tracked in the database as a counter in the quota_usages table, so if for whatever reason this counter doesn’t match the real usage, then users may become unable to create new resources or able to create resources beyond their project limits.

There are many possible reasons why resource tracking can become out of sync in Cinder, including bugs in the code and services dying during an operation.

As a workaround for quotas becoming out of sync, the Cinder service has code to make reservations expire after a given time and also has the possibility of resynching the quotas with a certain frequency.

Besides the impact on operators and users, the current quota implementation also has an impact on developers, because the system of reserve/commit and resource counting in the database has the downside of having to be very thorough in order to always keep track of everything, making the quota system a very manual and tedious process to code, which in turn results in hard to find bugs, since we don’t know at which point the counting went wrong.

Low level implementation details of the current quota system are also present everywhere in Cinder, and almost every single area of the code needs to be aware of the low level implementation details, making the code very verbose and obscuring the high level logic.

As an example, we can look at the code of the accept method of the transfer api in file cinder/transfer/api.py, where, at the time of this writing, out of the 106 lines of code that constitute the method, 70 are quota related!

Use Cases

A cloud administrator wants to prevent system capacities from being exhausted without notification so it limits quota systems based on the deployments capabilities.
A cloud administrator wants to limit how many resources each department can consume.
A Cinder contributor wants to add a new feature that creates or destroys resources, so it needs to write code to manage the quota.

Proposed change

This spec proposes a new quota system where most low level quota details will be hidden from developers working on features, simplifying feature development and the chances of introducing new bugs.

This quota system will support 2 different drivers:

StoredQuotaDriver: This will be similar to the old system, using counters in the DB, but instead of doing reservations and commits/rollbacks for every resource modification, it will only do reservations for the very few operations that really need it to keep track of the resources while the operation is in progress.
DynamicQuotaDriver: This driver will no longer store usage and resource tracking in the database table (quota_usages) and instead dynamically calculates each quota check based on the resources that exist in the database.

Calculations will be counting for resources (e.g. snapshots) or sum of values for sizes (e.g. gigabytes).

Just like the StoredQuotaDriver driver this will use reservations as little as possible.

The reason for having 2 drivers instead of a single one is because there are trade-offs with each of the drivers, and the default will be the DynamicQuotaDriver driver, for reasons explained later in the Performance Impact section.

The changes will try, as much as possible, to avoid over engineering the solution focusing on the 2 new drivers and current cinder and not solve all cases for potential different drivers that will probably never come.

Quota limits

In the original implementation of quota classes (quota_classes database table) it was mentioned the possibility of supporting different classes besides the existing default, and being able to pass it via the context, but more than 9 years after its implementation neither Nova nor Cinder support it, so this new quota system and drivers will just focus on the default quota class, which will be referred as system wide defaults, global defaults, global limits, or just defaults since the term is easier to understand.

Limits from the quotas table will be referred as per project limits.

The effective quota limits that apply to a specific project after considering the global and per project limits will be referred simply as quota limits.

The way quota limits are calculated will be kept as they are now:

System wide quota limit defaults are stored in the quota_classes table under the records that have the default value in their class_name column.
Per project quota limit overrides that replace the system wide limits are optional and will be stored in the quotas table.
A hard_limit value of -1 indicates that there are no limits.

Resources

The new quota system will not introduce or remove any of the existing quota resources, so available resources for the quota limits will still be: volumes, volumes_<volume-type>, snapshots, snapshots_<volume-type>, gigabytes, gigabytes_<volume-type>, backups, backup_gigabytes, groups, and per_volume_gigabytes. And quota usage will report in-use and reserved values for all existing limits except the per_volume_gigabytes, since there cannot be any usage for it.

Reservation values will be stored in the delta field of the reservations table just like they are today.

For the DynamicQuotaDriver these values will be dynamically added, grouping by resource for non deleted rows belonging to the specific project. On the other hand the StoredQuotaDriver will track the sum in the reserved field of the quota_usages table.

Both drivers will adhere to the following rules when reporting in-use values:

volumes and volumes_<volume-type> quotas must match the number of non deleted rows in the volumes table, with the use_quota field set to true plus the sum of positive values from the reservations table where the resource matches volumes or volumes_<volume-type>, in both cases only those values belonging to the specific project.
snapshots and snapshots_<volume-type> quotas must match the number of non deleted rows in the snapshots table, with the use_quota field set to true, and belonging to the specific project.
gigabytes and gigabytes_<volume_type> quotas must match the sum of the size of the quotable volumes (as defined above) incremented in the sum of the volume_size values of the quotable snapshots (as defined above) when the no_snapshot_gb_quota configuration option is set to false (default value) plus the sum of positive values from the reservations table where the resource matches gigabytes or gigabytes_<volume-type>, in both cases only those values belonging to the specific project.
groups quota must match the number of non deleted rows in the groups table.
backups quota must match the number of non deleted rows in the backups table, and backup_gigabytes must match the sum of their size values.
per_volume_gigabytes is a quota limit that doesn’t need any kind of calculation.

Mechanism

The new quota system will rely heavily on database transactions and database row locking using the SELECT FOR UPDATE SQL statement to control parallel operations and ensure quota limits are honored and all database changes happen or they are automatically rolled back.

A high level view of how this mechanism would work is:

Start a transaction
Get current quota limits creating a lock on those rows
Check operation doesn’t go over quota
Create the resource on the database or make reservations
Finish the transaction releasing the lock

The lock will only happen on the rows of the resources we are interested in, allowing operations on other projects and resources to be executed in parallel. For example, quota checks to create a volume will lock rows for volumes, volumes_<volume-type>, gigabytes, and gigabytes_<volume-type>, so cinder will be able to check for the quota to create a backup since that only requires backup and backup_gigabytes.

The new system will leverage Python context manager functionality and the Oslo DB transaction context provider available in the Cinder RequestContext (context.session) to facilitate the sharing of the transaction/session between different areas of the Cinder code.

This will allow developers to write cleaner code, for example when creating a volume, the create method of the Cinder Volume Oslo object will have to check that it can create 1 volume, that will consume additional gigabytes and that the size of the volume doesn’t exceed the largest volume size allowed, so the code will be something like this:

with self.quota_check(self._context, self.volume_type.id,
                      vol_gbs=self.size,
                      vol_qty=1,
                      vol_size=self.size):

   db_volume = db.volume_create(self._context, updates)

The quota_check is a property in the Volume OVO that returns a context manager that ensures quota limits are honored. Returned context manager depends on whether the volume consumes quota or not, returning a noop if it doesn’t and returning a context manager provided by the quota driver if it does.

The quota driver context manager starts a DB session/transaction in the provided context so the volume_create call will use that same session to create the volume record, and the transaction will be finalized when the code exits the context manager, thus ensuring that no other operations check the quota until the volume has been created.

From a developer’s point of view all this will be hidden, because at a higher level all they need to do is create the Volume OVO and the quota will be automatically checked. As an example this is the code in the create volume flow (cinder/volume/flows/api/create_volume.py):

volume = objects.Volume(context=context, **volume_properties)
volume.create()

In an effort to abstract the quota system implementation and hide its details from most of the code, the code interfacing with the driver directly will no longer use the resource names such as gigabytes and volumes_<volume-type>, instead the parameters that will be used for the volume and snapshot context manager checker are:

vol_qty: Delta on the number of volumes that will be consumed within the checker context manager. The quota system internal name for this is volumes in the database.
vol_type_vol_qty: Delta on the number of volumes for the specific volume type that will be consumed within the checker context manager. Defaults to the value of vol_qty since that’s the most common case. The quota system internal name for this is volumes_<volume-type> in the database.
vol_gbs: Delta on the number of volume gigabytes that will be consumed within the checker context manager. The quota system internal name for this is gigabytes in the database.
vol_type_gbs: Delta on the number of volume gigabytes for the specific volume type that will be consumed within the checker context manager. Defaults to vol_gbs since that’s the most common case. The quota system internal name for this is gigabytes_<volume-type> in the database.
snap_qty: Delta on the number of snapshots that will be consumed within the checker context manager. The quota system internal name for this is snapshots in the database.
snap_type_qty: Delta on the number of snapshots for the specific volume type that will be consumed within the checker context manager. Defaults to the value of snap_qty since that’s the most common case. The quota system internal name for this is snapshots_<volume-type> in the database.
snap_gbs: Delta on the number of snapshot gigabytes that will be consumed within the checker context manager. Will end up using the quota system internal name of gigabytes if the no_snapshot_gb_quota configuration option is set to false (default) or will be disregarded if set to true.
snap_type_gbs: Delta on the number of snapshot gigabytes for the specific volume type that will be consumed within the checker context manager. Defaults to the value of snap_gbs since that’s the most common case. The quota system internal name for this is gigabytes_<volume_type> in the database if the no_snapshot_gb_quota configuration option is set to false or will be disregarded if set to true.
vol_size: Total volume size when creating or extending it, in the internally the quota system uses the per_volume_gigabytes quota limit to check this value.

This change may seem worthless, but it has its value, because it abstracts the implementation details of the snapshots and volumes sharing the same quota size limits which provides:

Cleaner code since snapshot creation or transfer of a volume with snapshots doesn’t need to know about the no_snapshot_gb_quota configuration option.
If we want to add, in the future, snapshot specific quota limits - snapshot_gigabytes and snapshot_gigabytes_<volume-type>- we’ll be able to do so without affecting any of the Cinder code with the sole exception of the quota driver itself.

Reservations

For the new quota system the reservation commit and rollback operations will be grouped into a single context manager that handles both cases. Committing and rolling back reservations have different meanings for the 2 drivers.

For the DynamicQuotaDriver these are noop operations, since checks use the DB values every time and the database has already been modified in the same transaction that the reservations are removed. On the other hand the StoredQuotaDriver needs to modify the in_use and reserved counters in the quota_usages table accordingly to the operation.

As mentioned before, reservations will only be necessary for specific operations, to be exact on 3 operations: extend, transfer, and retype.

Each of these operations have different reasons for requiring reservations:

Extend: Until the operations completes, the size field of the volume in the database must be kept as it is to reflect its real value, but we need to reserve the additional gigabytes, for gigabytes and gigabytes_<volume_type> quotas, during the operation so we don’t go over quota due to other concurrent operations. If the operation completes successfully the size of the volume will be increased and the reservations will be committed.
Transfer: Under normal circumstances accepting a transfer would not require the use of a reservation, as we should be able to check the quota and do the database changes to accept the transfer in the same transaction. Unfortunately the SolidFire and VMDK drivers need to make some changes in their backend on transfer, so the volume service has to make a driver call.

We cannot keep the database locked while the driver call completes, as it can take some time and we don’t want to prevent the API from processing other operations.

That is why reservations will be created before calling the driver and cleared after accepting the resources.

In terms of reservations, transfers are complex for the StoredQuotaDriver, because when completing one it needs to modify 2 different projects. One to increase counters and the other to decrease them, so higher levels will need to make 2 different calls for 2 different projects, one with positive and one with negative numbers and negative numbers should ignore quota usage and limits.

When storing reservations for transfer of volumes with snapshots they have to be stored separately in case someone restarts the service after changing the no_snapshot_gb_quota_toggled configuration option as detailed in the vol_snap_check_and_reserve_cm method below.
Retype: When doing a retype the API needs to reserve gigabytes_<dest-volume-type> and volumes_<dest-volume-type> until the operation is completed as well as create negative reservations for gigabytes_<source-volume-type and volumes_<source-volume-type>.

This consumes volumes and gigabytes on both types until the operation completes for the following reasons:
- If the retype fails we will continue consuming volumes and gigabytes on the source volume type, but if we “released” that usage when we started the operation we may find that there is no longer enough quota available for the volume to stay there. This is the main reason.
- Even if the retype succeeds Cinder doesn’t know the reasons why the cloud administrator has set the quota limits, so freeing the source gigabytes and volumes as soon as the retype starts means that if a new volume for the source type is created during the retype Cinder will be exceeding the quota for that volume type.
This is the only operation where a race condition can happen, though it’s a corner case. It can happen if we are adding a new quota limit, global or per project, to a volume type resource (e.g. volume_<volume_type>) that didn’t have any limit in the database while at the same time we are doing volume retypes to that same volume type. This race should fall within reasonable expectations, as one would argue that the limit was added right after the retype already passed the quota check.

It is possible that while doing an operation on a resource the code flow doesn’t complete in an unexpected way leaving leftover reservations in the database, for example:

A coding bug in Cinder that leaves the volume in an unexpected status.
Service kill.
Node restart or shutdown.

For these situations the new quota system will add code to the os-reset_status REST API action on volumes to automatically clear any reservations that the volume may have when the status is changed, which is what happens when a volume is stuck in extending, retyping, etc. This way there is no need to wait until the reservation expires and the operator can do the cleanup in an easy way without needing additional API calls.

On volume deletion the code will also clear any existing reservations on the specific volume.

To facilitate the cleanup of these reservations the volume’s id will be used as the uuid field for all the reservations, instead of creating a random one, regardless of the value of the resource field in the reservations table.

Both drivers will create reservations the same way to facilitate switching the drivers without having usage numbers go out of sync.

Changing configuration

There are 2 Cinder configuration options that are crucial for the new quota system to operate correctly: quota_driver and no_snapshot_gb_quota.

The no_snapshot_gb_quota configuration option is used to determine whether snapshots should be counted towards the volume quota or not, so this is not something we want to be counting in some places and not counting in others; we want a consistent behaviour through all the Cinder services, which means that they must have the same value.

Currently Cinder has no way of enforcing the same value for the no_snapshot_gb_quota, and what’s worse, it cannot even know when the current quota calculations have become invalid because this configuration option has changed (Bug #1952635).

This is something we definitely don’t want in the quota system, and with the new quota system we have bigger problems, because it’s not only no_snapshot_gb_quota that can be changed, but also quota_driver, and changing the quota driver means that a quota system may need to recalculate things to ensure that it starts operating with the correct quota assumptions. For example when changing from the DynamicQuotaDriver to the StoredQuotaDriver all the counters in the DB will be wrong, so the StoredQuotaDriver needs to calculate the counters before it can start working or the whole quota system will not operate correctly.

These configuration options are not the kind of things that are frequently changed, and we expect most deployments to never have to change them at all, but Cinder should still provide a way for them to be safely changed since one of the cases we expect to happen is a deployment outgrowing the usefulness of the DynamicQuotaDriver and running into performance issues. In that case they will want to switch to the StoredQuotaDriver.

To support changing configuration option changes to the quota system there are 3 things that the new quota system needs to be able to do:

Detect changes in configuration options.
Signal drivers that the no_snapshot_gb_quota configuration option has changed and for driver to react to this change.
Signal drivers that they were not the quota driver that was running on the last start and they should see if they need to do some calculations.

To detect changes to these configuration options, a new global_data table will be created to store the currently used configuration values. This table will be used to signal quota drivers when things have changed.

A system administrator will have to follow these steps to change any of these 2 configuration options:

Stop all Cinder services.
Change the cinder.conf file in all the nodes where a Cinder service is running.
Run a cinder-manage command to apply the changed options.
Restart Cinder services.

The cinder-manage command will not only trigger the quota system recalculations, but it will also make the appropriate DB changes in the global_data table to reflect the new configuration options that are in effect.

Since we cannot allow Cinder services to run with mismatching configuration options they will fail to start if the quota configuration options from the database don’t match the one that the service has. This will prevent system administrators making an error and only realize it after their whole system has some crazy quotas.

Please see the Changing configuration alternatives for other possible mechanism to the one proposed here.

Interface

Here is the proposed interface for the new quota system drivers:

NAME

Unique string of maximum 254 ASCII characters that identifies the driver.

init

def __init__(self, driver_switched, no_snapshot_gb_quota_toggled):

Initialization method for the quota driver where the driver_switched parameter indicates whether the last run was done using the same Quota driver or if a different one was used and this is the first run with this one.

This is important because switching to the StoredQuotaDriver from the DynamicQuotaDriver means that in-use and reserved counters need to be recalculated since they could be out of sync or missing altogether.

This effort is going to focus on only supporting these 2 quota drivers and avoid unnecessary complexity, because if we wanted to support other kind of drivers that were not based on the Cinder database we would need to add a more complex mechanism, since the cinder code would need to notify drivers when the limits are changed in the DB and there would need to be a way for Cinder to request information from the old quota driver, such as current reservations, when switching.

The interface can be enhanced if a future quota driver finds it insufficient.

The no_snapshot_gb_quota_toggled parameter indicates whether the option has changed since the last run. This is important for the StoredQuotaDriver that would need to recalculate in-use and reserved counters. This is something that doesn’t work correctly right now.

Drivers can block the Cinder database when synchronizing when the driver has been switched or the snapshot quota configuration option has been toggled, because the driver will only be called with any of the parameters set to True on a single service in the deployment and the quota will not be in use at that time by any other service.

resync

def resync(self, context, project_id):

This is only relevant for the StoredQuotaDriver, and is intended to allow the cinder-manage command request a recalculation of quotas for a specific project or for the whole deployment.

set_defaults

def set_defaults(self, context, **defaults):

Set system wide default limits.

The keys for the keyword arguments defaults are in the internal form of the quota system, that is to say, they will be gigabytes and not vol_gbs.

This will be a common implementation for both database based quota drivers, where it modifies the record if it exists and creates it if it doesn’t.

set_limits

def set_limits(self, context, project_id=None, **limits):

Set project specific limits.

The keys for the keyword arguments limits are in the internal form of the quota system, that is to say, they will be gigabytes and not vol_gbs.

This will be a common implementation for both database based quota drivers, where it modifies the record if it exists and creates it if it doesn’t.

clear_limits_and_defaults_cm

def clear_limits_and_defaults_cm(self, context,
                                 project_id=None, type_name=None):

This context manager removes, on exit, all existing per project limits, for when a project is deleted, or all type specific global defaults and per project limits, for when a type is deleted.

Parameters project_id and type_name will be used used as filter in the deletion. So if only project_id is provided then only per project entries will be deleted (in the db driver those from the quotas table), and if only the type_name is provided then only gigabytes_<type-name>, volume_<type-name> and snapshots_<type-name> resources will be removed but for per-project (quotas db table) and global (quota_classes table).

If an error occurs within the context manager the limits and defaults should not be cleared.

This will be a common implementation for both database based quota drivers.

type_name_change_cm

def type_name_change_cm(self, context, old_name, new_name,
                        project_id=None):

Context manager to make necessary modification, on enter, to system wide defaults and per project limits to account for a volume type name change.

This will rename gigabytes_<old_name>, volume_<old_name> and snapshots_<old_name> to gigabytes_<new_name>, volume_<new_name> and snapshots_<new_name> respectively in all tables.

The database change to the volume type name is called within this context manager to ensure that the quota defaults and limits stay in sync with the volume type name and we don’t change one but not the other.

This will be a common implementation for both database based quota drivers.

get_defaults

def get_defaults(self, context, project_id=None):

Returns system wide defaults for quota limits. If project_id is not None then volume types quota resources (volumes_<volume-type>, gigabytes_<volume-type>, and snapshots_<volume-type>) will be filtered based on the project’s visibility of the volume types, if project_id is None then all defaults will be returned regardless of the is_public value of the volume types.

In terms of volume type visibility, a project can view all public volume types and private ones where it has permissions (entries in the volume_type_projects table).

System wide defaults are stored in the database in the quota_classes table with the default value on the class_name.

Returned data is a dictionary mapping resources to their hard limits, and must include all volume type resources even if there is no record in the database.

In the following example of returned data the gigabytes_lvmdriver-1, volumes_lvmdriver-1, and snapshots_lvmdriver-1 are not present in the database:

{
 "per_volume_gigabytes": -1,
 "volumes": 10,
 "gigabytes": 1000,
 "snapshots": 10,
 "backups": 10,
 "backup_gigabytes": 1000,
 "groups": 10,
 "gigabytes___DEFAULT__": -1,
 "volumes___DEFAULT__": -1,
 "snapshots___DEFAULT__": -1,
 "gigabytes_lvmdriver-1": -1,
 "volumes_lvmdriver-1": -1,
 "snapshots_lvmdriver-1": -1
}

This will be a common implementation for both database based quota drivers.

get_limits_and_usage

def get_limits_and_usage(self, context, project_id, usages=True):

Get all effective quota limits for a specific project, and optionally quota usage, for a specific project. If project_id is None the one from the context will be used.

Volume types quota resources (volumes_<volume-type>, gigabytes_<volume-type>, and snapshots_<volume-type>) will be filtered based on the project’s visibility of the volume types.

A project can view all public volume types and private volume types where it has permissions (entries in the volume_type_projects table).

A quota limit values defined in the quotas table overrides global values from the quota_classes.

Returned data will always be a dictionary (or defaultdict), but the contents will depend on whether we are getting quota usage or not. Just like the get_defaults method this returns all volume type resources even if there is no record in the database.

{
 "per_volume_gigabytes": -1,
 "volumes": 8,
 "gigabytes": 1000,
 "snapshots": 10,
 "backups": 10,
 "backup_gigabytes": 1000,
 "groups": 10,
 "gigabytes___DEFAULT__": -1,
 "volumes___DEFAULT__": -1,
 "snapshots___DEFAULT__": -1,
 "gigabytes_lvmdriver-1": -1,
 "volumes_lvmdriver-1": -1,
 "snapshots_lvmdriver-1": -1
}

With quota usage returned value will look like this:

{
 'per_volume_gigabytes': {'limit': -1, 'in_use': 0, 'reserved': 0},
 'volumes': {'limit': 8, 'in_use': 1, 'reserved': 0},
 'gigabytes': {'limit': 1000, 'in_use': 1, 'reserved': 0},
 'snapshots': {'limit': 10, 'in_use': 0, 'reserved': 0},
 'backups': {'limit': 10, 'in_use': 0, 'reserved': 0},
 'backup_gigabytes': {'limit': 1000, 'in_use': 0, 'reserved': 0},
 'groups': {'limit': 10, 'in_use': 0, 'reserved': 0},
 'gigabytes___DEFAULT__': {'limit': -1, 'in_use': 0, 'reserved': 0},
 'volumes___DEFAULT__': {'limit': -1, 'in_use': 0, 'reserved': 0},
 'snapshots___DEFAULT__': {'limit': -1, 'in_use': 0, 'reserved': 0},
 'gigabytes_lvmdriver-1': {'limit': -1, 'in_use': 1, 'reserved': 0},
 'volumes_lvmdriver-1': {'limit': -1, 'in_use': 1, 'reserved': 0},
 'snapshots_lvmdriver-1': {'limit': -1, 'in_use': 0, 'reserved': 0}
}

group_check_cm

def group_check_cm(self, context, qty=1, project_id=None):

Context manager to check group quota upon context entering.

Raises QuotaError if quota usage would go over the quota limits upon adding qty new groups.

Effective quota limits are determined based on the project’s quota limits (hard_limit for the groups resource in the quotas table) if defined or the global defaults (in the quota_classes table) otherwise.

The project is determined by the project_id parameter or the context’s project_id if the optional project_id parameter value is None.

The context manager must ensure that there are no race conditions with concurrent calls to group_check_cm within different threads and processes in the node as well as across different nodes.

For the database driver this can be achieved using a SELECT FOR UPDATE on the groups quota limit which blocks other requests until the context manager exists.

Users of this context manager should try to keep the code within the context manager to a minimum to allow higher concurrency.

For the DB driver, the context manager will start a database transaction/session, making it available in the session attribute of the provided context, and this transaction will be committed if the code enveloped by the context manager completes successfully, but if an exception is raised in the enveloped code then the transaction will be rolled back. So this context manager not only checks the quotas but also provides a transaction context.

An example of using this context manager within the create method of the Group Oslo Versioned Object:

with quota.driver.group_check_cm(self._context, qty=1):

    db_groups = db.group_create(self._context,
                                updates,
                                group_snapshot_id,
                                source_group_id)

group_free

def group_free(self, context, gbs, qty=1, project_id=None):

Context manager to free group quotas upon context exiting. The DB row soft deletion of groups will be enclosed by this call.

This is only relevant for the StoredQuotaDriver that needs to decrease its counters.

backup_check_cm

def backup_check_cm(self, context, gbs, qty=1, project_id=None):

Context manager to check backup quotas upon context entering.

Raises QuotaError if quota usage would go over the quota limits upon adding qty backups or gbs backup gigabytes.

Effective quota limits are determined based on the project’s quota limits (hard_limit for the backups and backup_gigabytes resources in the quotas table) if defined or the global defaults (in the quota_classes table) otherwise.

The project is determined by the project_id parameter or the context’s project_id if the optional project_id parameter value is None.

The context manager must ensure that there are no race conditions with concurrent calls to backup_check_cm within different threads and processes in the node as well as across different nodes.

For the database driver this can be achieved using a SELECT FOR UPDATE on the backups and backup_gigabytes quota limits which blocks other requests until the context manager exists.

Users of this context manager should try to keep the code within the context manager to a minimum to allow higher concurrency.

An example of using this context manager within the create method of the Backup Oslo Versioned Object:

with quota.driver.backup_check_cm(self._context, qty=1, gbs=self.size):

    db_backup = db.backup_create(self._context, updates)

backup_free

def backup_free(self, context, gbs, qty=1, project_id=None):

Context manager to free backup quotas upon context exiting. The DB row soft deletion of the backup will be enclosed by this call.

This is only relevant for the StoredQuotaDriver that needs to decrease its counters.

vol_snap_check_and_reserve_cm

def vol_snap_check_and_reserve_cm(self, context, type_id, type_name=None,
                                  project_id=None,
                                  *,
                                  uuid=None,
                                  vol_gbs=0, vol_qty=0,
                                  vol_type_gbs=None, vol_type_vol_qty=None,
                                  snap_gbs=0, snap_qty=0,
                                  snap_type_gbs=None, snap_type_qty=None,
                                  vol_size=0):

Context manager that, upon entering, checks volume and snapshot quotas and optionally makes reservations.

Volumes and snapshot are tightly coupled resources, since a snapshot cannot exist without a parent volume, so their quota checks are handled jointly in the same method.

Raises QuotaError if quota usage would go over the quota limits upon consuming provided resources:

vol_qty number of volumes being reserved.
vol_gbs additional volume gigabytes.
vol_type_vol_qty volumes of the specified volume type. Defaults to the value of vol_qty.
vol_type_gbs additional volume gigabytes of the specified volume type. Defaults to the value of vol_gbs.
snap_qty snapshots.
snap_gbs additional snapshot gigabytes.
snap_type_qty snapshots of the specified volume type. Defaults to the value of snap_qty.
snap_type_gbs additional snapshot gigabytes of the specified volume type. Defaults to the value of snap_gbs.

Unlike the vol_gbs, vol_type_gbs, snap_gbs, and snap_type_gbs parameters, the vol_size is not an increment over existing consumption, but an absolute value representing the total size of the volume. And the context manager also raises a QuotaError exception if it is greater than the per_volume_gigabytes limit.

Effective quota limits are determined based on the project’s quota limits volumes, volumes_<volume-type>, snapshots, snapshots_<volume-type>, gigabytes, gigabytes_<volume_type>, and per_volume_gigabytes if defined in the quotas table or the global defaults defined in the quota_classes table otherwise.

The project is determined by the project_id parameter or the context’s project_id if the optional project_id parameter value is None.

The volume type name (type_name) is necessary to perform quota checks, but the method can query this information based on the type_id. Due to current Cinder behavior (where a type can be changed to private even when projects have volumes) then the quota driver needs to confirm that the project still has access to it.

Volumes and snapshots are currently the only resources that can have reservations, and this method automatically creates them when a uuid is provided. This uuid must be of the primary resource for the operation, that is to say that if we are transferring a volume with all its snapshots the reservations will pass the volume’s uuid.

Both drivers must use different entries for volume and snapshot gigabyte reservations because the no_snapshot_gb_quota_toggled configuration option may be changed and the service restarted before a transfer is accepted, and the StoredQuotaDriver will need to make a decision both when recalculating (if driver has changed) and on transfer accept.

This context manager must ensure that there are no race conditions with concurrent calls to vol_snap_check_and_reserve_cm within different threads and processes in the node as well as across different nodes.

For the database driver this can be achieved using a SELECT FOR UPDATE on the volumes, volumes_<volume-type>, snapshots, snapshots_<volume-type>, gigabytes and gigabytes_<volume_type> quota limits which blocks other volume and snapshot requests until the context manager exists.

Users of this context manager should try to keep the code within the context manager to a minimum to allow higher concurrency.

When creating reservations the context manager must ensure that they are cleaned up if an exception is raised within the context manager. For the DB driver the context manager will start a database transaction/session, making it available in provided context, and will commit everything on normal context manager exit and roll everything back, including the reservations, when an exception is raised.

An example of using this context manager within the create method of the Volume Oslo Versioned Object:

with self.quota_check(self._context, self.volume_type.id,
                      volume_type and volume_type.name,
                      vol_gbs=self.size,
                      vol_qty=1,
                      vol_size=self.size):

    db_volume = db.volume_create(self._context, updates)

Where quota_check is a property that takes into account whether the volume uses quota or not:

@property
def quota_check(self):
    if self.get('use_quota', True):
        return quota.driver.vol_snap_check_and_reserve
    return self.nullcontext

vol_snap_free

def vol_snap_free(self, context, type_id, type_name=None, project_id=None,
                  *,
                  vol_gbs=0, vol_qty=0,
                  vol_type_gbs=None, vol_type_vol_qty=None,
                  snap_gbs=0, snap_qty=0,
                  snap_type_gbs=None, snap_type_qty=None):

Context manager to free volume and snapshot quotas upon context exiting.

This is only relevant for the StoredQuotaDriver that needs to decrease its counters.

reservations_clean_cm

def reservations_clean_cm(self, context, resource_uuid, commit=True):

Context manager that cleans all reservations, committing or rolling back, for the given uuid on exit.

The uuid is the “primary” uuid of the operation and it won’t be a different uuid for each resource that has been reserved. E.g. when accepting a volume transfer with its snapshots, all reservations will use the volume’s id.

For the DynamicQuotaDriver this is mostly just deleting the entries from the database, but for the StoredQuotaDriver it needs to adjust the in-use and reserved counters.

These counters may be from different projects, for the transfer of volumes, so the context’s project_id will be ignored.

The DynamicQuotaDriver driver must also take into account the no_snapshot_gb_quota_toggled configuration option when committing a transfer, because the snapshot reservations are stored in different row entries in case the option is changed and the service rebooted before a transfer is accepted.

Differences

There are some differences between the new and old system that are worth highlighting:

Resource consumption rules stated in the Resources section of this spec are absolute, so it doesn’t matter if a volume becomes in error status because scheduling failed or because the driver call on the volume service failed. If there is a quotable database record, then it will be counted towards the quota.
Negative reservations, created when retyping a volume, won’t be taken into account in usage calculations, because like we mentioned before we want the volumes_<volume-type> and gigabytes_<volume-type> of the source type to still be consumed while we do the operations, since we don’t know if we’ll succeed or not, and on failure we would need to consume them again.
The new quota system drops the illusion that Cinder can support multiple ORM systems and accepts the fact that Cinder is tightly coupled with SQLAlchemy and MySQL/InnoDB (this is not new, there is already a patch proposed that removes the intermediate layer, so instead of having all the quota code in cinder/db/sqlalchemy/api.py it will be under cinder/quota, including all the database queries.

This approach has the downside of having DB code in multiple places, with potential code duplication, but on the other hand it has the great benefit of having the quota code contained in fewer files and using less memory for custom quota drivers (currently the standard quota driver file is always loaded even if it’s not instantiated).
All deployments will use the default quota class instead of supporting the already deprecated configuration file quota limits.
The new quota system fixes a number of existing bugs, so there are some undesired behaviors that change:
- Now listing quota limits and quota usages won’t show private types the project doesn’t have access to (bug #1576717).
- A limit of 0 will be shown if a project has resources for a type we no longer have access to because it was made private after the resource was created (related bug #1952456). This can also happen if an admin creates a volume for a private type that the project doesn’t have access to.

Limitations

Spec is aimed at these 2 drivers, so additional drivers may not be easy to add. Though it shouldn’t be a problem if these 2 work as expected.
There is a bottleneck in concurrent code execution, because the code locks on the system wide defaults, which are common to all projects. So, even if the critical section code enveloped by the check context managers is very small, it will still limit to only 1 context entering at a time for the whole deployment for the given quota limits. As an example, if we are concurrently creating 100 volumes in as many projects, they will be happening mostly in parallel, but once they reach the point to check quota limits and create the DB record they will be serialized.
Race condition on the retype operation as explained in the reservations section.

Alternatives

Work with what we have

Some of the alternatives include:

Carefully go through the Cinder code looking for potential error causes and fixing them.
Refactor existing quota code to move part of the Quota Python logic into database queries.
Refactor existing code to reduce spillage of the quota system implementation details all over the code and reduce the usage of reservations to only the strictly necessary cases.

These alternatives have the same underlying issue as the current implementation, where it would be hard to tell whether we have resolved all the issues or not, and upon encountering another out of sync case in a deployment we would be, once again, in a position where we cannot tell how we reached that point.

Unified Limits

Another alternative is to use the KeyStone Unified Limits. At first glance this may seem like a perfect solution since it:

Allows for a unified limit system in all OpenStack (once all projects implement it).
Supports different enforcement models, including hierarchies.

But upon closer inspection it’s not without its disadvantages:

While Glance and Nova implemented its usage in the Yoga release this still cannot be considered a proven solution since there have not been enough time for users to actually evaluate it.
The Unified Limits system does not have any mechanism to prevent race conditions between concurrent operations. So we’ll have to implement our own mechanism that needs to work across all the Cinder services. It can be with a DLM, some database locking, or how Nova is going to do it, which is to check the limit, commit claim, then check limit again and revert if over usage is detected. The Nova mechanism means that we are always doing a double check and sometimes a revert, and we can even get a false failure due to a double race condition on the checks (2 concurrent requests pass the initial check and then both fail on the confirmation check, whereas only 1 of them on its own would have succeeded).
The oslo.limit project will fail a limit check if the limit has not been previously registered in KeyStone, which is the opposite of how our quota system currently behaves, as it assumes unlimited (-1). This means that Cinder will either have to manage the registrations of the limits when we create or destroy volume types, when a project is given access to a volume type, when a volume type’s public status changes, etc. or to force operators to manage all this on their own. A more reasonable alternative would be to modify the oslo.limit project to support alternative behavior on non defined values.
It will be slower since we have to call an external service, KeyStone, for the limit check, which has to check the user making the call, go to the database, etc. And because each of the resources that are checked requires its own REST API call to KeyStone

This could be improved in Keystone to allow multiple simultaneous checks.
Hierarchical support using the Strict Two Level enforce mechanism isn’t implemented in oslo.limit

Fix bottleneck

As mentioned before, in the proposed quota system there is a bottleneck in the concurrent code execution due to the DB locking, because it is locking using entries from the quotas table which are shared among all projects.

To resolve this bottleneck using the DB locking we would need to duplicate the system wide defaults. These entries can be duplicated in the quotas table or in the quota_classes table.

If they are duplicated into the quotas table, then a new column would need to be added (is_default) to flag the contents as being defaults or not. Because when a global default from the quota_classes table is changed it would need to be changed in the quotas table records that have the default values but not on the records that have been explicitly set, even if they have the same value as the default.

If the quota_classes table is used, then we would store the project_id into the name column, which means that we would have trouble in the future if we ever wanted to fully implement the Quota Classes concept. Though this is unlikely given how long it has been since the table was created and how the concept has never been implemented.

When setting a global default quota limit we would need to remove the restriction on the name column being default when using the quota_classes and if we used the quotas table we would need an additional query besides the one to the default quota_classes record, as we would need to update non deleted records from the quotas table for that resource that have the is_default value set to true.

Another thing to consider is that Cinder doesn’t know beforehand what projects exist, so it will also need to dynamically duplicate the global default quota records if they are not present when the first operation on a project is called. This can be done efficiently, only incurring into additional queries on the first request for a project, by assuming the values exist and querying with locking on them, and only if the result is missing the values we go and duplicate the global defaults.

This dynamic duplication is also tricky, because we don’t want to have races with a global quota limit update request or with other operation that triggers the same duplication. These 2 races can be prevented with a SELECT ... FOR UPDATE on the quota_classes table.

At the time of writing, we are hoping that the bottleneck is not significant enough to warrant the extra effort of removing it. If time proves us wrong we can go and implement one of these or other solution.

Changing configuration alternatives

The Changing configuration subsection of the Proposed change section presented the chosen mechanism to change quota_driver and no_snapshot_gb_quota configuration options, but those are not the only possibilities.

This subsection presents 2 alternatives to make changes to the options and ensure that all Cinder services run with the same configuration option values:

Build a complex system to orchestrate the change on running services: Signal the change to all Cinder services and make sure they complete ongoing quota operations before signaling the quota driver that it needs to do recalculations, then signal services that they have to reload the quota driver and finally continue their operation.

Implementing this is quite complex, starting with how difficult is to make sure that there are no services that have missed the notification: A service may have a temporary loss of connection to RabbitMQ or the DB.

We also have the difficulty of hitting pause in all running operations among others.
Only allow changing the configuration option when all services are down. Cinder services would be smart enough to detect on start that the configuration has changed and confirm that they are the only service that is currently running and can proceed to tell the quota driver that it needs to do the recalculations.

We find multiple challenges when trying to make Cinder smart enough to detect that there are no other services running:
- We have no way of knowing if a Cinder API service is running or not because they don’t issue DB heartbeats and they don’t receive any RPC calls via RabbitMQ. We can make them issue DB heartbeats and we may even make them listed on a message queue for RabbitMQ messages.
- When starting all Cinder services at the same time they need to avoid racing to tell the driver to recalculate the quotas. This can be resolved locking for update a DB row to prevent the race and allowing only 1 service to do the calculation.
- Even if all services are brought down, the DB heartbeat for the services won’t timeout for a while, so Cinder will have to wait until those heartbeats time out. This introduces an unnecessary delay on the Cinder restart.
- Spurious/temporary network issues that may make Cinder think that there are no running services. This is actually the biggest issues.

Since changing the quota configuration options is not something that’s going to be frequently changed, we actually expect it to be change once at most, passing the responsibility of doing it right to the system administrator seems like the best choice.

In any case this is not something that has to be like that forever. If this behavior becomes a real problem we can change it in the future.

Data model impact

The new quota system will place greater importance on the database queries and reduce the Python code, so the database needs to be ready to perform counting queries efficiently.

The main change to ensure efficiency will be adding the proper indexes to resource tables. The index that we’ll need to have is one on the project_id and deleted columns for the following tables (that currently don’t have them):

volumes
snapshots
backups
groups

This changes will bring additional benefits to the Cinder service, because right now listing resources on a deployment with many projects or with many deleted resources is not efficient, because the database has to go through all the resources to filter out non delete rows that belong to a specific project (bug #1952443).

Existing quota_usages tables will no longer be used, and it will be removed on the next release after the rollout of the new quota system.

The reservations table will remain in use, although only for a couple of operations.

Additionally we’ll need to track the current quota driver that is being used as well as the no_snapshot_gb_quota configuration option to be able to tell the quota drivers if they have changed.

For this purpose the proposal is to create a new table that can store global cinder information.

Table global_data will have the following fields:

created_at: When this key-value pair was created
updated_at: When this key-value pair was last updated
key: String value describing the value. For example no_snapshot_gb_quota or quota_driver.
value: String with the value of the key. For example true or StoredQuotaDriver.

REST API impact

There will be no REST API impact, because we currently only expose the usage and reservations (quota_usages table) through a listing API that we’ll still be able to provide with the current quota driver interface, and we still have reservations (even if fewer operations use them) so that information still remains relevant and we don’t need to remove it from the response.

Security impact

None.

Active/Active HA impact

None.

Notifications impact

No notification impact, since no new operations are added or removed.

Other end user impact

End users of the Cinder service should not have significant impact, except for how the quota is counted.

Current behavior can be erratic on how quota is counted, as it will depend on how and where things fail, so we can have volumes in ERROR status that have been counted towards quota and others that have not.

With this new approach the quota consumption rules will be very straightforward, users/admins just need to list resources and add all that have the consumes_quota field set to true to check if usage is correct.

This stable behavior, can have a positive impact on a deployment regardless of how services are going down, since end users will not be able to go over their allowed quota and be forced to clean failed resources instead of being able to leave them be.

Performance Impact

Some preliminary code was prototyped for the volume creation and get usage operations to evaluate the performance of the different quota drivers: the old, the new StoredQuotaDriver, and the new DynamicQuotaDriver.

The results showed that the new StoredQuotaDriver system was twice as fast as the old code in both operations, and the DynamicQuotaDriver was slower than the StoredQuotaDriver, as expected, but faster than the old one until there are around 26000 resources per project.

So the DynamicQuotaDriver is less likely to be out of sync with reality because it doesn’t store fixed values, but the StoredQuotaDriver has better performance, and that’s the reason why both drivers are going to be implemented, to allow system administrators decide which one is better for them.

The default driver will be DynamicQuotaDriver to prioritize the usage values always stay in sync, and large deployments or those looking for best performance will be able to use the StoredQuotaDriver.

Deployments may even start with one quota system and then switch to the other if necessary.

Other deployer impact

As soon as the new code is deployed and executed the new quota system will be used, there will be no backward compatibility support for old quota code.
Deployments using a custom external quota driver will no longer be able to start. This should not be a problem as we believe there is nobody using a custom driver.
During rolling upgrades the quota system will be more fragile than usual, and users may be able to go over quota.
New quota system will no longer have an internal brute force cleaning mechanism of quotas, the volume state change API will be used to clean reservations, and the cinder-manage quota sync command will be used for the StoredQuotaDriver, so the following configuration options will be deprecated and will no longer have any effect: reservation_expire, reservation_clean_interval, until_refresh, and max_age.
Configuration option use_default_quota_class will be deprecated, because all deployments will use the default quota class instead of supporting the already deprecated configuration file quota limits (related bug #1609937).

Developer impact

There should be a positive impact on Cinder developers, since the code should be more readable without all the quota code in between the higher level logic, and adding new code should not require touching the quota manually.

The new code will probably break the cinderlib project, so changes to the project will also be necessary.

Implementation

Assignee(s)

Primary assignee:: Gorka Eguileor (geguileo)
Other contributors:: Rajat Dhasmana (whoami-rajat)

Work Items

As discussed in the PTG/mid-cycle this work may be split in 2 phases that may be implemented in different releases:

Phase 1: `DynamicQuotaDriver`

Deprecate configuration options and log warnings for deployments that are using custom quota drivers.
Add required indexes to volumes, snapshots, backups, and groups tables.
Add missing backup and backup_gigabytes default quota limits to the quota_classes table.
Remove deprecated consistencygroups resources from the quota_classes, quotas, quota_usages and reservations table.
Write the DynamicQuotaDriver database quota driver.
Make the following operations use the new quota driver:
- Create volume
- Delete volume
- Manage volume
- Extend volume
- Retype volume
- Transfer volume
- Create snapshot
- Delete snapshot
- Manage snapshot
- Backup create
- Backup restore
- Group create
- Group delete
Remove code for the old quota driver.
Make the cinder-manage quota sync and check be noop.
Write the DynamicQuotaDriver database quota driver unit tests.
Update existing unit tests.
Write initial documentation and mention that a more efficient driver will be coming in the future.

Phase 2: `StoredQuotaDriver`

Write the StoredQuotaDriver database quota driver.
Write the StoredQuotaDriver database quota driver unit tests.
Update the cinder-manage quota sync and check commands.
Add the cinder-manage quota change command.
Do basic manual performance comparison of old and new quota system.
Add support for the new quota system to cinderlib with a noop quota driver and use it as the default value of the quota_driver configuration option.
Update documentation.

Dependencies

The database engine cannot lock on non-existent rows, so the new code needs the database to hold default quota limit records for all the basic resources in the quota_classes table. So this new code depends on us ensuring that the backups and backup_gigabytes records are present in the database and we should also have the consistencygroups removed since they haven’t been used for a long time (bug #1952420).

Testing

Besides some manual testing that will be performed to do some basic performance comparison between the old and the new quota system, most of the testing will be focused on the testing of the SQL queries.

Currently supported database engines are InnoDB and SQLite, and the second one has some limitations and quirks that may make testing some of the queries difficult or impossible, so some of the unit tests will be skipped on SQLite.

We may explore the possibility of running a tempest job that checks the quota usage after finishing the tempest run and reports if it has become out of sync.

Documentation Impact

The Cinder quota documentation will be updated to reflect how resources will be tracked now, to contain description and use cases of the different quota drivers, as well as the procedure to change the quota driver driver and the no_snapshot_gb_quota_toggled configuration option.

References

In the Yoga PTG it was accepted that the quota system will be flaky during rolling upgrades.
WIP patch showing how the new quota system could look like

Related bugs:

Backup creation quota warning in logs: bug #1952420.
Remove default_quota_class configuration option: bug #1609937
Warnings when creating volumes and snapshots with a type that doesn’t have values in quota_classes: bug #1435807.
Inneficient listing of resources: bug #1952443
Showing quotas for private types: bug #1576717.
Incorrect limit for private volume types: bug #1952456
Incorrect usage when changing no_snapshot_gb_quota: bug #1952635

Default volume type overrides

Wed, 23 Aug 2023 00:00:00

https://blueprints.launchpad.net/cinder/+spec/multiple-default-volume-types

Cinder’s current default volume type options are limited and are insufficient for big and/or complex deployments.

This spec proposes adding per project volume types.

Problem description

On complex and/or big OpenStack deployments we may have multiple availability zones (AZs) with many projects, increasing the complexity of properly structuring and managing resources.

One of the problems we encounter when having multiple AZs is that we usually want our projects to have volumes and instances collocated within the same AZ, but we can only define one default volume type, forcing our users to always remember to create volumes with a volume type (if it has the AZ defined in it) or to explicitly pass the AZ.

This is error prone, and users forgetting it will lead to unwanted cross AZ attachments, with the consequent performance issues and network throughput costs.

Right now there are 2 workarounds that users are adopting to mitigate this situation:

Disabling cross AZ attachments in Nova: For deployments that don’t want to do cross AZ instance live migrations, they can disable cross AZ attachments in Nova so the attachment to the instance fails, thus preventing performance issues on running VMs. This solution usually forces users to recreate the volume in the right AZ.
Unusable default volume type: Setting a default volume type with a self explanatory name, such as Need_To_Select_Volume_Type and setting it in a way that will fail the scheduling and leave the volume in an error state. This is less wasteful, but still not a nice user experience.

This spec addresses this specific problem.

Use Cases

On a country wide OpenStack deployment that spreads across multiple data centers where each one has its own storage array and OpenStack project, an administrator wants to make sure that volumes are created in their own data center by default. For this purpose the admin defines a per project default volume type where it has the appropriate extra specs.
On an OpenStack deployment with 3 different volume types (high performance, replicated, compressed) and different department (each with a different project) have different requirements, so the system admin wants to assign a different OpenStack volume type to each one of them.

Proposed change

The proposal is to add the possibility to override the existing Cinder default Volume Type on a per project basis to make it easier to manage complex deployments.

With the introduction of this new default volume type configuration, we’ll now have 2 different default volume types. From more specific to more generic these are:

Per project
Defined in cinder.conf (defaults to __DEFAULT__ type)

So when a user creates a new volume that has no defined volume type (explicit or in the source), Cinder will look for the appropriate default first by checking if there’s one defined in the DB for the specific project and use it, if there isn’t one, it will continue like it does today, using the default type from cinder.conf.

Administrators and users must still be careful with the normal Cinder behavior when creating volumes, as Cinder will still only resort to using the default volume type if the user doesn’t select one on the request or if there’s no volume type in the source, which means that Cinder will not use any of those defaults if we:

Create a volume providing a volume type
Create a volume from a snapshot
Clone a volume
Create a volume from an image that has cinder_img_volume_type defined in its metadata.

By default the policy restricting access to set, unset, get or list all project default volume type will be set to system admins only.

Alternatives

An alternative would be allow setting default volume types overrides on a per user and per project basis, as well as globally.

Such alternative was discarded, given the added complexity and effort needed to implement to provide very little benefits, as configurable per user and global default volume types don’t seem to be a user necessity.

Data model impact

We will add a new table called default_volume_type that will have 3 fields:

id primary integer key.
volume_type_id foreign key string of length 36.
project_id string.

We won’t reuse existing volume_type_projects table and just add an is_default field because then the combinations of private/public volume types and defaults get complex as we update or remove them.

In terms of database migrations we’ll just have to create the table.

REST API impact

We’ll need a new set of REST API calls to provide the CRUD operations:

Set the default volume type (create or update) for a specific project.

Besides ensuring that the volume type exists and the project has access to it, this method will also call keystone to validate that the project exists.
- PUT /v3/default-types/{project_id}
  - JSON body to set default type (by name or id) for a project:
    { "volume_type": "lvm" }
  - Response Codes:
    Success - 200 (with body)
    
    { "project_id": "248592b4-a6da-4c4c-abe0-9d8dbe0b74b4", "type_id": "f8a82360-0b1a-4649-8615-114341dd06e0" }
    
    Error - 400 (volume type not found), 404 (project id not found)
Unset a default volume type. It will fail if there is not a default volume type for the given project or if the project no longer exists in keystone.
- DELETE /v3/default-types/{project_id}
  - Response Codes: - Success - 204 - Error - 404 (project id not found)
Get volume types
- Show default type for a project GET /v3/default-types/{project_id}
  - A request with project_id will return a single JSON object containing the project_id and type_id of the specified project. Returns 404 error code if the entry is not in the DB, independent of the existence of the project in keystone.
    { "project_id": "248592b4-a6da-4c4c-abe0-9d8dbe0b74b4", "type_id": "f8a82360-0b1a-4649-8615-114341dd06e0" }
  - Response Codes: - Success - 200 (with body) - Error - 404 (project id not found or no default type)
- List all default types GET /v3/default-types
  - A request without a project_id will return all the volume type defaults defined. A sample response would be:
    [ { "project_id": "248592b4-a6da-4c4c-abe0-9d8dbe0b74b4", "type_id": "f8a82360-0b1a-4649-8615-114341dd06e0" }, { "project_id": "1234567-4c4c-abcd-abe0-1a2b3c4d5e6ff", "type_id": "5c4df055-571e-4430-9823-416b82f337b2" } ]
  - Response Codes: - Success - 200 (with body)
    
    Notice that we only list overrides, we won’t return the value of default_volume_type.

A user can get its effective default type using existing cinder type-default command: GET /v3/{project_id}/types/default.

Security impact

None

Active/Active HA impact

None

Notifications impact

None

Other end user impact

There will be a new set of commands in the python-cinderclient to match the new REST API endpoints:

Set default: cinder default-type-set <type-name> <project-id>
Unset default: cinder default-type-unset <project-id>
List defaults: cinder default-type-list [--project <project-id>]

Performance Impact

Create volume operations that don’t define a default volume type (explicitly or via a source) will have a tiny performance impact since we’ll add a DB query to get the defaults.

Other deployer impact

None

Developer impact

We should no longer refer directly to the default_volume_type configuration option throughout the code and instead use the get_default_volume_type method from cinder.volume.volume_types.

Implementation

Assignee(s)

Primary assignee:: <whoami-rajat>

Work Items

Cinder service
- Check if caller is authorized to do the operation: First we’ll check the normal policy to see if it’s a system admin, etc, but then we’ll have to check the project, and we’ll only authorize the operation if caller’s context has system scope.
  
  For this we have introduced a new policy to check if the caller is a system admin and then leverage the get_project_hierarchy method in cinder.quota_utils to validate that the project actually exists (since the method does a get of the project).
- Add the DB field and the DB migration.
- Add 3 DB layer methods:
  - Set default volume type: Given a volume type id and a project id this method sets its default volume type.
    
    It will try to update the volume_type_id for the project_id and if it fails because the row doesn’t exist, then it will create the DB row.
  - Unset default volume type: This will set deleted and deleted_at fields for the project_id. If it fails because it doesn’t exist, the failure will not be propagated and it will be considered a success.
  - Get projects default volume type: Returns a list of project and volume type ids limited by the project_id if it is provided.
- Update get_default_volume_type to return the effective volume type for the current project. Basically calling the get project default type DB method, and if it returns None, then we’ll continue with the current code we have to use the one from the config.
- Updating the volume type methods to ensure we don’t try to delete a volume type that is used as a default, and making sure we don’t set as private a volume type that a project is using as a default, and such operations.
- Ensure that purge_deleted_rows from cinder.db.sqlalchemy.api works as expected.
- Add a new API microversion and implement the 4 REST API methods.
- Write appropriate unit-tests for the DB methods, REST API methods, and update existing tests for the changes we introduced.
Cinder client: Add the 3 commands mentioned earlier in the Other end user impact section.
Tempest tests: Add tempest tests as describe in the Testing section.
Documentation as describe in the Documentation impact section.

Dependencies

None

Testing

Besides writing the appropriate unit-tests for the DB methods, REST API methods, and update existing tests for the changes we introduced, we also need a series of Tempest tests to test existing functionality.

Confirm that the priority of the default types are observed:
- Admin creates a custom volume type and sets it as the project’s default
- Create an empty volume with a normal user and check that the volume type is the one we created, then delete it.
- Create another volume with the admin user and see that it works the same.
- Create an empty volume with the alternative project admin user and confirm it doesn’t use our custom volume type.
- Unset the custom volume type we set in step 1.
- Create an empty volume and check that the volume type is not the custom one.
Confirm that cinder type-default works as expected:
- Admin confirms that there are no default overrides for the project and alternative project.
- Get current default using type-default.
- Create 2 custom volume types: #1 and #2
- Set default volume type #1 for project and #2 for alternative project.
- Request the type default with both projects and confirm we get the ones we have just set.
- Unset the custom volume types.
- Get current default using type-default and confirm is the same as in step 2.
Confirm that listing/showing default type overrides works as expected:
- Admin confirms that there are no default overrides for the project by first listing the default-types, and then by showing the default type for a project (we’ll get 404), for both the normal project and the alternative.
- Create 2 custom volume types: #1 and #2
- Set default volume type #1 for project and #2 for alternative project.
- Admin lists all default volume type and validates them.
- Admin gets default volume type for project and confirms that it only gets that one.
- Repeat previous 2 steps for the alternative project.
- Unset the default types.
- Confirm that default type list returns empty list.
- Confirm that showing default for a project id returns 404.
- Show default for a fake project id and confirm we get 404 error code.

Documentation Impact

A description of the default volume types overrides behavior will be added to the admin section, for example on path doc/source/admin/default-volume-types.rst.

This file will be linked from doc/source/admin/index.rst and doc/source/configuration/index.rst since it is part of the configuration but it’s also an administrative task.

The new CLI commands will be listed and explained in doc/source/cli/cli-manage-volumes.rst with examples of the new CLI commands.

Additionally the new REST API calls will need to be documented in api-ref/source/v3/default-volume-types.inc and samples added to api-ref/source/v3/samples.

And the microversion history will need to be updated in cinder/api/openstack/rest_api_version_history.rst.

References

Untyped volume to default volume type

Dedicated backup status field for volumes

Wed, 28 Dec 2022 00:00:00

This spec proposes to introduce a new backup_status field for volumes to remove the blocking or serialization that active backup tasks impose on volume attachment workflows.

Problem description

Currently all cinder tasks use the status field to check for suitable volume status while performing any particular operation. During the active phase of a task, its status is also held and updated via the same volume status field. And finally also certain errors thrown by a task are communicated back via this field. This single field in essence creates a locking or synchronization mechanism to have only one task act on a volume at any one time.

While this is helpful for coordinating tasks affecting the volume itself, applying the same logic to backups is actually not required or helpful:

Actions on the volume itself (such as resize or attaching) and backups backups (backing-up) not techically relate to each other. The backup-driver and block device-driver act independently and backups are read off a volume snapshot or, in case the config option backup_use_temp_snapshot is set to false, a clone of the volume.
A backup might take quite a long time to finish and is blocking any other task for the volume in the meantime. If we assume a 8TB volume is being backed up in full and even if the backup was running at 1 GB/s the volume backup will still take ~2.5 hrs to complete. Decoupling this from a state machine perspective (since it already is for most drivers or implementations which work via snapshots) seems to be quite beneficial.

Use Cases

There are two sides to the use-cases of decoupling backup tasks from other tasks on a volume:

1. For cloud operators - Currently detaching volume from source compute node and attaching it to the destination compute node during an instance live-migration is blocked by a concurrently run volume backup. To make matters worse, the potentially long running backup (task) could also have been triggered by the user and then be blocking administrative actions such as the live-migration of all instances to others hosts to take a hypervisor down for maintenance.

Provided sufficiently large volumes or slow backup transfer rates this could cause users to “lock out” administrative tasks indefinitely.

2. For cloud users - Urgent operational tasks are blocked by an active backup task currently. A running backup blocking the users ability to quickly resize a volume that is running low of available space or attach it to another instance. This issue is worsened if the backup tasks is even triggered by the cloud provider or some automatic scheduling.

Proposed change

For volumes there shall be field backup_status to hold the backup related values currently stored in status:

‘backing-up’
‘error_backing-up’
‘restoring-backup’
‘error_restoring’

(The backup_status certainly can also be null, to indicate there currently is no backup related status for this volume.)

Those shall then be removed from the list of (valid) values of status and be values for backup_status.

There will be some changes to the conditional checks for certain tasks to be started, but most usually depend on those volume status values that would remain in the status field anyways.

Alternatives

There was a discussion around a spec [1] moving all task status to a new field. This ended up being way too complex and not really suitable for the described use-cases of decoupling volume backups.

Data model impact

An additional field backup_status would have to be added to the volume table, together with a change to the list of valid values it might hold.
This will be introduced via a schema change to the database first, to add the new field.
This change is then followed by an online update/upgrade to split up the “moved out” status related to backups in their newly dedicated fields.
The valid values for volume status would then also have to be reduced. https://opendev.org/openstack/cinder/src/commit/5c23c9fbe41baef22a71eac4406fd9db269d1271/cinder/objects/fields.py#L168-L190 As the status backing-up, error_backing-up and are only to be stored in the backup status.
The method conditional_update needs to support different versions for the volume data model during the update process of the database. In addition all calls to the method in the cinder project have to be updated to use the new field.

REST API impact

Due to the introduction of a new field and the following split up of the status field for a volume a new API microversion is required. To serve older microversions a mapping between the two data-models has to be introduced. In essence the API has to present the status from both fields (status, backup_status) in the (previously used) single status field when older microversions are used.

The additional field of a backup_status would need to be sent to the user. The valid status of a volume would only allow the reduced set. In addition endpoints the reset_status_backup action needs to be adapted to the new data model to allow (admin only) setting a backup status.

NOTE: The list of endpoints to be changed is based on the current proposed change and is subject to change.

Show volume

{
  "volume": {
      "attachments": [],
      "availability_zone": "nova",
      "bootable": "false",
      "consistencygroup_id": null,
      "created_at": "2018-11-29T06:50:07.770785",
      "description": null,
      "encrypted": false,
      "id": "f7223234-1afc-4d19-bfa3-d19deb6235ef",
      "links": [
          {
              "href": "http://127.0.0.1:45839/v3/89afd400-b646-4bbc-b12b-c0a4d63e5bd3/volumes/f7223234-1afc-4d19-bfa3-d19deb6235ef",
              "rel": "self"
          },
          {
              "href": "http://127.0.0.1:45839/89afd400-b646-4bbc-b12b-c0a4d63e5bd3/volumes/f7223234-1afc-4d19-bfa3-d19deb6235ef",
              "rel": "bookmark"
          }
      ],
      "metadata": {},
      "migration_status": null,
      "multiattach": false,
      "name": null,
      "os-vol-host-attr:host": null,
      "os-vol-mig-status-attr:migstat": null,
      "os-vol-mig-status-attr:name_id": null,
      "os-vol-tenant-attr:tenant_id": "89afd400-b646-4bbc-b12b-c0a4d63e5bd3",
      "replication_status": null,
      "size": 10,
      "snapshot_id": null,
      "source_volid": null,
      "status": "creating",
      "backup_status": null,
      "updated_at": null,
      "user_id": "c853ca26-e8ea-4797-8a52-ee124a013d0e",
      "volume_type": "__DEFAULT__",
      "provider_id": null,
      "group_id": null,
      "service_uuid": null,
      "shared_targets": true,
      "cluster_name": null,
      "volume_type_id": "5fed9d7c-401d-46e2-8e80-f30c70cb7e1d",
      "consumes_quota": true
  }
}

Update volume
- return additional field for the backup_status
List volumes
- show additional field for the backup_status
- additional filter for the backup_status
List detailed volumes
- show additional field for the backup_status
- additional filter for the backup_status

Additionally (only available with the use of the new micro version)

Set volume backup_status
Unset volume backup_status

Security impact

None

Notifications impact

None

Other end user impact

Performance Impact

While not a performance impact per se, having the backup state-machine decoupled from the status will reduce the serialization of tasks happening for a volume.

Other deployer impact

Developer impact

The state of a volume can only be set to the reduced list of status. In addition the backup status can now only be set or unset via the introducted backup_status field.

The change above has the impact that all methods have to be checked which allow concurrent interaction with a volume and use the backup_status field instead of the status field to indicate a running process. This change should be communicated to other developer teams that rely on the cinder api to check on the status to either use an old microversion or update to use the status and the backup_status.

Because the status is currently used as a locking mechanism to prevent actions to start if an invalid status is reached, the method calls in the api have to be updated to also include a check for the backup_status if necessary. Some of the work here is currently done by the conditional_update method which needs to receive support for the updated volume model. This versioning is only needed if an old database model is received. The API always sends the new model even if an older API version is used due to the translation layer. This translation layer guarantees backwards compatibility with older API versions and translates (status, backup_status) -> status and status -> (status, backup_status).

To be able to perform an online migration of the database for an update of openstack a method to remap the status -> (status, backup_status) and save it in the database is necessary. This method should only be called if the schema update is done. This method will allow older openstack versions to be able to perform as before and set or check on their status as needed. These methods should be removed in further releases.

Upgrade impact

none

Implementation

The following existing restrictions on which actions / tasks can happen on a volume have to be maintained with the changes implemented:

Reject volume deletion while volume is currently being backed up
Reject concurrent backups of a volume if one is already in progress
Reject volume or “block storage” migration if a backup is currently running

Assignee(s)

Primary assignee:: Christian Rohmann (IRC: crohmann)

Other contributors:

Work Items

Split up the status field to the above mentioned status and backup_status fields
- Update SQL data model by adding a backup_status column to the volumes table
- Create DB migration (Alembic)
- additional method(s) for the model to allow an online migration of the database
- Update volume class by adding backup_status field and bumping the version
Introduce validation methods for the new backup_status field to guarantee that no wrong status can be set
Update validation methods for the status field to guarantee that no wrong status can be set
Add versioning to the conditional_update method for old database models
Update method calls to conditional_update to use status and backup_status
Introduce the API-Layer as a translator to serve older micro versions
Introduce a new microversion to allow backups to be performed in isolation from other operations that use volume’s status field as a locking mechanism.
Documentation
- Breaking changes
- API Documentation
  - Models
  - Endpoints
- Upgrade guide/scripts for older database models
  - online migration
  - schema updates

Dependencies

Dependencies to other developer teams have to be communicated to ensure they use the old microversion to avoid breaking changes and to switch to the new split up fields. This change should especially be communicated to the nova team which checks regularly for the status of an attached volume.

Testing

Since only a reduced set of states are handled via the status field and with backup_status as newly introduced field, some functional tests have to be adapted to use the new data model.
Further tests have to be added to ensure the translation layer for older API microversions work as expected. E.g. the backup status is presented via either status for an older microversion and then via backup_status for the new version.
Because the conditional_update` method needs to support versioning in this release, test should be written to verify that the versioning happens correctly.

Documentation Impact

Add a release note explaining the motivation and effect of the change
Document the state-machines for the volume itself and the backup and restore tasks.
Document the translation layer for the older microversions and how the translation behaves.

References

[1] Previously proposed spec to add a task status: https://review.opendev.org/c/openstack/cinder-specs/+/818551

History

Revisions
Release Name	Description
2023.02	Introduced

Image Encryption and Decryption

Mon, 02 May 2022 00:00:00

OpenStack already has the ability to create encrypted volumes and ephemeral storage to ensure the confidentiality of block data. In contrast to that, images are currently handled without protection towards confidentiality, only providing the possibility to ensure integrity using image signatures. For further protection of user data - e.g. when a user uploads an image containing private data or confidential information - the image data should not be accessible for unauthorized entities. For this purpose, an encrypted image format is to be introduced in OpenStack.

To enable this, several adjustments to support image encryption/decryption in various projects, e.g. Nova, Glance, Cinder and OSC, need to be implemented.

The proposal documented in this spec describes the Cinder-related part of the overall cross-project concept for image encryption.

Problem description

An image, when uploaded to Glance or being created through Nova from an existing server (VM), may contain sensitive information. The already provided signature functionality only protects images against alteration. Images may be stored on several hosts over long periods of time. First and foremost this includes the image storage hosts of Glance itself. Furthermore it might also involve other systems such as volume hosts. As a result they are exposed to a multitude of potential scenarios involving different hosts with different access patterns and attack surfaces. The OpenStack components involved in those scenarios – including Cinder - do not protect the confidentiality of image data.

Using encrypted storage backends for volume and compute hosts in conjunction with direct data transfer from/to encrypted images can enable workflows that never expose an image’s data on a host’s filesystem. Storage of encryption keys on a dedicated key manager host ensures isolation and access control for the keys as well. With such a set of configuration recommendations for security focused environments, we are able to reach a good foundation. Future enhancements can build upon that and extend the provided security enhancements to a broader set of variants.

That’s why we propose the introduction of an encrypted image format.

Use Cases

1. A user wants to create a new volume based on an encrypted image. The corresponding volume host has to be able to decrypt the image using the symmetric key stored in the key manager and transform it into the requested volume resource.

2.1. A user wants to create an encrypted image from a volume. The volume does not use an encrypted volume type. The target image is to be encrypted using the proposed image encryption mechanism and a specified key.

2.2. A user wants to create an encrypted image from a volume. The volume uses an encrypted volume type (LUKS-based). The target image is not to be re-encrypted but will instead reuse the LUKS encryption and key. This use case is already implemented as part of the default behavior of Cinder.

2.3. A user wants to create an encrypted image from a volume. The volume uses an encrypted volume type (LUKS-based). The target image is to be re-encrypted using the proposed image encryption mechanism and a specified key. This involves opening a dmcrypt mapping for the LUKS encryption and streaming the plain data of the volume directly into the target image file while encrypting it using the specified image encryption mechanism (e.g. GPG).

Proposed change

In Glance we propose to add a new container_format called ‘encrypted’. Furthermore, we propose the following additional metadata properties carried by images of this format:

‘os_encrypt_format’ - the main mechanism used, e.g. ‘GPG’
‘os_encrypt_type’ - encryption type, e.g. ‘symmetric’
‘os_encrypt_cipher’ - the cipher algorithm, e.g. ‘AES256’
‘os_encrypt_key_id’ - reference to key in the key manager
‘os_decrypt_container_format’ - format after payload decryption
‘os_decrypt_size’ - size after payload decryption

All these metadata properties are only used for the decryption of an image and won’t be needed in the lifecycle of a volume anymore. When creating an image from a volume these metadata properties are newly generated and saved in the Glance metadata of the new image.

For Cinder we want to add support for encrypted image decryption when a volume is created from it. Cinder should select a suitable decryption mechanism according to the new image container_format and metadata. The symmetric key will be retrieved from the key manager (e.g. Barbican) according to the key id stored in the metadata. Additionally, Cinder should be able to encrypt volume data in a similar fashion if the user requests an encrypted image to be created from a volume according to the use cases 2.1 and 2.3 illustrated above.

The implementations in Cinder should make use of a centralized encryption implementation provided by a global library, shared between all involved OpenStack components to prevent individual implementations of the encryption mechanism.

For general image encryption, we propose to use an implementation of symmetric AES 256 provided by GnuPG as a basic mechanism supported by this draft. It is a well established implementation of PGP and supports streamable encryption/decryption processes, which is important as we require the streamability of the encryption/decryption mechanism for two reasons:

Loading entire images into the memory of volume hosts is unacceptable.
We propose direct decryption-streaming into the target volume to prevent the creation of temporary unencrypted files.

There is already one existing case in Cinder’s current implementation where encrypted images are created (use case 2.2). This is when an image is created directly from an encrypted volume. Since the encrypted block data is simply copied into the image, the encryption (usually LUKS) is automatically inherited - as is the encryption key, which is simply cloned in Barbican. We will not change this behavior as a part of this spec. Our changes will only apply, when the user actively wants to create an encrypted image from any volume.

The key management is handled differently than with encrypted volumes or encrypted ephemeral storage. The reason for this is, that the encryption and decryption of an image will never happen in Glance but in all other services, which consume images. Therefore the service which needs to create a key for a newly created encrypted image may not be the same service which then has to delete the key (in most cases Glance). To delete a key, which has not been created by the same entity, is bad behavior. To avoid this, we choose to let the user create and delete the key. To not accidently delete a key, which is used to encrypt an image, we will let Glance register as a consumer of that key (secret in Barbican [1]) when the corresponding encrypted image is uploaded and unregister as a consumer when the image is deleted in Glance.

The methods for encryption and decryption of files - in this case images - will be written in a driver like manner in os-brick so the image encryption can be extended with another encryption format easily. The encryption driver should focus a specific encryption format and implement exactly one encrypt and one decrypt method, both based on a cipher implementation of GPG aes. This driver may be simple wrappers around an existing implementation. An abstract base class should be defined and be used for the implementation of GPG encryption (and might be used for other implementations in the future).

Alternatives

Regarding the image encryption, we also explored the possibility of using more elaborate and dynamic approaches like PKCS#7 (CMS) but ultimately failed to find a free open-source implementation (e.g. OpenSSL) that supports streamable decryption of CMS-wrapped encrypted data. More precisely, no implementation we tested was able to decrypt a symmetrically encrypted, CMS-wrapped container without trying to completely load it into memory or suffering from other limitations regarding big files.

We also evaluated an image encryption implementation based on LUKS which is already used in Cinder and Nova as an encryption mechanism for volumes and ephemeral disks respectively. However, we were unable to find a suitable solution to directly handle file-based LUKS encryption in user space. Firstly, the handling of LUKS devices (even when file-based) via cryptsetup always requires the dm-crypt kernel module and corresponding root privileges. Secondly, in contrast to native LUKS used by LibVirt, the LUKS handling available via cryptsetup creates temporary device mapper endpoints where data is read from or written to. There is no direct reading/writing from/to an encrypted LUKS file and LUKS opening/closing needs to be handled accordingly. Lastly, LUKS is a format primarily designed for disk encryption. Although it may be used for files as well (by formatting files as LUKS devices), the handling is rather inconvenient; for example, the size of the LUKS container file needs to be calculated and allocated beforehand since it acts like a disk with a fixed size.

Data model impact

None

REST API impact

For creating encrypted images from volumes, additional properties in the request body of “os-volume_upload_image” will need to be introduced to specify the desired encryption format and key id.

Security impact

There are impacts on the security of OpenStack:

confidentiality of data in images will be addressed in this spec
image encryption is introduced, thus additional cryptographic algorithms will be used in Cinder to implement this functionality

Notifications impact

None

Other end user impact

Users should be able to optionally, but knowingly create a differently encrypted image from a volume.
If an administrator has configured Glance to reject unencrypted images, such images will not be accepted when attempted to be uploaded to Glance.

Performance Impact

The proposed encryption/decryption mechanisms in Cinder will only be utilized on-demand and skipped entirely for image container types that aren’t encrypted.

Thus, any performance impact is only applicable to the newly introduced encrypted image type where the processing (or creation) of such image will have increased computational costs and longer processing times than for regular images. Impact will vary depending on the individual host performance and supported CPU extensions for cipher algorithms.

For Ceph-based setups, the usual cloning performance benefit provided by the shared storage between Glance and Cinder is lost for encrypted images, due to the need of converting the encrypted image data format to the volume format.

Other deployer impact

A key manager - like Barbican - is required.
The key manager needs to be accessible from volume hosts, requiring appropriate cinder.conf adjustments.

Developer impact

To use the encoding and decoding of images in os-brick, we need to execute priviledged functions. We decided to use privsep for this as in nova.

Upgrade impact

none

Implementation

Assignee(s)

Primary assignee:: Markus Hentsch (IRC: mhen)
Other contributors:: Josephine Seifert (IRC: Luzi)

Work Items

Add a decryption implementation in Cinder for creating volumes from GPG encrypted images
Add a dedicated encryption workflow and implementation in Cinder for creating encrypted images from volumes using the proposed image encryption format (GPG)
Add encryption and decryption methods for the GPG format in os-brick

Dependencies

GPG is required to be installed on all systems that are required to perform encryption/decryption operations in order to support the proposed base encryption mechanism.
This spec requires the implementation of an encrypted container_format and corresponding metadata property support in Glance
This spec requires the implementation of appropriate encryption/decryption functionality in a global library shared between the components involved in image encryption workflows (Nova, Cinder, OSC), like os-brick

Testing

Documentation Impact

It should be documented for deployers, how to enable this feature in the OpenStack configuration. An end user should have a documentation on how to use encrypted images in Cinder and how to create them respectively. Furthermore, any resulting limitations such as reduced performance should be mentioned, especially for Ceph-based setups usually benefiting from shared storage cloning.

References

[1] Barbican Secret Consumer Spec: https://review.opendev.org/#/c/662013/

Nova-Spec: https://review.openstack.org/#/c/608696/

Glance-Spec: https://review.openstack.org/#/c/609667/

History

Revisions
Release Name	Description
Train	Introduced
Ussuri	Postponed

Add Capacity Factors to scheduler get pools API

Wed, 13 Apr 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-capacity-factors-to-pool

This spec proposes adding the new capacity factors in the response to the get-pools API request.

Problem description

Typically there is some monitoring and alerting in place against a cinder deployment and it’s storage array to warn/alert the operator of the cloud that they are running out of storage, or are out of storage. The only mechanism in Cinder now to see the capacity factors of the backends that cinder is managing is to call the cinder api to get-pools. This returns some basic information about the backend/pools. But it doesn’t really tell the whole story.

The new capacity factors are various aspects of the capacity management that cinder calculates on each provisioning request. Each of those factors are used at runtime to decide if a volume can land on a particular pool. Those factors decide if a pool is full or not depending on several factors including,

thin or thick provisioning support
total capacity reported by the storage array
free capacity reported by the storage array
reserved percentage
max over subscription ratio

From those configuration settings and pool capabilities a wider set of factors is calculated to determine the capacity availability on a backend/pool. All of those factors determine if a backend/pool has space, is running out of space, or is out of space for cinder to use.

Use Cases

As an operator of cinder, I want cinder to report all of the factors it uses and calculates to determine if cinder’s backend/pools have space available. Only cinder has all information and the calculate_capacity_factors() can create the information the scheduler uses. It should also report that in the pools api request, so tooling for monitoring and alerting can be in sync with the cinder scheduler.

Proposed change

Add the dictionary that is created from calculate_capacity_factors() to each pool information in the get-pools response. The capacity factors are not the same thing as the capabilities being reported now. The capacity factors are the breakdown of the capacity calculations based on the driver’s reported capabilities seen by the backend storage. Depending on the capabilities in each pool such as thin_provisioning_support and thick_provisioning_support, factors will be calculated for each thin and thick if the pool supports.

{
  "total_capacity": 5120.0,
  "free_capacity": 4616,
  "reserved_capacity": 1024,
  "total_reserved_available_capacity": 4096,
  "max_over_subscription_ratio": None,
  "total_available_capacity": 4096,
  "provisioned_capacity": 500,
  "calculated_free_capacity": 3596,
  "virtual_free_capacity": 3596,
  "free_percent": 87.79296875,
  "provisioned_ratio": 0.1220703125,
  "provisioned_type": "thick"
}

Alternatives

The calculate_capacity_factors() function in utils.py can get copy/pasted into some external tooling that can do the alerting, but it’s subject to getting out of sync with cinder’s version. Therefore Cinder itself should report all of those capacity factors for each backend/pool that it calculates. Then cinder will be the definitive answer on the capacity it sees and can use.

Data model impact

None

REST API impact

A new microversion would be required and if compatible cinder would return the capacity_factors dictionary in the pools response.

GET /v3/{project_id}/scheduler-stats/get_pools?detail=True

{
    "pools": [
        {
            "name": "pool1",
            "capabilities": {
                "updated": "2014-10-28T00:00:00-00:00",
                "total_capacity_gb": 1024,
                "free_capacity_gb": 100,
                "volume_backend_name": "pool1",
                "reserved_percentage": 5,
                "driver_version": "1.0.0",
                "storage_protocol": "iSCSI",
                "QoS_support": false,
                "thin_provisioning_support": true,
                "thick_provisioning_support": true,
            },
            "capacity_factors": [
                {
                    "total_capacity": 1024,
                    "free_capacity": 100,
                    "reserved_capacity": 51,
                    "total_reserved_available_capacity": 973,
                    "max_over_subscription_ratio": None,
                    "total_available_capacity": 973,
                    "provisioned_capacity": 100,
                    "calculated_free_capacity": 873,
                    "virtual_free_capacity": 873,
                    "free_percent": 89.72,
                    "provisioned_ratio": 0.1028,
                    "provisioned_type": "thick"
                },
                {
                    "total_capacity": 1024,
                    "free_capacity": 100,
                    "reserved_capacity": 51,
                    "total_reserved_available_capacity": 973,
                    "max_over_subscription_ratio": 2,
                    "total_available_capacity": 1946,
                    "provisioned_capacity": 100,
                    "calculated_free_capacity": 1846,
                    "virtual_free_capacity": 1846,
                    "free_percent": 94.86,
                    "provisioned_ratio": 0.05,
                    "provisioned_type": "thin"
                }
            ],
        }
    ]
}

Security impact

None

Active/Active HA impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: hemna (Walter A. Boring IV)

Work Items

Add new microversion
Add capacity_factors in get-pools API response

Dependencies

This requires the new calculate_capacity_factors() function that is in the following review: https://review.opendev.org/c/openstack/cinder/+/831247

Testing

Add new unit tests to show the factors being returned in API call.

Documentation Impact

Add documentation to describe the capacity factors and the API response change

References

This was discussed at length at the Zed PTG: https://etherpad.opendev.org/p/zed-ptg-cinder

Youtube Video of discussion: https://www.youtube.com/watch?v=6yuOlGckkGE

The capacity factors definitions https://specs.openstack.org/openstack/cinder-specs/specs/queens/provisioning-improvements.html

Support Linux on System z (S/390) as a hypervisor platform

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/linux-systemz

There are some platform-specific changes needed in order to allow Cinder manage FCP-based block storage on Linux on System z.

Additional OpenStack functionality beyond initial Cinder support is not part of this blueprint; we will have specific additional blueprints for that, as needed.

Required support in Nova is described by a separate blueprint which is listed as a dependency below.

iSCSI does not require any changes.

Problem description

Linux on System z differs from other Linux platforms in the following respects:

System z uses a different format for device file paths (ccw-based, rather than pci-based)
Auto-discovery for fibre-channel devices can be configured online, or offline. In case auto-discovery is turned off, devices need to be added, and removed explicitly by OpenStack (unit_add, unit_remove)
vHBAs may be turned online, or offline. Offline vHBAs need to be ignored.

Use Cases

Proposed change

Change code in Cinder to address these issues, dependent on the host capabilities indicating a CPU architecture of arch.S390X, and arch.S390. For details, see section Work Items.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None (no need for platform-specific parameters in cinder.conf as part of this blueprint)

Developer impact

None (changes should not affect other platforms)

Implementation

Assignee(s)

Primary assignee:: stefan-amann
Other contributors:: mzoeller maiera

Work Items

In cinder/brick/initiator/connector.py:

connect_volume needs to support the System z specific format of the device file paths and issue the unit_add command (which is done by calling the new configure_scsi_device() function)

if platform.machine() == 's390x':
    target_lun = "0x00%02d000000000000" %
                 int(connection_properties.get('target_lun', 0))
    host_device = ("/dev/disk/by-path/ccw-%s-zfcp-%s:%s" %
                  (pci_num,
                   target_wwn,
                   target_lun))
    self._linuxfc.configure_scsi_device(pci_num, target_wwn, target_lun)
else :
    host_device = ("/dev/disk/by-path/pci-%s-fc-%s-lun-%s" %
                  (pci_num,
                   target_wwn,
                   connection_properties.get('target_lun', 0)))
host_devices.append(host_device)

disconnect_volume needs to support the System z specific format of the device file paths and issue the unit_remove command (which is done by calling the new deconfigre_scsi_device() function)

for device in devices:
    self._linuxscsi.remove_scsi_device(device["device"])
if platform.machine() == 's390x':
    ports = connection_properties['target_wwn']
    wwns = []
    # we support a list of wwns or a single wwn
    if isinstance(ports, list):
        for wwn in ports:
            wwns.append(str(wwn))
    elif isinstance(ports, basestring):
        wwns.append(str(ports))
    hbas = self._linuxfc.get_fc_hbas_info()
    for hba in hbas:
        pci_num = self._get_pci_num(hba)
        if pci_num is not None:
            for wwn in wwns:
                target_wwn = "0x%s" % wwn.lower()
                target_lun = "0x00%02d000000000000"
                    % int(connection_properties.get('target_lun', 0))
                host_device = ("/dev/disk/by-path/ccw-%s-zfcp-%s:%s" %
                              (pci_num,
                               target_wwn,
                               target_lun))
                self._linuxfc.deconfigure_scsi_device(
                               pci_num,
                               target_wwn,
                               target_lun)

In cinder/brick/initiator/linuxfc.py:

Utility functions to execute the unit_add, or unit_remove command.

def configure_scsi_device(self, device_number, target_wwn, lun):
    out = None
    err = None
    zfcp_device_command = ("/sys/bus/ccw/drivers/zfcp/%s/%s/unit_add" %
                            (device_number,
                             target_wwn))
    try:
        self.echo_scsi_command(zfcp_device_command, lun)
    except putils.ProcessExecutionError as exc:
        LOG.warn(_("zKVM unit_add call failed exit (
                               %(code)s), stderr (%(stderr)s)")
                 % {'code': exc.exit_code, 'stderr': exc.stderr})



def deconfigure_scsi_device(self, device_number, target_wwn, lun):
    out = None
    err = None
    zfcp_device_command = ("/sys/bus/ccw/drivers/zfcp/%s/%s/unit_remove" %
                            (device_number,
                             target_wwn))
    try:
        self.echo_scsi_command(zfcp_device_command, lun)
    except putils.ProcessExecutionError as exc:
        LOG.warn(_("zKVM unit_remove call failed
                    exit (%(code)s), stderr (%(stderr)s)")
                 % {'code': exc.exit_code, 'stderr': exc.stderr})

Have get_fc_hbas_info() only return enabled vHBAs for System z.

hbas_info = []
    for hba in hbas:
        if (platform.machine() != 's390x')
            or (hba['port_state'] == 'Online'):
                ...same as today

Dependencies

Nova blueprint to add support for KVM/libvirt in Linux on System z https://blueprints.launchpad.net/nova/+spec/libvirt-kvm-systemz

Testing

Unit test:

Unit tests will be added and performed on System z, as well as Intel-based machines.
We will provide an environment for CI testing on System z. This is described by the Nova blueprint which is listed as a dependency. We will test Cinder and Nova on System z in this environment.

Documentation Impact

No changes needed in config docs.
Doc changes for the platform will be made as needed (details are to be determined).

References

Linux on System z Device Driver book, http://public.dhe.ibm.com/software/dw/linux390/docu/l316dd25.pdf
Linux on System z, http://www.ibm.com/developerworks/linux/linux390/

Support ECKD volumes on Linux on System z

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/linux-ficon-support

FICON (Fibre Connection) is a Fibre-Channel FC-4 layer protocol. It is used on mainframes to attach ECKD volumes via Fibre-Channel. Several changes are needed in order to allow Cinder to manage FICON-based block storage on Linux on System z.

Besides, support is to be added to the existing Cinder drivers for storage subsystems supporting ECKD (for example, IBM DS8000).

Required support in Nova is described by a separate blueprint which is listed as a dependency below.

Problem description

FICON and FCP are Fibre-Channel FC-4 layer protocols. FICON uses a different scheme to address volumes, compared to FCP.

ECKD volumes are addressed by a Control Unit and Unit Address, rather than WWPN and LUN.
System z uses a different format for device file paths (ccw-based, rather than pci-based), similar to the file paths used for FCP on System z
ECKD volumes need to be set online and formatted before they can be used

Use Cases

Allow an end-user to create and manage Cinder block devices based upon FICON attached storage subsystems supporting the ECKD protocol. For example: create, delete, image deploy, snapshot, ..

We will extend a Cinder driver for IBMs DS8000. Other vendors, such as EMC, Hitachi, or HP support ECKD, too. They are potential candidates.

Proposed change

Change code in os-brick to address these issues, dependent on a new protocol type of ‘fibre_channel_eckd’, provided by the driver For details, see section Work Items.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Platforms that support FICON/ECKD will need to implement this for their drivers.

Implementation

Assignee(s)

Primary assignee:: stefan-amann
Other contributors:: arecknag maiera

Work Items

In os-brick/initiator/connector.py:

provide a new connector FibreChannelECKDConnector
connect_volume needs to enable interrupts for the volume and set the volume online. This is done by calling the new configure_eckd_device() function)
disconnect_volume needs to set the volume offline. This is done by calling the new deconfigure_eckd_device() function

In os-brick/initiator/linuxeckd.py:

Utility functions to set the volume online, validate it is formatted appropriately, and format it if required

def _is_online(self, device):
    """
    Return True if device is online, else return False
    :param device: device id
    """
    if os.access('/sys/bus/ccw/devices/%s/online' % device, os.R_OK):
        online_file = None
        try:
            online_file = open('/sys/bus/ccw/devices/%s/online' % device)
            value = online_file.readline()
            if value and value.strip() == '1':
                return True
        finally:
            if online_file:
                online_file.close()
    return False

def _is_formatted(self, device):
    """
    Return True if device is online, else return False
    :param device: device id
    """
    if os.access('/sys/bus/ccw/devices/%s/status' % device, os.R_OK):
        formatted = None
        (out, _err) = self._execute('cat',
                            '/sys/bus/ccw/devices/%s/status' % device,
                            run_as_root=True,
                            root_helper=self._root_helper)
        if out and out.strip() == 'unformatted':
            return False
    return True

def format_eckd_volume(self, path):
    """
    formats ECKD volume
    :param device: device path
    """
    name = os.path.realpath(path)
    if name.startswith("/dev/"):
        format_cmd = 'dasdfmt -y -b 4096 -d ldl ' + name
        (out, _err) = self._execute(format_cmd, run_as_root=True,
                            root_helper=self._root_helper)
        return True
    else:
        return False

In os-brick/initiator/linuxfc.py:

new class LinuxFibreChannelECKD

def configure_eckd_device(self, device_number):
    """Add the eckd volume to the Linux configuration. """

    full_device_identifier = "0.0.%04x" % device_number
    eckd_device_command = ('cio_ignore', '-r',
        '%(dev_id)s' % {"dev_id": full_device_identifier})
    LOG.debug("issue cio_ignore for s390: %s", eckd_device_command)

    out, info = None, None
    try:
        out, info = self._execute('cio_ignore', '-r',
                '%(dev_id)s' % {"dev_id": full_device_identifier},
                run_as_root=True,
                root_helper=self._root_helper)
    except putils.ProcessExecutionError as exc:
        LOG.warning(_LW("cio_ignore call for s390 failed exit"
                        " %(code)s, stderr %(stderr)s"),
                    {'code': exc.exit_code, 'stderr': exc.stderr})

    eckd_device_command = ('chccwdev', '-e',
            '%(dev_id)s' % {"dev_id": full_device_identifier})
    LOG.debug("add ECKD command for s390: %s", eckd_device_command)
    out, info = None, None
    try:
        out, info = self._execute('chccwdev', '-e',
                '%(dev_id)s' % {"dev_id": full_device_identifier},
                run_as_root=True,
                root_helper=self._root_helper)
    except putils.ProcessExecutionError as exc:
        LOG.warning(_LW("add ECKD call for s390 failed exit"
                        " %(code)s, stderr %(stderr)s"),
                    {'code': exc.exit_code, 'stderr': exc.stderr})

def deconfigure_eckd_device(self, device_number):
    """Remove the eckd volume from the Linux configuration. """

    LOG.debug("Deconfigure ECKD volume: device_number=%(device_num)s ",
              {'device_num': device_number})

    full_device_identifier = "0.0.%04x" % device_number
    eckd_device_command = ('chccwdev', '-d',
            '%(dev_id)s' % {"dev_id": full_device_identifier})
    LOG.debug("remove ECKD command for s390: %s", eckd_device_command)
    out, info = None, None
    try:
        out, info = self._execute('chccwdev', '-d',
                '%(dev_id)s' % {"dev_id": full_device_identifier},
                run_as_root=True,
                root_helper=self._root_helper)
    except putils.ProcessExecutionError as exc:
        LOG.warning(_LW("remove ECKD call for s390 failed exit"
                        " %(code)s, stderr %(stderr)s"),
                    {'code': exc.exit_code, 'stderr': exc.stderr})

Adaptations to volume.filters to allow the new commands to be sent

Cinder drivers need to report the Control Unit address, and Unit Address of the ECKD volume and set the driver_volume_type to ‘fibre_channel_eckd’

Any drivers wishing to support FICON will need to have CI reporting results from running against a FICON attached host.

Dependencies

Nova blueprint to add support for ECKD for Linux on System z. The blueprint can be found here: https://blueprints.launchpad.net/nova/+spec/linux-ficon-support

Testing

Unit test:

Unit tests will be added and performed on System z, as well as Intel-based machines.

CI environment:

We will provide a 3rd party CI environment for DS8000.

Documentation Impact

We will update documentation as required.

References

Linux on System z Device Driver book, http://public.dhe.ibm.com/software/dw/linux390/docu/l316dd25.pdf
Linux on System z, http://www.ibm.com/developerworks/linux/linux390/

Use ‘LIKE’ operator to filter resource

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-regexp-based-query

This blueprint proposes changing some filter behaviour to use ‘LIKE’ expression on some columns(name, description, etc).

Problem description

Currently, we could only filter cinder resource by exact match, and this is not flexible enough when user would like to retrieve some resource whose name or attribute is partly alike, especially for the users who have lots of volumes. As it’s already introduced in many other projects, we can take advantage of those some existing mechanism, nova and ironic.

Use Cases

Possible use case is to support client bulk operation, collect the desired resources with single command.

Proposed change

Although we can provide filtering resource based on regex filter, but there is a possibility that we could have ReDos attack. Considering the ‘LIKE’ operator is flexible and safe enough for this case. We could only introduce ‘LIKE’ operator (‘NOT LIKE’ is another useful operator, but it’s not as common as ‘LIKE’). And we can easily apply this filter at the existing common filtering function individually with a decorator:

@apply_like_filters(model=models.Volume)
def _process_volume_filters(query, filters):
    pass

This spec intends to support ‘LIKE’ based filter on some specified resources and columns, as we would have the generalized resource filtering feature, this one will take advantage of that spec in order to make it configurable for administrators, so this is what we would finally have in our filters’ configuration file

{
    "volume_filters": ["availability_zone", "status", "name~"]
}

‘~’ here means the attribute ‘name’ supports query with like operator (not indicating the regex operator), once this is configured, the input key-value pair name~=value from API will finally be translated into this sql statement:

# '%' is automatically appended at both sides
SELECT * FROM volumes
WHERE name LIKE '%value%'

so with that switch on, end user could filter volume with these inputs below:

cinder list --filters name=volume_preserved   #exact matches
cinder list --filters name~=volume_preserved  #inexact matches

assuming we have two volumes in the name of ‘volume_preserved_01’, ‘volume_preserved_02’, so with first command we will get none of them while last command would have both of them.

Alternatives

There is an option that we can deploy searchlight which mainly focus on various cloud resource querying, and that’s a more widely topic. But we should also consider the cloud environment that doesn’t have a searchlight.

Also, there is another option that user can gather all the raw data and do the filtering on their own, but it’s obviously that that is what we try to avoid, cause it costs a lot of unnecessary resource expense.

Data model impact

None

REST API impact

Microversion bump is required for this change.

Cinder-client impact

Help text will be updated to advertise this change.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: tommylikehu(tommylikehu@gmail.com)

Work Items

Update the generalized resource filtering logic to accept new symbol ‘~’ in config file.
Update target object’s common filter method.
Add related unit testcases
Update cinder-client and OSC.

Dependencies

Depended on generalized resource filtering

Testing

Add unit tests to cover filter process change.

Documentation Impact

Update API documentation.

References

nova: https://review.openstack.org/#/c/45026/

ironic: https://review.openstack.org/#/c/266688/

reDos: https://en.wikipedia.org/wiki/ReDoS

resource filtering: https://review.openstack.org/#/c/441516/

Support retrieve pools filtered by volume-type

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-volume-type-filter-to-get-pool

Add feature that administrators can get back-end storage pools filtered by volume-type, cinder will return the pools filtered by volume-type’s extra-specs.

Problem description

Now cinder’s get-pools API doesn’t support filtering pools by volume type, and this is inconvenient when administrators want to know the specified pool’s state which also can meet volume type’s requirements, the administrators also can get all pools and filter them on their own, but it’s more complicated and inefficient. This change intends to cover this situation and bring more convenience to administrators.

Use Cases

In production environment, administrators often need to have an overall pool available statistics filtered by volume type, this will help them to make an adjustment before resources run out.

Proposed change

As we will introduce generalized resource filter in cinder, from the view of outside cinder, the only thing we should do is to advertise that we can support this filter now:

From the view of inside cinder. We will mark volume-type recognizable, and support for this filter in logic:

once the volume-type (name or id) is detected, the volume type object will be retrieved before scheduler api is called, and will be passed as a filter item.
the scheduler.get_pools called, which will call the host_manager.get_pools in result.
the host_manager.get_pools will collect the pools information as normal, and before it returns, the result will be filtered by host.get_filtered_hosts.
the filter properties of get_filtered_hosts only consists of volume-type properties.
As already proposed by generalized resource filtering [1], the changes on cinder-client for this feature are not needed.

Alternatives

Administrators also can retrieve and filter on their own, but it’s more complicated and inefficient. This change can reduce the request amount and filter unnecessary data transmitted from server to client.

Data model impact

None

REST API impact

Get-Pool API will accept new query string parameter volume-type. Administrators can pass name or ID to retrieve pools filtered.

GET /v3/{tenant_id}/scheduler-stats/get_pools?volume-type=lvm-default

Security impact

None

Notifications impact

None.

Other end user impact

Within generalized resource filtering, we would ultimately have a get-pools command like this below:

cinder get-pools --filters volume-type='volume type'

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: TommyLike(tommylikehu@gmail.com)

Work Items

Add Get-Pools’s filter.
Add filter logic when retrieving pools.
Add related tests.

Dependencies

Depended on generalized resource filtering [1]

Testing

Unit test to test whether volume-type filter can be correctly applied.
Tempest test whether volume-type filter work correctly from API perspective.

Documentation Impact

The cinder API documentation will need to be updated to reflect the REST API changes.

References

Cinder volume revert to snapshot

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/revert-volume-to-snapshot

It makes sense to be able to revert a volume to a previously taken snapshot. The revert function can be used to revert a volume to a previous snapshot, restoring the volume to the state at the time the snapshot was created. The feature supports reverting a volume to the most recent snapshot taken, also manila (shared file system service) has already supported what we are proposing in Ocata [1].

The revert process will overwrite the current state and data of the volume. If the volume was extended after the snapshot, the request would be rejected (reason is described in proposed change section). It’s assumed that the user doing a revert is discarding the actual state and data of the volume.

The purpose of this feature is to give users the possibility of recover instances and volumes more conveniently and with minimal downtime.

Problem description

Currently, a user can recover an instance by creating a volume from specified snapshot and attaching it using the following workflow:

User creates a volume(s)
User attach volume to instance
User creates a snapshot
Something happens (crash) causing the need of a revert
User creates a volume(s) from the snapshot(s)
User detach old volumes
User attach new volumes

This workflow has the following problems:

To create a volume from a snapshot, some of vendors need to copy data to the new volume. This process takes more time than simply reverting to the most recent snapshot.
To create a volume from a snapshot, the network configuration will change and that leads to problems in most of applications demanding admin intervention.
If the instance was created from a boot volume, initial customization scripts might not be able to configure the instance to the same status it was before the crash.
It consumes quota from the user and in some cases real space on the backend.

Another alternative is to restore the data from a backup, but that process can’t be done by the cloud user, it’s not a common case (data inconsistency is more likely the main cause), also, the end user would expect a less time consuming operation.

As mentioned above, we hope that can offer a way to recover instance quickly using an easiest way, revert snapshot may be a good choice.

Use Cases

The primary use case is the situation when an instance or volume crashes due to any adverse situation, attack or malfunctioning while storage data is kept alive. In this situation, the following workflow will be used:

User creates a volume.
User attaches the volume to instance.
User creates snapshots of safe points.
Something happens causing the need of a revert.
User detaches the volume.
User reverts to the safe snapshot(s).

Proposed change

Theoretically, if multiple snapshots exist, a storage system could restore a volume to any one of those snapshots. But revert a volume to a snapshot which is not the latest one typically deletes the later snapshots, which is a form of data loss and may not be what the user intends. So this feature will limit cinder to revert the most recent volume snapshot known to cinder.

For this use case, reverting backwards in time starting with the most recent snapshot makes sense. Only the user can determine whether a volume is usable or must be reverted, so getting back to a good state is a manual iterative process:

Observe corruption in volume
Revert to most recent snapshot
While corruption is present in volume:
    Delete most recent snapshot
    Revert to most recent snapshot

A suggestion with simple API which merely include the volume ID and allow cinder to determine the most recent snapshot has been put forward, but it could led to potential concurrency bugs:

If two users call volume-revert-to-snapshot and snapshot-create at the same time, the two calls race and one user could be surprised about which snapshot was used to revert the volume.
if two users call volume-revert-to-snapshot and snapshot-delete at the same time, the calls race and someone may be surprised when the volume is reverted to an even earlier snapshot than expected (which is a form of data loss).

To keep the REST interface explicit, and to avoid the race conditions described above, the interface will accept the ID of the snapshot to be restored. cinder will verify that the snapshot is the most recent one via the created_at field on the snapshot object, returning an error if not. At no time will cinder delete any snapshots during the revert operation, and it will not operate on any snapshot other than the one specified in the REST call.

There are two cases when reverting a volume, one is detached volume, and the other is the attached volume. We would only support reverting a detached volume.

As we support extend volumes at present, we could meet the case that snapshot is smaller than volume when reverting to snapshot and it’s obviously safe to revert in that case. But we will still restrict to revert the volume which current volume size is equal to snapshot, cause we don’t support shrink volume yet [2] (that ability will be used in the generic method, if the driver don’t support revert to snapshot).

Therefore, we need to do the following changes in order to support volume revert to snapshot.

Add volume revert to snapshot API.

We will add a revert to snapshot API, in the API layer, the validations ( existence, status and size) on both snapshot and volume will be proceeded along with whether the snapshot is the latest one.

Cinder will accept the request only if the volume’s status is ‘available’.
Implement revert to snapshot in cinder volume service.

During a snapshot restoration, there are two objects that are affected whose status must be reflected. The volume’s status is ‘reverting’, the snapshot’s status is also ‘restoring’. Other operations must not be allowed to either object during the restoration.

If the cinder driver can not support the revert feature, it will use the generic implementation to run revert process.

After a successful restoration, both objects’ status will again be set to the status before reverting. If the restoration fails, the volume will be set to ‘error’, and the snapshot will be set to ‘available’.
Create snapshot for backup

The volume’s revert process would possibly fail, in order to prevent data loss, before the reverting process, we will create a snapshot for backup and then delete it if the operation succeed, end users can take advantage of that snapshot to recover their data.
Implement revert to snapshot in cinder drivers.

Add a function that vendors can use to overwrite the default revert_to_snapshot function with an optimized version. Vendors can implement the optimized version of the revert snapshot feature in their drivers:
```
def revert_to_snapshot(self, context, volume, snapshot):
    """Is called to perform revert volume from snapshot.

    :param context: Our working context.
    :param volume: the volume to be reverted.
    :param snapshot: the snapshot data revert to volume.
    :return None
    """
```
Exception will be caught and stored if any inner error raised from this function.
Add a generic revert to snapshot implementation function.

If the cinder driver can not support the revert to snapshot feature, the framework will use the generic way to implement the revert snapshot feature.
1. create a temporary volume from snapshot (if the backend supports mount snapshot, we will directly mount the snapshot, and don’t need to create and mount the temporary volume anymore).
2. mount temporary volume to host.
3. copy data from temporary volume or snapshot to original volume.
4. delete the temporary volume.

Alternatives

There is an alternative that we can create a new volume from the snapshot (already implemented and used). But this has some drawbacks as we described in the ‘Problem description’ section.
Implement snapshot revert allowing revert to any snapshot the volume has.

Data model impact

New status ‘reverting’ to volume object and new status ‘restoring’ to snapshot object.

REST API impact

Add a new revert action to volume action collections, the ‘volume_id’ and ‘snapshot_id’ are both required:

URL = /v3/{tenant_id}/volumes/{volume_id}/action

Method: POST

Body = {
           'revert': {
               'snapshot_id': <snapshot_id>
           }
       }

Normal Response: Status code 202 with empty response body

Error responses:

1. 404, if the volume is not found.
2. 409, if volume and snapshot's status are not 'available' or
   the sizes of volume and snapshot are not equal.
3. 400, if the snapshot is not found or not belongs to the specified
   volume or is not the latest one.

Calling this method reverts a volume to the specified snapshot. It is intended for both tenants and admins to use, and the policy.json file will be updated to reflect allowed use by all.

Security impact

None

Notifications impact

Notifications about the revert process(start, end, error) and so does the related change notification will be add.

Cinder-client impact

The cinder-client will add command to expose the revert API:

usage: cinder revert-to-snapshot <snapshot>

Revert a volume to the specified snapshot.

snapshot: Name or ID of the snapshot to restore.

OpenStack-client impact

The openstack-client will add command to expose the revert API:

usage: openstack volume snapshot restore <snapshot>

Revert a volume to the specified snapshot.

snapshot: Name or ID of the snapshot to restore.

Other end user impact

This feature would also be implemented to cinder-ui.

Performance Impact

The in effect revert process might lock storage and consume a long time, depending on the size of the volume.

Determining which snapshot is the latest requires a database query that sorts by a timestamp (the created_at field on the snapshot object). This would be slightly slower than a query that does not care about result ordering.

Other deployer impact

None

Developer impact

There will be one new driver entry point revert_to_snapshot, driver maintainers can implement optimised version of this functionality. Also driver should pay attention to this cases.

During the reverting process, the specified snapshot can’t be deleted (this means the snapshot should be recreated in the revert_to_snapshot if that is deleted by the backend).

Implementation

Assignee(s)

Primary assignee:

zhongjun(jun.zhongjun2@gmail.com)
TommyLike(tommylikehu@gmail.com)

Work Items

Create APIs
Create RPC calls
Add generic revert_to_snapshot function.
Add revert_to_snapshot interface and lvm’s implementation
Implement unit test and functional test
Implement tempest test
Implement cinder client
Implement OSC
Implement cinder UI
Add API documents
Update develop reference and support matrix

Dependencies

None

Testing

Unit tests and manual testing, as well as tempest test for these proposed APIs.

To guarantee that a volume restore actually took place, a new scenario test will be needed that writes data to a volume, creates a snapshot, modifies the volume, restores the snapshot, and ensures the original data is present in the volume.
To guarantee that this feature could work in the HA deploy mode.

Documentation Impact

API Ref: Add content about the API.
Add revert to snapshot feature to develop reference.
Add RevertToSnapshot capability to cinder support matrix.

References

Explicit user messages

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/better-user-message

Use ‘action’, ‘resource’, ‘message’ to replace ‘event’ in user messages.

Problem description

We have introduced user messages since Ocata, that’s a beneficial feature, because it solves the problem that we don’t provide enough message when an async task is failed (especially for volume and snapshot resources). It’s easy to find out when and where the task failed with our new messages APIs.

But, the better user experience all depend on whether we have hard coded enough user messages in our projects. So that raises a new question, how could we add these messages without changing our code frequently and how could we maintain this ‘better user experience but not functionality involved’ code cleanly.

Here are some of our user message related codes:

# CODE1: defined event ids
UNKNOWN_ERROR = 'VOLUME_000001'
UNABLE_TO_ALLOCATE = 'VOLUME_000002'
ATTACH_READONLY_VOLUME = 'VOLUME_000003'
IMAGE_FROM_VOLUME_OVER_QUOTA = 'VOLUME_000004'

# CODE2: try/except and write the user messages
with excutils.save_and_reraise_exception():
    if isinstance(error, exception.ImageLimitExceeded):
        self.message_api.create(
            context,
            defined_messages.EventIds.IMAGE_FROM_VOLUME_OVER_QUOTA,
            context.project_id,
            resource_type=resource_types.VOLUME,
            resource_uuid=volume_id)

We would possibly have several issues if we try to add more user messages based on those presets.

[CODE1]: We would redefine a lot of similar event ids, such as UNABLE_CREATE_VOLUME_DRIVER_NOT_READY, UNABLE_CREATE_SNAPSHOT_DRIVER_NOT_READY and UNABLE_CREATE_BACKUP_DRIVER_NOT_READY, it’s obvious that they are all due to the same error and are created at same procedure (create resource).
[CODE2]: To have an explicit message, we would have a bunch of ‘if/else’ codes which only focus on detecting the error class and write down the corresponding messages, and even worse if the exception gets more and more, we have to update the code again and again.

So could we have a better way to fix these?

Use Cases

The main use-case here is for an explicit user message to end users and a better maintaining experience for cinder developers.

Proposed change

The proposed changes are:

Introduce message (or name it detail to avoid confusion) to user messages, the ‘message’ only represent what (no matter error or warning) happened at that explicit time, that means QUOTA_EXCEED is a valid one while UPLOAD_TO_IMAGE_OVER_QUOTA isn’t.
Build the relationship from exceptions to messages, we have a bunch of exceptions which could directly point out what happened and how to fix, but we couldn’t expose this to the end user because it would expose some sensitive information. We couldn’t hard code the cleaned messages into exceptions either, because the exceptions would be defined at different places (take cinder exceptions and driver exceptions for example) and different exceptions can be translated into one user messages. so this is proposed how to do, build a dictionary from exception name to messages (introduced above):
```
# multi_key dictionary
['HBSDBusy','XtremIOArrayBusy']: MessageIds.DEVICE_IS_BUSY
['ConfigNotFound',
 'ViolinInvalidBackendConfig']: MessageIds.INVALID_CONFIGURATION

message_map = {
MessageIds.DEVICE_IS_BUSY: _("Backend device is busy at present."),
MessageIds.INVALID_CONFIGURATION: _("Invalid configuration."),
}
...
```
Introduce action to user messages. This one should be introduced along with the messages, and can use it to indicate when the message is created, such as CREATE_SNAPSHOT_IN_BACKEND, UPLOAD_TO_IMAGE.
Generate event_id automatically, user message is used to build user friendly messages for project which directly display the information for administrators. But when referring to the projects which detect, classify and use our messages, the event_id is more useful at this case, so we should have ‘event_id’ no matter how we change the underground behaviour. Instead of manually define and maintain this event_ids we could build them by a combination of RESOURCE, ACTION and MESSAGE, take these for example:
```
# This is how we define event currently
IMAGE_FROM_VOLUME_OVER_QUOTA = 'VOLUME_000004'

# This is how we would build the event_id at response after this change.
VOLUME_BACKUP_001_0002 : ({PROJECT}_{RESOURCE}_{ACTION_ID}_{MESSAGE_ID})
```
We could have these benefits:
1. We don’t need to define that much events. (we only need to define less messages).
2. It’s also unique cross all of OpenStack.
3. It’s reading friendly and easy to classify or analysis.

Alternatives

The alternatives is we could keep adding user messages in the way of we currently have [1]. (There could be more alternatives or better solutions, but I failed to figure out.)

Data model impact

Database update is required to store the ‘action_id’ and ‘message_id’, also we can deprecate the ‘event_id’, because we could generate it anytime we want.

REST API impact

We don’t have API impact because we didn’t expose the create API.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

For a better experience, developers have to maintain the relationship from exception to messages when any related change is made.

Implementation

Assignee(s)

Primary assignee:: tommylikehu(tommylikehu@gmail.com)

Work Items

Upgrade database to reflect new user message object.
Support ‘exception’ and ‘action’ in user message APIs.
Add some unit tests.
Add script for database migration.

Dependencies

None

Testing

Unit tests

Documentation Impact

Update developer documentation for ‘exception-to-message’ dictionary.

References

Show resource’s total count info in list APIs

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-amount-info-in-list-api

This blueprint proposes adding total count info in Cinder’s list APIs.

Problem description

Show how many resources a user or tenant has is usually required in the web portal’s summary section, but since we introduced the pagination mechanism, we only could get the {resource}_link which tells us you have more resources. In order to show this kind of total number to the end user, many clouds have to collect all of the resources while they only have to display the first page of the resource.

Use Cases

The main use case is to have the administrators or users to know how many resources they have in total without retrieving and accumulating them all.

Proposed change

Having the total count information in list API response is not only a requirement for Cinder or OpenStack, it’s a common requirement for the REST APIs, so find out what are others are doing may help us to have a better API model for this case.

Facebook API support the total_count attribute in their list APIs [1]. And this is their API response:

{
  "data": [],
  "paging": {},
  "summary": {
    "total_count": 100
  }
}

JSON API org adds an example to demonstrate the usage of total-pages or count in their recommended examples [2]:

{
    "meta": {
        "total-pages": 10,
        "count": 100
    },
    "data": [],
    "links": {
        "self": "",
        "prev": "",
        "next": ""
    }
}

StackExchange API also adds the total attribute in their Common Wrapper Object [3]. And this is how their response looks like:
```
{
   "items": [],
   "page": 12,
   "page_size": 100,
   "total": 1200,
}
```

Since we have already added the {resources}_links attribute into our list API response. It makes more reasonable to have the amount info added into the response (take volume for example):

{
  "volumes_links": [],
  "volumes": [],
  "count": 100
}

So, this bp proposes to add new attribute count in our list APIs ( including index and detail).

Based on our current pagination system [4], if we add the count attribute into our response body in default, the db’s query statement would be executed twice for only one query which obviously has a performance impact, considering not every request requires this kind of info, the additional query parameter is required to turn this on when listing resource:

/v3/{tenant_id}/{resource}?with_count=true
/v3/{tenant_id}/{resource}/detail?with_count=true

Also, this is recommended by OpenStack API-WG [5].

For the first step we only plan to add the summary support for our main resources: volumes, snapshots and backups.

Alternatives

There are few alternative solutions for this requirement, let’s list and compare them all here.

Add amount information in response header:
```
X-resource-count: 200
```

The disadvantage of this is it will add some burden to the API customers, we don’t have any history for adding useful content in response headers. Also it makes more difficult for documentation.

Add explicit API for each resources:

POST: /v3/{tenant_id}/{resources}/action {os-count}

This change involves a lot of modifications and creates several new APIs. Adding this amount of APIs for such a simple API change is overdesign.

Create a new API for different resources:

POST: /V3/{tenant_id}/action {os-count}
BODY:
    {
        "resource": "volume",
        "user": "user_1",
        "other_filter": "other_value"
    }

For this change, one more API request is required if the end user wants to know how many resources in total when listing resources.

Data model impact

None

REST API impact

Microversion bump is required for this change.

Cinder-client impact

All of the list commands will be upgraded to support this.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Since we will add additional COUNT() statement if the list APIs are requested with the summary option, there would be negative performance impact on those APIs, especially when there are a lot of data in database.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: tommylikehu(tommylikehu@gmail.com)

Work Items

Add summary option support in list APIs
Add related unit testcases
Update cinder-client and OSC.

Dependencies

None

Testing

Add unit tests to cover this change.

Documentation Impact

Update API documentation.

References

Support create volume from backup

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-create-volume-from-backup

This blueprint proposes support for creating a volume from backup.

Problem description

Cinder has supported create volume from volume, snapshot and image. But when considering create volume from backup, users have to create a new volume first and then restore the volume with backup. Although we can restore backup without volume id, in that case Cinder will create a new volume in the background, but we still have some issues because we don’t have any access to control the volume’s az, volume type, size and many other attributes.

Use Cases

User can directly create volume from backup rather than create a new volume and then restore it.

Proposed change

This spec proposes to support create volume from backup, after this change users can directly create a new volume with backup id as below:

$ cinder create [size] --backup-id <id> [other_options]

Volume’s size is equal to the backup’s original volume size if size option is omitted.

Since we already introduced the generic backup implementation [1] for our existing volume drivers, it makes sense to add this ability to our volume drivers, so when create volume from backup, the request will be scheduled to the volume backend as normal, and first we would try the vendor-specific create_volume_from_backup method as below:

def create_volume_from_backup(self, volume, backup):
    """Creates a volume from a backup."""

    raise NotImplementedError()

If the driver reports NotImplemented or NotSupported, Cinder will directly create the raw volume at the backend and then schedule the request to the backup service to restore the volume with backup.

One of the differences between creating volume from snapshot and creating volume from backup is the latter could be time consuming, so we need to update the backup’s status to prevent possible usages by other requests. For instance, when creating volume from backup the possible status for volume and backup are:

1. Volume: creating, available, error
2. Backup: restoring, available, error

Note: We will use ‘creating’ instead of ‘restoring-backup’ for the volume’s transition status here in order not to expose the creating detail.

Alternatives

Keep using current restore API or create and restore mechanism to cover this use case.

Data model impact

None

REST API impact

Microversion bump is required for this change.

Cinder-client impact

Cinder-client will be updated to support create volume from backup.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

When creating a volume from backup and the vendor-specific method does not exist, the creation process could take a little longer than a typical creation due to the restore action.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

luqitao (qtlu@fiberhome.com)

tommylikehu(tommylikehu@gmail.com)

Work Items

Support create volume from backup in Cinder.
Add related unit testcases.
Add related tempest testcase.
Update cinder-client and OSC.

Dependencies

Depends on generic backup implementation [1]

Testing

Add unit tests to cover creating volume from backup.

Documentation Impact

Both API documentation and CLI documentation should be updated.

References

Support availability zone in volume type

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-az-in-volumetype

This blueprint proposes to support setting a volume type’s availability zones.

Problem description

In a newly deployed region environment, the volume type (SSD, HDD or others) may only exist on part of the AZs, but end users have no idea which AZ is allowed for one specific volume type and they can’t realize that only when the volume failed to be scheduled to backend.

Use Cases

Administrators can update the available AZs for volume type and UI or end users can know valid volume types in current AZ.

Proposed change

As the discussion result during Rocky PTG [1], we would propose to use volume type’s extra specs to support this. Since extra specs is designed for generic use, we would propose to introduce a reserved key os-extended:availability_zones for extra specs. Administrator can create/update this attribute in the format as below to support AZ in volume type:

{
"volume_type": {
    "id": "6685584b-1eac-4da6-b5c3-555430cf68ff",
    "name": "az_type",
    "description": "availability zone type 001",
    "is_public": true,
    "extra_specs": {
        "os-extended:availability_zones": "az1,az2,az3"
        }
    }
}

NOTE: Our capability filter will skip any extra spec which has key in the format of “prefix:key_name”. Since this use case is more related to UI or other endpoint concerns, we would also propose to support filter volume type via extra specs as well:

Request example:
/v3/{project_id}/types?extra_specs={'os-extended:availability_zones':'az1'}

Also, it’s mentionable that the extra spec availability zones will be handled in a slightly different way when compared with the rest during filtering. Cinder will always try inexact match for this specific value, for instance, when extra spec os-extended:availability_zones is set as below, both az1 and az2 are valid input for query:

"os-extended:availability_zones": "az1,az2,az3"

In order to fully support type’s AZ in Cinder, there are also some changes on Cinder framework.

Now the intersection of AZs between volume type’s and other inputs (user input, source resource or default configuration) will be calculated when creating a new volume and 400 Bad Request will be raised if this intersection is empty.
It’s possible now we could pass an AZ list to the scheduler when creating or retyping, workflow and AvailabilityZoneFilter will be updated to support this.

NOTE: Cinder will raise 400 Bad Request as well if type’s availability_zones attribute is set but value is empty.

Alternatives

None

Data model impact

None

REST API impact

Now Cinder supports filter volume type with extra specs:

GET: /V3/{tenant_id}/types?extra_specs={'spec_name':'spec_value'}

Response BODY:
{
    "volume_types": [{
        "name": "volume_type1",
        "description": "volume_type1",
        "extra_specs": {"spec_name": "spec_value"},
    }]
}

API version bump will be required for these changes.

Cinder-client impact

Volume type’s CLIs will be updated corresponding to the API change at server side.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There would be a slight performance impact when filtering volume types with extra specs.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: tommylikehu(tommylikehu@gmail.com)

Work Items

Support availability zone and extra spec filtering in volume type API.
Update cinder workflow to support availability zone lists.
Add related unit testcases.
Update cinderclient to support filter volume type with extra specs.

Dependencies

None

Testing

Add unit tests to cover this change.

Documentation Impact

Update the API reference.
Add administrator documentation to advertise the reserved key os-extended:availability_zones and explain why&when&how should this can be used.

References

Support filter backend based on operation type

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-mark-pool-sold-out

This blueprint proposes to support filter backend based on operation type.

Problem description

Now our filter and weigher plugins’ algorithm are all designed based on two factors, backend information and user input. But there could be another factor that may also affect the scheduler result, operation type. For instance, if administrators want to disable some of the backend pool only for creating new volume operation, it’s impossible based on our current logic.

Use Cases

We have discussed the concept of sold out during Rocky PTG [1] and that word is more related to administrative operation, when most of an specific pool’s capacity has been consumed, administrators need the functionality to mark the backend pool sold out, that is to say, it will become unavailable for allocating new volumes, but end users still can perform any operations on their existing resources (extend volume, create snapshot).

Adding this feature in cinder will help cloud vendors write their own plugins which can filter backend based on the operation type as well as backend information.

Proposed change

New attribute operation will be added into RequestSpec object and plugins can filter backend based on backend states as well as request type:

def backend_passes(self, backend_state, filter_properties):
    #get request type from filter_properties
    spec = filter_properties.get('request_spec', {})
    request_type = spec.get('operation', None)
    if request_type in ['create_volume']:
        #check backend state when request is for creating new volume
        return backend_state.free_capacity_gb > MIN_REQUIRED_GB
    if request_type in ['extend_volume', 'retype_volume']
        #perform other validation when request is ......

NOTE: This spec doesn’t propose to update any plugin’s logic, the code above only intends to demonstrate how the operation can be used.

Based on our current features, the value of operation can be one of these below:

create_group
manage_existing
extend_volume
create_volume
create_snapshot
migrate_volume
retype_volume
manage_existing_snapshot

And we will inject scheduler manager’s corresponding methods in order to append this info to RequestSpec automatically:

@append_operation
def extend_volume(self, context, volume, new_size, reservations,
                  request_spec=None, filter_properties=None):
    #other existing codes

Alternatives

The alternative is to implement a custom scheduler manager and configure it in scheduler_manager and then use either a custom filter or the JSON filter.

Data model impact

RequestSpec will be updated to store this new attribute.

REST API impact

None

Cinder-client impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Developers can develop new scheduler plugins based on backend state as well as request type.

Implementation

Assignee(s)

Primary assignee:: tommylikehu(tommylikehu@gmail.com)

Work Items

Add operation to those actions which will be scheduled to scheduler to perform filtering and weighing logic.

Dependencies

None

Testing

Add unit tests to cover this change.

Documentation Impact

Update the developer document to mention the new attribute will be delivered to scheduler plugins as well.

References

https://review.openstack.org/#/c/308869/

Support Image Signature Verification

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-support-image-signing

Cinder currently does not support signature validation of downloaded signed images. Equipping Cinder with the ability to validate image signatures will provide end users with stronger assurances of the integrity of the image data they are using to create volumes. This change will use the same data model for image metadata as the accompanying functionality in Glance, which will allow the end user to sign images and verify these image signatures upon upload [1].

Problem description

Previously, OpenStack’s protection against unexpected modification of images was limited to verifying an MD5 checksum. While this may be sufficient for protecting against accidental modifications, MD5 is a hash function, not an authentication primitive [2], and thus provides no protection against deliberate, malicious modification of images. An image could potentially be modified in transit, such as when it is uploaded to Glance or transferred to Cinder. An image that is modified could include malicious code.

Currently, Glance has support for image signature verification upon upload, but Cinder does not support the feature to ensure the integrity of the image data before using it. Providing support for signature verification would allow Cinder to verify the signature before creating volume from image. This feature will secure OpenStack against the following attack scenarios:

Man-in-the-Middle Attack - An adversary with access to the network between Cinder and Glance is altering image data as Cinder downloads the data from Glance. The adversary is potentially incorporating malware into the image and/or altering the image metadata.
Untrusted Glance - In a hybrid cloud deployment, Glance is hosted on machines which are located in a physically insecure location or is hosted by a company with limited security infrastructure. Adversaries may be able to compromise the integrity of Glance and/or the integrity of images stored by Glance through physical access to the host machines or through poor network security on the part of the company hosting Glance.

Please note that our threat model considers only threats to the integrity of images while they are in transit between the end user and Glance, while they are at rest in Glance and while they are in transit between Glance and Cinder. This threat model does not include, and this feature therefore does not address, threats to the integrity, availability, or confidentiality of Cinder.

Use Cases

A user wants a high degree of assurance that a customized image which they have uploaded to Glance has not been accidentally or maliciously modified prior to creating a volume from the image.

With this proposed change, Cinder will verify the signature of a signed image while downloading that image. If the image signature cannot be verified, then Cinder will not create a volume from the image and instead place the volume into an error state. The user will begin to use this feature by uploading the image and the image signature metadata to Glance via the Glance API’s image-create method. The required image signature metadata properties are as follows:

img_signature - A string representation of the base 64 encoding of the signature of the image data.
img_signature_hash_method - A string designating the hash method used for signing. Currently, the supported values are SHA-224, SHA-256, SHA-384 and SHA-512. MD5 and other cryptographically weak hash methods will not be supported for this field. Any image signed with an unsupported hash algorithm will not pass validation.
img_signature_key_type - A string designating the signature scheme used to generate the signature. For more detail on which are currently supported, please check Glance’s documentation [7].
img_signature_certificate_uuid - A string encoding the certificate uuid used to retrieve the certificate from the key manager.

The image verification functionality in Glance uses the cursive.signature_utils module to verify this signature metadata before storing the image. If the signature is not valid or the metadata is incomplete, this API method will return a 400 error status and put the image into a “killed” state. Note that, if the signature metadata is simply not present, the image will be stored as it would normally.

The user would then create a volume from this image using the Cinder API’s volume create method. If the verify_glance_signatures flag in cinder.conf is set to ‘True’, Cinder will call out to Glance for the image’s properties, which include the properties necessary for image signature verification. Cinder will pass the image data and image properties to the signature verification module, which will verify the signature. If signature verification fails, or if the image signature metadata is either incomplete or absent, creating volume from the image will fail and Cinder will log an exception. If signature verification succeeds, Cinder will create volume from the image and log a message indicating that image signature verification succeeded along with detailed information about the signing certificate.

Proposed change

Since Nova has implemented this feature and all of the verification process has been moved into cursive module [4], it’s more convenient to support this in Cinder now.

Verify image signature with certificate

We propose an initial implementation by incorporating cursive into Cinder’s control flow for Creating volumes from images, and use new configuration verify_glance_signatures to turn this on or off. Initially it will have two options (default is enabled):

enabled: verify when image has complete signature metadata.
disabled: verification is turned off.

NOTE: We have discussed to add required option to introduce a strict mode on verification, but this can’t be guaranteed as we can’t do verification when image volume is cloned in backend. Strict mode will still be considered when we can cover every approach.

Upon downloading an image, Cinder will both check the new configuration flag and image’s metadata. If needs, the module will perform image signature verification using image properties passed to Cinder by Glance. If this fails, or if the image signature metadata is incomplete or missing, Cinder will not create the volume from the image. Instead, Cinder will throw an exception and log an error. If the signature verification succeeds, Cinder will proceed with creating the volume. The code sample is below:

if CONF.glance.verify_glance_signatures != 'disabled':
   verifier = None
   image_meta_dict = self.show(context, image_id,
                               include_locations=False)
   image_meta = objects.ImageMeta.from_dict(image_meta_dict)
   img_signature = image_meta.properties.get('img_signature')
   img_sig_hash_method = image_meta.properties.get(
       'img_signature_hash_method'
   )
   img_sig_cert_uuid = image_meta.properties.get(
       'img_signature_certificate_uuid'
   )
   img_sig_key_type = image_meta.properties.get(
       'img_signature_key_type'
   )
   try:
       verifier = signature_utils.get_verifier(
           context=context,
           img_signature_certificate_uuid=img_sig_cert_uuid,
           img_signature_hash_method=img_sig_hash_method,
           img_signature=img_signature,
           img_signature_key_type=img_sig_key_type,
       )
   except cursive_exception.SignatureVerificationError:
       #Image signature verification failed
   # Collect image data
   try:
       if verifier:
           verifier.verify()
   except cryptography.exceptions.InvalidSignature:
       #Image signature verification failed

NOTE: We will try different approaches when creating volume from images, so we have to mention this feature will not cover every approach especially when volume is created at backend.

To be clear, we will verify the image’s signature only when image is downloaded from glance and content is copied to volume on host. So when image volume is created via clone_image or clone_image_volume we will skip this verification process regardless of configuration option and provided signature metadata, in order not to confuse end users, we will add verification flag signature_verified in volume’s image metadata when creating from image.

Verify certificate with trusted certificates

This feature tries to find a way to determine if the certificate used to generate and verify that signature is a certificate that is trusted by the user, we could find more detail in Nova spec [5]. In short, within that feature end user can also validate the image’s certificate with the given trusted certificates (specified via API or config option). Considering the feature is in the process of being added to Nova now, we will follow this up with another spec when it’s merged into Nova for the purpose of consistency.

Alternatives

An alternative to signing the image’s data directly is to support signatures which are created by signing a hash of the image data. This introduces unnecessary complexity to the feature by requiring an additonal hashing stage and an additional metadata option. Due to the Glance community’s performance concerns associated with hashing image data, the developers initially pursued an implementation which produced the signature by signing an MD5 checksum which was already computed by Glance. This approach was rejected by the Nova community due to the security weaknesses of MD5 and the unnecessary complexity of performing a hashing operation twice and maintaining information about both hash algorithms.

An alternative to using certificates for signing and signature verification would be to use a public key. However, this approach presents the significant weakness that an attacker could generate their own public key in the key manager, use this to sign a tampered image, and pass the reference to their public key to Cinder along with their signed image. Alternatively, the use of certificates provides a means of attributing such attacks to the certificate owner, and follows common cryptographic standards by placing the root of trust at the certificate authority.

An alternative to using the verify_glance_signatures configuration flag to specify that Cinder should perform image signature verification is to use a “verify_glance_signatures” type-key for a volume type to specify that individual volume should be created from signed images. The user, when using the Cinder CLI to create a volume from image, would specify a volume type which includes a type-key “verify_glance_signatures=True” to indicate that image signature verification should occur as part of the control flow for creating the volume. This may be added in a later change, but will not be included in the initial implementation. If added, the “verify_glance_signatures” type-key option will work alongside the configuration option approach. In this case, Cinder would perform image signature verification if either the configuration flag is set, or if the user has specified creating a volume of the volume type which includes “verify_glance_signatures=True” type-key.

Another alternative to using the verify_glance_signatures configuration flag to specify that Cinder should perform image signature verification is amending the Cinder create command to accept an additional parameter specifying whether image signature verification should occur. This may be added in a later change, but will not be included in the initial implementation. If added, the additional parameter will work alongside the configuration option approach. In this case, Cinder would perform image signature verification if either the configuration flag is set, or if the user has specified creating a volume of the additional parameter.

We maybe only need to choose one of the above two alternatives.

Data model impact

The accompanying work in Glance introduced additional Glance image properties necessary for image signing. The initial implementation in Cinder will introduce a configuration flag indicating whether Cinder should perform image signature verification before booting an image.

REST API impact

None

Security impact

Cinder currently lacks a mechanism to validate images prior to creating volumes from them. The checksum included with an image protects against accidental modifications but provides little protection against an adversary with access to Glance or to the communication network between Cinder and Glance. This feature facilitates the creation of a logical trust boundary between Cinder and Glance; this trust boundary permits the end user to have high assurance that Cinder is creating a volume from an image signed by a trusted user.

Although Cinder will use certificates to perform this task, the certificates will be stored by a key manager and accessed via Castellan.

Notifications impact

This change will involve adding log messages to indicate the success or failure of signature verification and creation.

A later change will involve notifying the user about failure in case signature verification fails, this will use async error notification feature [3].

Other end user impact

If the verification of a signature fails, then Cinder will not create a volume from the image, and an error message will be logged and recorded. The user can get the error messages through the log file or CLI command, and know the reason for the error. In this case, the user will have to edit the image’s metadata through the Glance API, or the Horizon interface; or reinitiate an upload of the image to Glance with the correct signature metadata in order to create a volume from the image.

Performance Impact

This feature will only be used if the verify_glance_signatures configuration flag is set.

When signature verification occurs there will be latency as a result of retrieving certificates from the key manager through the Castellan interface. There will also be CPU overhead associated with hashing the image data and decrypting a signature using a public key.

Other deployer impact

We will recommend you deploy Barbican service [6] to store your certificate information as other projects suggest, although you can integrate any other secret manager service via Castellan [8].

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ji-xuepeng TommyLike(tommylikehu@gmail.com)
Other contributors:: None

Work Items

The feature will be implemented in the following stages:

Add functionality to Cinder which calls the cursive module when Cinder downloads a Glance image and the verify_glance_signatures configuration flag is set.
Generate notification messages at the time of failure for signature verification.

Dependencies

None

Testing

Unit tests and also, tempest tests will be added into barbican-tempest-plugin to cover the case of create volume from signed image.

Documentation Impact

Instructions for how to use this functionality will need to be documented.

References

Cryptography API: https://pypi.org/project/cryptography/0.2.2

Support deferred volume deletion in the RBD driver

Tue, 25 Jan 2022 00:00:00

https://blueprints.launchpad.net/cinder/+spec/rbd-deferred-volume-deletion

This spec proposes configurable support for deferred volume deletion in the RBD driver, using the trash_* API calls, as introduced with the Luminous version of Ceph.

Problem description

Volume deletion in Cinder is synchronous: the c-vol worker thread deleting a volume will return only when the backend confirmed the removal of the volume. This approach ensures that the space used on the backend will at most be equal to the quota of the corresponding project, and hence allows for proper resource management. The current implementation of the RBD driver follows this approach by using RBD’s ‘remove()’ API call.

For deployments with thousands of volumes/users and large volumes (10-100TB) this can create two potential issues:

a c-vol worker thread will be blocked during deletion and depending on the size of the volume, the speed of the backend, the number of concurrent requests, and the total number of worker threads, this may block other operations, makes the service seem unresponsive, and can lead to timeouts.

[1] was proposed to mitigate this problem by making the number of native threads configurable and profit more efficiently from the backend’s potential capabilities.
users have to wait until the deletion is completed on the backend before they can re-use the freed up quota/space – despite having signalled to Cinder that the volume should be removed and the quota be freed. For large volumes, this waiting time can be significant (hours for large volumes) and has hence a corresponding negative impact on the user experience.

To address these issues, this spec proposes to optionally support asynchronous volume deletion in the RBD driver.

Use Cases

Administrators can optionally enable a deferred deletion: the Ceph backend will acknowledge the deletion immediately adressing the two issues above.

Proposed change

The proposal is to

amend the RBD driver to support deferred deletion by optionally calling RBD’s ‘trash_move()’ rather than RBD’s ‘remove()’ in the driver’s implementation of ‘delete_volume()’;
add a periodic timer in the RBD driver to trigger a regular purge.

For this, the following additional config options would be needed:

‘enable_deferred_deletion’ (default: ‘False’)
If enabled, RBD’s ‘thrash_move’ will be called rather than ‘remove’.
‘deferred_deletion_delay’ (default: 0)
Number of seconds before an image moved to trash is regarded as expired and can be purged.
‘deferred_deletion_purge_interval’ (default: 3600)
Number of seconds between runs of the periodic timer to purge the backend’s trash.

Alternatives

None

Data model impact

None

REST API impact

None.

Cinder-client impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

For the end users, deletions will appear to be instantaneous. The freed up quota will become available immediately.

Performance Impact

Volume deletions will become independent from the backend’s speed or the size of the volumes.

Other deployer impact

As the deletion is deferred, the space occupied will not be freed immediately. Deployers need to be aware that the maximum space needed can exceed the sum of the allocated quota of all projects.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Arne Wiebalck (arne.wiebalck@cern.ch)

Work Items

Update RBD driver to respect options for deferred deletions and call ‘trash_move’ if enabled.
Add periodic task to call ‘trash_purge’ for backend which have deferred deletion enabled.
Add related unit testcases.

Dependencies

None

Testing

Add unit tests to cover this change.

Documentation Impact

Add administrator documentation to advertise the option of deferred deletions for RBD and explain why&when&how this should be used, emphasizing in particular that by enabling this option the space of deleted volumes will not be freed on the RBD backend until the trash is purged.

References

https://review.openstack.org/#/c/550858/

Remove the need for project_id from API endpoints

Thu, 11 Nov 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/project-id-optional-in-urls

Problem description

Cinder currently assumes API URLs include an associated project_id. Including a project_id in the URL is redundant because the project_id is typically available in the keystone context for the API request. In addition, requiring a project_id is incompatible with keystone’s notion of system scope, in which an API is not associated with a specific project. System scoped API requests will not have a project_id, and therefore the URL for those requests should not include a project_id.

Furthermore, cinder’s inclusion of a project_id in its API URLs is out of sync with other OpenStack services. The Images (glance API v2), Compute (nova API as of v2.16), Identity (keystone API v3) and Shared File Systems (manila API as of v2.60) services do not require a project_id in their API URLs.

Use Cases

A cloud administrator wishes to utilize keystone personas in order to grant appropriate access permissions to users that are assigned to different roles, e.g. lab operators and project administrators. Lab personnel are typically required to manage system-wide cinder resources, such a storage backends and volume types. However, these personnel should not have access to project specific resources, such as the cinder volumes. Likewise, project administrators need the ability to manage the resources owned by projects for which they have been given responsibility, but should have limited access to system wide resources, such as the ability to configure volume types.

In order to support system scoped personas, the URL for API requests should not be required to include a project_id.

Proposed change

This spec proposes the cinder API be enhanced to not require a project_id in the API’s URL. Inclusion of a project_id in the URL would be optional, which means the proposed change retains backward compatibility. A new microversion will be introduced so that clients may know whether inclusion of a project_id in API URLs is optional or mandatory.

The microversion will only be used as an indicator that the API can handle URLs without project IDs, and this applies to all requests reqardless of the microversion in the request. For example, if the new proposed microversion is v3.67, an API node serving v3.67 or greater will properly route requests without project_id even if you ask for v3.0. Likewise, it will properly route requests to URLs containing project_id even if you ask for v3.67.

The proposal does not introduce any behavioral or functional changes to the API. The same API method will be invoked regardless of whether the URL includes a project_id, and the actual project_id associated with the call will be the one contained in the keystone context for the API request. The current behavior of returning HTTP 400 will be retained if the request URL contains a project_id that doesn’t match the one in the keystone context.

Alternatives

A dummy project_id value (perhaps all zeroes) could be defined for use by system scoped API requests. This would not be an elegant solution, and would require changes in keystone.

Cloud administrators could be required to create an actual project solely for use when making system scoped API requests. This is also not elegant, a shifts the burden of managing system scope onto cloud administrators.

Data model impact

None

REST API impact

The behavior of each API method will be unchanged. However, the API routing will be enhanced to provide duplicate routes to each method.

The existing route for which a project_id is in the URL
A new route for when the URL does not include a project_id

Security impact

There is no security impact because keystone will continue to provide an authorization context for every API request, regardless of whether the request URL includes a project_id.

Active/Active HA impact

None

Notifications impact

None

Other end user impact

End users typically interact with the cinder API through other clients, such as the python-cinderclient, python-openstackclient, or openstacksdk. In that regard, there is no direct impact on the end user.

End users who interact with the API directly using tools such as curl may continue to include a project_id in the URL. They will have the option of not including a project_id in the URL.

Performance Impact

None

Other deployer impact

Deployment tools that create cinder endpoints in the keystone catalog no longer need to use project_id templating. Endpoints in the catalog should look like this:

$ openstack endpoint list --service cinderv3 -c 'Service Name' -c URL
+--------------+---------------------------------+
| Service Name | URL                             |
+--------------+---------------------------------+
| cinderv3     | http://192.168.100.30/volume/v3 |
+--------------+---------------------------------+

rather than this:

$ openstack endpoint list --service cinderv3 -c 'Service Name' -c URL
+--------------+------------------------------------------------+
| Service Name | URL                                            |
+--------------+------------------------------------------------+
| cinderv3     | http://192.168.100.30/volume/v3/$(project_id)s |
+--------------+------------------------------------------------+

Developer impact

None

Implementation

Assignee(s)

Primary assignee: abishop

Work Items

Define a new microversion
Add API routes to connect API requests with no project_id in the URL to the associated method
Add or enhance unit tests
Update devstack so the cinder endpoint doesn’t use project_id templating

Dependencies

None

Testing

Existing tempest tests will certainly verify whether the API functions correctly. The challenge is verifying both the legacy behavior, where URLs include a project_id, and the new behavior when URLs don’t include a project_id. One possiblity is to employ a periodic zuul job that specifically tests the legacy behavior. The details can be worked out during the development phase.

Documentation Impact

The existing API reference is largely unaffected. This is because the reference guide is unbranched, and older versions of cinder require a project_id. A brief note will be added to explain the project_id in the API reference is optional as of the new microversion.

References

OpenStack’s Support Common Operator & End User Personas community goal
Cinder’s Secure RBAC Ready spec

NVMeoF Multipathing

Wed, 27 Oct 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/nvmeof-multipath

Extend the NVMeoF connector to support two types of multipathing:

Native NVMeoF multipathing
DeviceMapper based multipathing

At the time of writing, only TCP and RDMA protocols are supported and contemplated by the spec (so it does not contemplate the case for other transport protocols such as FC).

Problem description

Currently the NVMeoF connector only supports single pathed connections.

The problem description can be summed up by the statement: A single pathed connection is a single point of failure.

For resiliency, and to catch up with industry standards, support for multi pathed connections is needed, mitigating the single point of failure.

Another potential benefit is performance while maintaining replication, such as in cases where there is synchronous replication on the backend, and the multipath mechanism can switch on failure.

This can be done in either of two ways:

NVMeoF ANA (Asynchronous Namespace Access) is the NVMeoF native way of doing multipathing, and it offers an interesting set of features. A spec is available for more under-the-hood details [1].

Using DeviceMapper for multipathing is an industry standard approach (while NVMeoF ANA support is not yet) with connections such as iSCSI, and can be equally utilized with NVMeoF connections. The goal of DM component of this spec’s implementation is to bring the NVMeoF connector up to speed with using DeviceMapper for multipathed connections.

Use Cases

Whether via native multipath or DeviceMapper, the use case is the ability to expose multipathed connections to volumes, mitigating connection single point of failure and improving performance.

For example, an NVMeoF volume has two paths (portals) and a multipath connection is established to it. If one of the paths breaks (such as in a partial network failure) - the attached device would remain functional, whereas with single pathed connections, the device would become unavailable to the host it was attached to.

Proposed change

The multipathing implementation will be broken up into its two main component features - Native and DM multipathing:

Phase 1 (Native)

The NVMeoF connector will check the host’s multipath capability and report it in the connector properties passed to the volume driver, this will also allow any driver / backend specific handling.

The value of /sys/module/nvme_core/parameters/multipath can be Y or N - this should be returned as a True or False nvme_native_multipath property in get_connector_properties method from the NVMeOFConnector connector class.

The “new” NVMeoF connector API (introduced in Wallaby) supports passing multiple portals for the same NVMeoF device. Currently only the first successful connection is established.

In multipath mode, the portals connection method should simply connect to all portals by running the nvme connect cli command for each one.

The NVMe system under the hood will create invisible devices for each of the portal connections, and expose a single multipath device on top of them.

It should be noted that the native multipath spec requires the target portals to expose the device under the same target NQN and namespace id for multipath to happen. (This is the backend’s responsibility and should be transparent to OpenStack code)

Finally, the new device will be matched by its uuid as is already implemented in the baseline single path connector.

The connection information for a native multipath volume will look like:

{
    'vol_uuid': '...someuuid...',
    'target_nqn': '...somenqn...',
    'portals': [
        ('10.1.1.1', 4420, 'tcp'),
        ('10.1.1.2', 4420, 'tcp')
    ]
}

Regarding enforce_multipath - it is a DeviceMapper multipathing parameter and it will not have any interaction with the native NVMe multipathing.

Phase 2 (DeviceMapper)

For DeviceMapper multipathing, use dm_replicas in the connection information to specify all the replicas. This approach allows for multiple NQNs for one synced volume, which is often a requirement for DM-based multipathing.

dm_replicas cannot be used together with volume_replicas which is for RAID replication only. The drivers should know not to pass that. If both are passed, the operation should fail.

As in the baseline connector, connect to all dm_replicas (it is worth mentioning that each replica could have multiple portals, essentially allowing this DeviceMapper aggregation to run on top of native multipath devices)

Once connections are established, DeviceMapper will handle the multipathing under the hood (should be transparent to OpenStack code)

DeviceMapper needs to recognize that two devices should be “merged” It does so based on device id information Therefore, it is the responsibility of the backend to expose devices with id information that conforms to the device matching requirements of DeviceMapper.

Since DeviceMapper recognizes two devices should be “merged” based on device id information (udev?) - it is the responsiblity of the backend to handle its exposed devices to conform to the requirement.

The connection information for a DeviceMapper multipath volume will look like:

{
    'vol_uuid': '...someuuid...',
    'dm_replicas': [
        {
            'vol_uuid': '...someuuid1...',
            'target_nqn': '...somenqn1...',
            'portals': [('10.1.1.1', 4420, 'tcp')]
        },
        {
            'vol_uuid': '...someuuid2...',
            'target_nqn': '...somenqn2...',
            'portals': [('10.1.1.2', 4420, 'tcp')]
        }
    ]
}

Connector API Summary

With this proposal, there will be multiple replication and/or multipath mode permutations of operation for the connector: - Single pathed connection - Native multipathed connection - RAID connection (single or multipathed) - DeviceMapper connection (single or native multipath)

Note: RAID on top of DM is not allowed as it is inconsistent with the connection information API (and would need an unnecessary overcomplication of it as well as the physical implementation). It could theoretically be added one day based on need, but it is overkill on multiple levels.

Single path connection information for cinder driver:

{
    'vol_uuid': '...someuuid...',
    'target_nqn': '...somenqn...',
    'portals': [('10.1.1.1', 4420, 'tcp')]
}

Connection information for cinder driver that supports native multipathed connections (os-brick will iterate them if a single path connection is requested):

{
    'vol_uuid': '...someuuid...',
    'target_nqn': '...somenqn...',
    'portals': [
        ('10.1.1.1', 4420, 'tcp'),
        ('10.1.1.2', 4420, 'tcp')
    ]
}

For DeviceMapper, the example below is given with no native multipath for the replica devices. For these replica devices to do native multipathing, portals for each of each device’s paths should be specified in the portals list.

Connection information for DeviceMapper:

{
    'vol_uuid': '...someuuid...',
    'dm_replicas': [
        {
            'vol_uuid': '...someuuid1...',
            'target_nqn': '...somenqn1...',
            'portals': [('10.1.1.1', 4420, 'tcp')]
        },
        {
            'vol_uuid': '...someuuid2...',
            'target_nqn': '...somenqn2...',
            'portals': [('10.1.1.2', 4420, 'tcp')]
        }
    ]
}

Alternatives

Both alternative for NVMeoF multipathing that I am aware of are described in this spec: Native and DeviceMapper

Data model impact

None

REST API impact

None

Security impact

Same as the baseline NVMeoF connector:

Sudo executions of nvme
Needs access for reading of root filesystem paths such as:

/sys/class/nvme-fabrics/…

/sys/class/block/…

Active/Active HA impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

The main benefit is of increased I/O while maintaining replication - such as in a synchronous replication active-passive failover multipath configuration.

One performance impact should happen during attachments on the host, where now multiple NVMeoF devices (rather than just one) would need to be connected, and a new device exposed on top of them.

Depending on the mode of multipath connection (for example active-active vs active-passive) - there could be an increase in network resource usage for the multipath device io.

Other deployer impact

None

Developer impact

Driver developers that want to use NVMeoF will need to be aware of the connector API - specifically the connection_information spec described

It is also worth mentioning that storage backends (although transparent to OpenStack code) will need to keep in mind the system level requirements for whichever multipath mode they use (Native or DeviceMapper)

Implementation

Assignee(s)

Zohar Mamedov (zoharm)
Anyone Else

Work Items

Phase 1 (Native):

Connector properties check kernel supports native multipath
Establish all (multiple) necessary connections to multipath volume and ensure proper device is exposed.

Phase 2 (DeviceMapper):

Ensure all sync replicated devices are connected.
Ensure the proper DeviceMapper device is exposed.

Dependencies

Baseline dependencies are same as baseline single pathed connector: relevant nvmeof kernel modules, and nvme-cli

For native NVMe multipathing, the kernel needs to support it

For DeviceMapper multipathing, DeviceMapper needs to be operational

Testing

Unit tests. Also there is an upcoming gate job that will use LVM to test NVMeoF (where multipath testing for it can be included there)

Documentation Impact

Document the new connection_information API described in this spec - especially for drivers to be able to follow and use it.

References

Optimze upload volume to image for RBD backend

Wed, 22 Sep 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/optimize-upload-volume-to-rbd-store

This spec proposes to optimize the upload volume to image operation from cinder rbd backend to glance rbd store by using COW cloning.

Problem description

Currently when doing a upload volume to image operation and the source (cinder) and destination (glance) backends are rbd, we don’t have any optimization and the generic code path to copy data chunk by chunk is executed. This can be improved by doing a COW clone as we do it incase of creating volume from image [1].

Use Cases

We want to improve the performance when uploading a volume from cinder rbd backend to glance rbd store using COW cloning.

Proposed change

The changes will be needed on both glance and cinder side.

Glance

Expose store type and rbd specific store info (like rbd pool name for glance in this case) via stores info API [2]. This can be extended to other stores by providing store specific details with the stores info API like we do with get pools API [3] on cinder side. Further information regarding its implementation can be provided with a glance side spec.

Cinder

Get the stores info from Glance and find the default store. If the glance default store is rbd and cinder is using rbd backend to upload the volume to image then we will proceed with the COW clone operation from the volume pool to the images pool (pool name provided by glance) and register the image location on glance side.

We will also introduce a new config parameter enable_clone_optimization to enable/disable this optimization on cinder side. More information about it can be found in Security impact section.

Alternatives

None

Data model impact

None

REST API impact

None. This is RBD driver specific feature.

Security impact

Since this optimization skips the writing of image that happens on the glance side, it will also skip the checksum and hash value calculated in that scenario. We will introduce a new config parameter enable_clone_optimization to enable/disable this optimization and also mention the security risks that comes with enabling it. By default, it will be disabled.

Active/Active HA impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Uploading volume to image in case of cinder RBD to glance RBD will be significantly improved.

Image size	2GB	3GB	5GB
Time without COW clone	1min17Sec	1min15Sec	2min49Sec
Time with COW clone	1.29Sec	2.32Sec	1.63Sec
	-98%	-97%	-99%

Other deployer impact

The Cinder service user will need to have read and write to the Glance pool. Note that Read access might already be granted and only write access needs to be provided.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Rajat Dhasmana (whoami-rajat)

Work Items

Query glance for stores info (After exposing details from glance side)
Do a COW clone if the volume to be uploaded is in glance RBD store (volume is uploaded in glance default store)

Dependencies

Glance side changes are required to expose store type and RBD store info like glance pool name.

Testing

Unit tests
Tempest or manual testing for checking if RBD image dependency blocks deletion of source resource
- Upload volume to image
- Delete the original volume
AND
- Upload volume to image
- Create volume from image
- Delete the image

Documentation Impact

None

References

Make Cinder Consistent and Secure RBAC Ready

Fri, 17 Sep 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/s-rbac-ready-x

Revise Cinder’s policy definitions to support the community-wide Consistent and Secure RBAC effort.

Problem description

Cinder policies currently rely only on roles and don’t recognize scope, which was introduced into Keystone in Queens. As a result, implementing something like an administrator who can only perform nondestructive actions in Cinder is very complicated and error-prone. (See, for example, Policy configuration HowTo in the Cinder documentation.)

Using token scope and other Keystone Queens-era improvements such as role inheritance, it is possible to define policy rules that recognize a set of useful “personas”. If all OpenStack services define policy rules to support this (which is the “consistent” part), operators will not have to rewrite policies in an attempt to create such personas themselves, but can instead use tested default policies (which is the “secure” part).

Use Cases

Some examples:

An operator wants to have an administrator who can perform only nondestructive actions.
An operator wants to have different levels of end user in each project (for example, a “project administrator” who can do more in the project than just a “project member”).

Proposed change

Make changes to the cinder default policies so that the default configuration recognizes the keystone ‘reader’ role and treats it appropriately. This will entail rewriting all policies that don’t govern “Admin API” calls.

For Xena, we’ll implement three personas using project-scope only. What this means is that any cinder user must have a role on a project in order to pass policy checks. This is basically what we have now, except that we’ll distinguish the ‘member’ role on a project as distinct from a ‘reader’ role on a project.

Note

What exactly “personas” are and what they can do relative to the Block Storage API are described in the Policy Personas and Permissions document.

Additionally, we’ll address the following issues:

New tests will need to be added, because previously we only needed to distinguish between a “cinder administrator” and an “end user”, whereas now we’ll have to make distinctions between a larger number of “personas” who can make API calls.
We currently have some unrestricted policies (that is, the check string is ""). Their checkstrings will be rewritten to something more specific and appropriate.
We currently have some single policies that govern create, read, update, and delete operations on cinder resources. These will need to be split up into finer-grained policies so that a user with only a ‘reader’ role can read a resource without modifying it.

See the “Implementation Strategy” outlined in the Policy Personas and Permissions for more details.

Alternatives

Do nothing. This is not really an option, however, because in order to keep backward compatibility with legacy policy configuration, it’s not possible for some projects to use scoping and others to ignore it. So if we don’t upgrade our policy definitions, we will block all OpenStack clouds from being able to use Consistent and Secure Policies.

Data model impact

None.

REST API impact

The ability to make various Block Storage API calls is already governed by policies.

Security impact

Overall this should improve security by presenting operators a suitable set of “personas” that will function consistently across project that they can use out of the box instead of implementing their own.

Active/Active HA impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None, we already have to make calls out to keystone to validate tokens and retrieve user privileges.

Other deployer impact

None.

Developer impact

None, other than doing the implementation work.

Implementation

Assignee(s)

Primary assignee:: lbragstad rosmaita
Other contributors:: tosky enriquetaso abishop eharney geguileo whoami-rajat jobernar

Work Items

Natural language description of the (eventual) default configuration. This will be helpful to operators, but has also located some deficiencies in the current cinder policies. It also gives us something against which to validate tests of the default policies. https://review.opendev.org/c/openstack/cinder/+/763306
Policy update patches: https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac
Testing patches. Groundwork patch is https://review.opendev.org/c/openstack/cinder/+/805316

We’ll work with the current structure in cinder of individual files in the cinder/policies directory for specific sets of policies, and add the tests to the same patch where the policies are redefined.
Tempest testing. Groundwork patch is https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915

Initial test patches: https://review.opendev.org/q/project:openstack/cinder-tempest-plugin+topic:secure-rbac

Dependencies

None, the required changes in Keystone and oslo.policy merged long ago.

Testing

We’ll need a flexible testing framework because we are going from 2 personas in Wallaby to 3 personas in Xena to 5 personas in Yoga, and we will need to be sure that the deprecated policy checkstrings continue to work appropriately until they are removed. Thus some kind of ddt-based approach where we have some base tests and run all the different kinds of users through them makes a lot of sense.
As a stretch goal, because of the complexity of the policy configuration, it would be good to have testing in the cinder-tempest-plugin so that each persona can be tested against a real Keystone instance. This would also allow using tempest for testing the consistency of the Secure and Consistent RBAC across openstack projects.

Documentation Impact

The primary documentation for Cinder will be: https://review.opendev.org/c/openstack/cinder/+/763306

We expect that there will be more general documentation for operators in the Keystone docs given the OpenStack-wide nature of this effort.

References

Summary of the general status of the effort at the time of the Xena PTG, including links to more information: http://lists.openstack.org/pipermail/openstack-discuss/2021-April/022117.html

What the personas are and how they are intended to work in Cinder are described in https://review.opendev.org/c/openstack/cinder/+/763306

Add volume re-image API

Fri, 17 Sep 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-volume-re-image-api

This blueprint proposes to add volume re-image API to enable the ability to re-image a specific volume.

Problem description

Currently, users can create a volume with a specific image, but once this volume is created, the image of this volume can’t be changed anymore. If users want to re-image a specific volume, they can only create a new volume with new image, and then delete the old volume, which is not very convenient. However, if the image-backed volume is already used as a boot volume which is already attached to a server, it makes the case more complicated, because changing the image in an attached root volume is not supported.

In addition, as mentioned in nova spec ‘volume-backend-server-rebuild’ [1], if users want to rebuild a volume-backed server, they can complete it by using re-image API, this way there is no new volume with new volume id, we don’t have to worry about types, quota problem, etc.

As the discussion result during Yoga PTG [2], the idea still makes sense and we would propose to implement a new API to support this.

Use Cases

The user wants to re-image a volume without re-creating a new volume with new volume id.
Nova wants a new API to re-image an attached root volume when the server is rebuilt.

Proposed change

This spec proposes to add a new ‘os-reimage’ action to volume actions API.

Add a new ‘os-reimage’ action to volume actions API.
The request will be rejected if cinder microversion is not new enough to support ‘os-reimage’ action.

As mentioned above, the main purpose of this spec is for Nova’s rebuild use case. When Nova calls cinder for re-image then cinder will perform these following steps:

See nova side operations in the nova spec here ‘volume-backend-server-rebuild’ [1]
Receive the re-image request from nova and check if the volume is in reserved state.
Perform the re-image operation. During this, the volume will transition into downloading state which is similar to the state when creating a bootable volume from image (more detailed information is captured later in this section).
Add a new ‘volume-reimaged’ external event and use it notify nova when the re-image operation is complete, like we use for volume-extend. See perform_resize_volume_online for details.
After successful completion of the re-image operation, nova will call cinder to update the attachment with connector info.
Cinder will update the attachment and return connection info to Nova.
After Nova completes the connection with brick, Nova will make the attachment complete call marking the volume in-use.

So, we propose to add a new microversion to support users to re-image a specific volume with new image id. Only reserved, available and error volume can be re-imaged. The reserved volume can only be re-imaged when the reimage_reserved parameter is True. The volume state will be changed to downloading first.

Two separate policies will be introduced, one for re-imaging an available or error volume and another for re-imaging an reserved volume, the default policies for these new actions are SYSTEM_ADMIN_OR_PROJECT_MEMBER.

If the image status is not active, image size or image min_disk size is larger than volume size, will raise InvalidInput (400) error.

Add a new reimage cast rpc. This rpc is used to cast an async message from cinder API to the cinder-volume service or cluster.
Add a new method for volume manager. There will be a new method for volume manager which will be called ‘reimage’, this method will first download the image from glance and then perform the re-image operation.
Add an external event volume-reimaged and call it to notify nova API about completion of reimage operation.

We propose to only support the basic implementation in this spec, the re-image method will call driver’s ‘copy_image_to_volume’ to re-write the specific volume with new image. If the specific image is encrypted, then it will call driver’s ‘copy_image_to_encrypted_volume’. It’s similar to what we have done when we create volume from image.

After the re-image operation is complete, the volume will set back to original state, and all previously existing volume_glance_metadata of this volume will be replaced with the metadata from new image.

If the re-image operation fails, we will set a user message to give hints to the user how they can recover from a potentially now corrupt boot volume and the volume status will changed to error.

There are also some other optimized mechanisms that should be considered to support in future, but will not be implemented in this spec:

Re-image an in-use volume. If we want to support it, we should make sure this volume can be multi-attached, so that we can attach this volume to a host to support copy image to volume. And also re-image an in-use volume is a very dangerous operation, we must make sure there is no IO, onging or cached on VMs before we complete re-imaging, otherwise will damage the volume data. We will not support this operation in this spec.
Re-image a volume from image cache. Currently, Cinder supports creating a volume from image cache, which can improve the performance of creating a volume from an image, we can also consider to add this improvement in the advanced implementation.
Add image signature verification support when re-imaging a volume. There is also a signature when we create volume from image, we will also support it in the advanced implementation. This will be called out in the API reference documentation as a limitation until it is implemented.

Alternatives

If the users want to re-image a volume to same image, the users could revert the volume to a snapshot created after image.

But for different image, the users could first delete the volume and then re-create a volume with a specific new image. And nova could orchestrate creating a new root volume with the new image, then replace it during the server rebuild operation, but there are several issue with that as mentioned above like quota and the questions about what nova should do about the old volume, you can see more detail information in ML [3].

REST API impact

A new microversion will be created to support volume ‘os-reimage’ action API

The rest API look like this in v3:

POST /v3/{project_id}/volumes/{volume_id}/action

{
    "os-reimage": {
        "image_id": "71543ced-a8af-45b6-a5c4-a46282108a90",
        "reimage_reserved": true
    }
}

The <string> ‘image_id’ refers to the id of image. No default value since this is a required parameter.
The <boolean> ‘reimage_reserved’ refers to re-image a volume and ignore its ‘reserved’ status. The ‘available’ and ‘error’ volume can be re-imaged directly, but the ‘reserved’ volume can only be re-imaged when this parameter is ‘true’. Defaults to ‘false’, this is an optional parameter.

The response body of it is like:

{
  "volume": {
      "migration_status": null,
      "attachments": [],
      "links": [
          {
              "href": "http://10.79.144.144/volume/v3/ffc60994a7274553905e5e5a8f890ab3/volumes/d90bfc0e-babf-4478-a591-23ca883ba2be",
              "rel": "self"
          },
          {
              "href": "http://10.79.144.144/volume/ffc60994a7274553905e5e5a8f890ab3/volumes/d90bfc0e-babf-4478-a591-23ca883ba2be",
              "rel": "bookmark"
          }
      ],
      "availability_zone": "nova",
      "os-vol-host-attr:host": "ubuntubase@lvmdriver-1#lvmdriver-1",
      "encrypted": false,
      "updated_at": "2018-09-26T01:55:41.084080",
      "replication_status": null,
      "snapshot_id": null,
      "id": "d90bfc0e-babf-4478-a591-23ca883ba2be",
      "size": 2,
      "user_id": "f33299af48b44050b96bc51104f2290a",
      "os-vol-tenant-attr:tenant_id": "ffc60994a7274553905e5e5a8f890ab3",
      "os-vol-mig-status-attr:migstat": null,
      "metadata": { },
      "status": "downloading",
      "volume_image_metadata": {
          "container_format": "bare",
          "min_ram": "0",
          "disk_format": "qcow2",
          "image_name": "cirros-0.3.5-x86_64-disk",
          "image_id": "24d0c0c3-9e03-498c-926f-ac964cbe2e08",
          "checksum": "f8ab98ff5e73ebab884d80c9dc9c7290",
          "min_disk": "0",
          "size": "13267968"
      },
      "description": null,
      "multiattach": false,
      "source_volid": null,
      "consistencygroup_id": null,
      "os-vol-mig-status-attr:name_id": null,
      "name": null,
      "bootable": "true",
      "created_at": "2018-09-26T01:55:38.735749",
      "volume_type": "lvmdriver-1"
    }
}

The <string> ‘status’ will be ‘downloading’.
The <dict> ‘volume_image_metadata’ refers to the image metadata of the volume. It will include the original image until the re-image operation is complete in cinder-volume.

Normal response codes: 202
Error response codes: 400, 403, 404, 409

Data model impact

None

Security impact

Two new policy rules will be introduced as noted in the Proposed change section.

Notifications impact

Notifications will be added for re-image volume.

Other end user impact

A new command, cinder reimage <volume_id> <image_id> [--ignore-reserved], will be added to python-cinderclient. This command mirrors the underlying API function.

Callers of the new API will need to poll the status of the volume until it goes back to its original status or error in case the operation failed. The only exception here is Nova which will be notified via external events API and doesn’t need to poll for the re-image to be completed.

Since this is a data path change, it will only modify the contents of volume and dependent resources like snapshots or backups won’t be affected by it. Just to keep in mind that restoring to an earlier backup/snapshot will also revert the volume to the old image.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Rajat Dhasmana <rajatdhasmana@gmail.com>

Work Items

By supporting re-image volumes, we need to do the following changes:

Add a new ‘os-reimage’ action to volume actions API.
Add a new ‘reimage’ cast rpc.
Add a new ‘reimage’ method in volume manager.
Adopt the new microversion in python-cinderclient.
Set a user message in the event when the re-image fails.
Add a call to nova external events API with ‘volume-reimaged’ event

Dependencies

None

Testing

Unit-tests, tempest and other related tests will be implemented.

Documentation Impact

Need to document the new behavior of the volume re-image, as well as related client examples, etc.

We also need to mention in the documentation that when the volume is re-imaged, all current content on the volume will be destroyed. This is important as cinder volumes are considered to be persistent, which is not the case with this operation.

References

The Return of Make Cinder Consistent and Secure RBAC Ready

Fri, 17 Sep 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/s-rbac-ready-y

Revise Cinder’s policy definitions to support the Consistent and Secure Default RBAC community goal [1].

Problem description

Use Cases

Some examples:

An operator wants to have an administrator who can perform only nondestructive actions.
An operator wants to have different levels of end user in each project (for example, a “project manager” who can do more in the project than just a “project member”).
An operator wants to have an system administrator who can manage the cinder services, but cannot mess with resources owned by projects that the administrator isn’t a member of.
An operator wants to have an customer support administrator who can can perform admin-only actions on resources owned by various projects, but who cannot modify cinder services.

Proposed change

There’s a long comment in the base policy definition file in the cinder code repository that gives a fairly precise description of how the policy changes made during the Xena cycle would be continued into Yoga. Unfortunately, however, as services have begun implementing Consistent and Secure RBAC, some problems in the initial proposal have been identified, resulting in a direction change for the effort [2]. This means that the above-mentioned strategy for cinder in the Yoga cycle [3] must be completely revised.

The direction change affects cinder in the following ways:

The “system-*” personas are defined to act on the service itself (not on project-owned resources) to the greatest extent possible. (In Xena, these personas were envisioned to act as a cinder super-user and hence be able to perform all operations, both on project resources and on the system itself.)
The “project-admin” persona is now intended to be an administrator (that is, a representative of the operator–definitely not an end user) who can perform admin type actions on project-owned resources (for example, migrating a volume).
A new “project-manager” persona is introduced. This is intended to be an end user who has some extra privileges beyond those of a “normal” user in a project. For example, a project-manager could be given the ability to set a volume type as the default type for the project.

In Yoga, we’ll do the following:

The Cinder Policy Personas and Permissions document merged in Xena [4] contained forward-looking information about what would be done in Yoga. Unfortunately, this is no longer accurate. The stable/xena version of this document should be revised to describe only the Xena changes so that operators aren’t misled about the direction the effort is taking.

There is a patch up addressing this: https://review.opendev.org/c/openstack/cinder/+/818696
The Cinder Policy Personas and Permissions document in the Yoga development branch will need to be revised:
- The range of actions allowed to the “system-*” personas will be restricted to operations on cinder services. These will be “system-scoped” actions.
- The range of actions allowed to the “project-admin” persona will allow this person to perform administrative operations on resources owned by any project on which this person has the ‘admin’ role. These will be “project-scoped” actions.
- The “project-member” and “project-reader” actions will likely be unchanged from Xena, though they will be explicitly “project-scoped” in Yoga.
- There will be some multiple scoped actions. Both system- and project- personas should be able to list volume types, for example (though only a system-admin should be able to create a volume type).
Once the document has been revised, we’ll be able to add a ‘scope_types’ field to each policy rule.
Additionally, we’ll be able to update the check strings for all policies.
Ideally, when a project-admin makes a GET /v3/volumes?all_tenants=1 call, for example, the response should include the volumes owned by all and only the projects on which that project-admin has the ‘admin’ role. For Yoga, we will allow anyone with the ‘admin’ role to see the volumes in all projects (as is the case now). (See the discussion of “Listing project resources across the deployment” [5] in the community goal statement for why it’s being done this way for Yoga.)

At this point, we will have completed Phase 1 [6] of the community goal. This will allow cinder to ship with the oslo.policy configuration option enforce_scope=True in the Z release. (The importance of this is that Phase 3 [7] of the community goal cannot be implemented until a service requires enforce_scope=True.)

We will plan to do Phase 2 [8] of the community goal during the Z development cycle. See the community goal document for details.

As mentioned above, by completing Phase 1 in Yoga, we should also be in a positon to complete Phase 3 during the Z development cycle.

Alternatives

Do nothing. Despite all the work we did in Xena, this is still not really an option, because it’s not possible for some projects to use scoping and others to ignore it while at the same time having consistent personas across projects. So if we don’t upgrade our policy definitions, we will block all OpenStack clouds from being able to use Consistent and Secure Policies.

Data model impact

None.

REST API impact

The ability to make various Block Storage API calls is already governed by policies.

Security impact

Active/Active HA impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None, we already have to make calls out to keystone to validate tokens and retrieve user privileges.

Other deployer impact

None.

Developer impact

None, other than doing the implementation work.

Implementation

Assignee(s)

Primary assignee:

rosmaita
abishop

Other contributors:

tosky
enriquetaso
eharney
geguileo
whoami-rajat
jobernar

Work Items

Update the Cinder Policy Personas and Permissions document in the stable/xena branch.
Update the Cinder Policy Personas and Permissions document in the master for the Yoga policy changes described above.
Add the appropriate scope_types to all policy rules.
Update policy checkstrings to separate system policies from project policies.
Update all tests to reflect the above changes, adding new tests as necessary.
Client changes to support system scope: https://review.opendev.org/c/openstack/python-cinderclient/+/776469

Dependencies

None, the required changes to complete Phase 1 merged long ago in Keystone and oslo.policy.

Testing

Continue to use the testing framework developed in Xena.
We continue to have the stretch goal (mentioned in the Xena spec) to have testing for secure RBAC in the cinder-tempest-plugin, but we do not consider it a requirement for successful completion of this spec.

Documentation Impact

The primary user-facing documentation for Cinder is Policy Personas and Permissions in the Cinder Service Configuration Guide.

Additionally, we expect that there will be more general documentation for operators in the Keystone docs given the OpenStack-wide nature of this effort.

References

NVMe Connector Healing Agent

Tue, 29 Jun 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/nvmeof-client-raid-healing-agent

Daemon that monitors NVMe connections and MDRAID arrays created by the NVMe connector, identifies faulted volume replicas, requests new replicas and replaces faulted replicas with new ones.

Note

This spec has been superseded by NVMeoF Connection Agent.

Problem description

When the NVMe connector connects a replicated volume, OpenStack will see it as one volume, and has no way of monitoring managing and healing the replicas in these MDRAID arrays. This agent will take care of that.

It will monitor the state of the MDRAID arrays and reconcile their physical state on the host with expected state from the volume provisioner, replacing broken legs.

For backend volume replicas, it’s the storage array that takes care of monitoring and replacing unhealthy replicas.

NVMe MDRAID moves the data replication responsibility from the backend to the consumer.

Currently there’s no mechanism to monitor and heal these replicated volumes.

We cannot do it on the Cinder side, because even if the Cinder driver detected the issue and created a replacing volume, we have no mechanism to report the connection information of the replacing volume to the consumer.

So the monitoring and healing needs to be on the volume consumer side.

This agent will also be greatly beneficial for scenarios where certain replicas of an attached replicated volume go faulty, by notifying the volume provisioner of the faulty devices, they can be marked as faulty to avoid using old data on re-attachments and to replace them entirely.

Use Cases

When working with replicated NVMe volumes that are attached to an instance for a long time, one of the replicas may go faulty. This agent will detect it and attempt to replace it (self heal the MDRAID, without the need to detach and re-attach the volume).

Proposed change

Add an “NVMe agent” class that will be initialized by the NVMe connector during volume connection on a host.

Initializing this agent will spawn a monitoring task which will repeat periodically. We are proposing this to be a native thread if possible, but if necessary it can be an independent process.

First proposal was to use python Event Scheduler sched.scheduler, but other alternatives, such as spawning a separate process communicated to via socket, may be chosen instead. One key problem that would need to be addressed by this selection is a scenario where compute service goes down, while the VMs continue operating (and their volumes remain attached) - we don’t want to lose this agent in this case.

When initialized, the agent will read access information to the volume provisioner from a pre-determined config file location, with vendor specific format, the content of which should be provided there by the systems operator.

The task will monitor NVMe devices and MDRAID arrays built over them.

It will know which NVMe devices and MDRAID arrays to monitor based on metadata from the volume provisioner (backend) - which it will have a custom interface to.

It will notify volume provisioner if necessary of failed devices.

It will attempt to connect to new NVMe devices / replicas, replacing them in the MDRAID.

Typical self healing flow:

volume replica goes faulty
agent notices faulty replica, reports to provisioner
provisioner marks replica as bad (so it wont be used later unless synced)
agent keeps pulling volume information from provisioner
certain grace period passes, agent sees no state changes of faulty replica from provisioner, so it sends explicit request to replace replica
provisioner replaces replica and updates volume information
agent pulls volume replica information, notices a replica has changed
agent carries out replica replacement

Alternatives

Operator could use some own script to monitor connections and fix them manually

Data model impact

None

REST API impact

None

Security impact

Will call NVMe connector methods that do sudo executions of nvme and mdadm This will happen in the new agent task that will be spawned from os-brick.

Active/Active HA impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

To allow multiple vendor implementations, the specific methods / logic for:

probing the volume provisioner
pulling / parsing volume metadata from provisioner
reporting volume state changes to provisioner
requesting provisioner to replace replica

Will need to be implemented on a per vendor basis.

The architecture is such that the agent will be a generic class that will provide the interface, and the kioxia implementation will be the first example of vendor-specific implementation.

Implementation

Assignee(s)

Zohar Mamedov: zoharm

Work Items

NVMe connector will launch monitoring task on connect_volume if not running.

Task monitors NVMe devices and MDRAID arrays created by the connector.

When a replica goes faulty (as well as other events such as disconnects) call interface method for notifying volume provisioner.

When replicated volume devices are changed by the volume provisioner, reconcile the physical state of NVMe devices and MDRAID arrays on the host.

Dependencies

None

Testing

We should be able to accept this with just unit tests.

Documentation Impact

Document that using NVMe connector with replicated volumes will optionally launch this agent.

References

Architectural diagram https://wiki.openstack.org/wiki/File:Nvme-of-add-client-raid1-detail.png

Reset State Robustification

Tue, 29 Jun 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/reset-state-robustification

Problem description

I have seen a number of users get volumes into “invalid” states by not understanding how resetting the state of a volume interacts with resetting the state of attachments.

For example, on a volume with an attachment, run “cinder reset-state –state available <vol>”

What now?

Use Cases

Help prevent users who aren’t Cinder experts from shooting themselves in the foot with reset-state.

Proposed change

If a user requests to reset the state of a volume to something that Cinder knows is not a valid state of the universe, reject the request with a reason.

If volume1 has an attachment active. $ cinder reset-state –state available volume1 ERROR: Cannot reset-state to available because attachments still exist.

$ cinder reset-state –state available volume1 –attach-status detached (command succeeds)

Cases to block:

volume reset to available w/ attachments
snapshot reset to in-use with volume in available

Sometimes a knowledgeable operator may need to reset the state anyway and then manually make the current state valid. cinder-manage is the place where forcing a change will be allowed instead of ‘–force’ flag in the API.

Alternatives

Hope users don’t do these things.
Handle the “reset to available while attached case” by forcefully detaching the volumes instead of rejecting the request.

Data model impact

None

REST API impact

os-reset-status actions on volumes, snaps, backups, groups, will now return a 400 in some cases where they would previously succeeded. This does not require a microversion bump.

Security impact

None

Active/Active HA impact

These checks in reset-state could concievably race against updates in a cluster. Will determine what that means when we get further into implementation.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Improves safety for deployers trying to clean up from issues in their cloud.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: TusharTgite
Other contributors:: eharney

Work Items

Implement a check for the reset-state to available while attached case
Add code to cinder-manage to handle the case where an operator needs to override the API and reset the state anyway.
Research other sensible cases we could prevent for volumes, snaps, groups, etc.
Tempest test for a couple of the big cases

Dependencies

None

Testing

New tempest tests

Documentation Impact

Should document common cases where this fails.

References

Original proposal: https://review.opendev.org/c/openstack/cinder-specs/+/682456

Sizing encrypted volumes

Tue, 29 Jun 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/sizing-encrypted-volumes

Problem description

If you have an unencrypted volume and migrate to an encrypted type, it fails because there isn’t enough space on the target to fit the entire source volume. This happens because an encrypted volume must have a header, which takes up some space.

In fact, the exact amount of space lost isn’t clear, it may vary by LUKSv1/LUKSv2 and PLAIN dm-crypt formats. It’s probably on the order of 1-2 MB, but we allocate space in GB. Whether you can do really fine-grained space allocation or not probably depends on the backend containing the volume. However, we also need to keep the gigabyte boundaries for accounting purposes (usage, quotas, charges, etc.) because that’s what everyone’s tooling expects.

In addition, drivers optimize migration in different ways, and since we previously didn’t have a new size parameter, there’s no easy way to get this info into the drivers.

Taking everything into consideration, we could have a flag indicating that it’s OK for the volume to be increased in size if it’s required for the retype to succeed. This way we don’t go behind the user’s back – we fail if the volume won’t fit, and allow them to retype to a new size if they want the retype to succeed.

Use Cases

The main use-case here is when upgrading an old deployment and would like to encrypt old volumes after that.

Proposed change

The manager.py will create a destination volume bigger <source-volume-size+1> GB size if the flag ‘allow-resize==on_demand’ and the destination volume-type is encrypted.
Driver-specific method ‘_copy_volume_data’ for copy volume data when using block devices (i.e LVM will still use the actual _migrate_volume_generic). This method will be called after _before_volume_copy during volume migration.
RPC minor version for a new param 3.x - Add allow_resize to retype method and increase_size to migrate method.
Add a new optional field, “allow_resize”, to the os-retype action. Possible values will be “on-demand” or “never” (which are the same as the current “migration_policy” field). If not specified, the default is “never”.
Add a new optional field, “increase_size”, to the os-migrate action. The operator specifies how many GB to grow the volume (so the final size is current size + increase_size).

Add the next parameters to Python-cinderclient for retype and migrate commands:

Retype: New flag --allow-resize indicating that it’s OK for the volume to be increased in size if it’s required for the retype to succeed.

Optional Arguments:
--allow-resize <never|on-demand>
Allow the volume to be increased in size during the retype if the system
decides it is necessary. This argument is recommended when retyping to
an encryption type.
...

Migration: allow a new size to be specified during a volume migration. New optional argument added --increase-size'

Optional Arguments:
--increase-size [<size>] Number of GiBs to grow the volume during migration.
This is recommended when migrating to an encrypted host.
...

The displayed size of the volume will be the actual size of the volume.

Alternatives

Leave everything as it is and don’t allow this kind of migration. To help users add some workaround to the documentation (i.e. backup the volume and then restore the volume to a bigger one). <1GB size increases won’t be allowed, as explained in the problem description above.

Data model impact

None

REST API impact

Microversion bump
New parameters should be passed to retype and migrate commands as explained in the ‘proposed change’ section.

Security impact

None. If you’re allowed to migrate at all, you should be able to increase the size, and if you’re allowed to retype, you should be able to allow-resize.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Sofia Enriquez <lsofia.enriquez@gmail.com>

Work Items

Implement new logic in cinder
Implement new logic in python-cinderclient
Unit-tests
Tempest tests

Dependencies

None

Testing

Unit tests and current devstack-based jobs won’t be enough to test these changes. At least two tests should be added to cinder_tempest_plugin/api/volume/admin/test_volume_retype.py

retype unencrypted volume to LUKS type.
retype encrypted LUKS volume to regular type.

Documentation Impact

Currently I’ve added some documentation notes warning about retyping an unencrypted/encrypted volume. However, these notes should be replaced with proper documentation.

Notes to be removed - https://review.opendev.org/#/c/732988/ - https://review.opendev.org/#/c/745199/

References

https://etherpad.opendev.org/p/sizing-encrypted-volumes
Victoria Mid Cycle https://wiki.openstack.org/wiki/CinderVictoriaMidCycleSummary
Victoria PTG https://wiki.openstack.org/wiki/CinderVictoriaPTGSummary#Sizing_encrypted_volumes

Expose user-visible extra specs in volume types

Thu, 24 Jun 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/expose-user-visible-extra-specs

This spec proposes definition of a class of user visible extra specs. These will be shown to regular users (members or readers in a keystone project) when they list extra specs or show volume types or extra specs.

Problem description

Cinder only allows cloud administrators to see the extra specs in volume types. This makes sense for specs that control knobs for use by drivers, that select a particular backend, or that would otherwise reveal backend specifics. But it is problematic for backend independent extra specs like multiattach or RESKEY:availability_zones – specs that carry critical information for regular users when they select volume types to use as they create volumes.

The traditional response to this problem has been that cloud administrators should add this critical information to the volume-type description field or provide it in the user documentation for their cloud.

Whatever one thinks of the adequacy of that response when the cloud users are human beings, it clearly fails when the end user is an automated agent.

Use Cases

OpenShift cluster storage operators are automated agents that set up Kubernetes Storage Classes on underlying infrastructure. When that infrastructure is provided by OpenStack, the Cinder CSI operator will create a Storage Class corresponding to each Cinder volume type. OpenShift users in turn reference these Storage Classes in Persistent Volume Claims when dynamically provisioning Persistent Volumes for use by containerized applications.

When provisioning a persistent volume via Cinder CSI with ReadWriteMany access mode, the persistent volume claim should specify block volumeMode (default is filesystem) and reference a Storage Class corresponding to a Cinder volume type with a "multiattach": "<is> True" extra spec.

Because OpenShift administrators and all the software that runs on their behalf are in the general case just regular OpenStack users rather than OpenStack administrators, they currently lack the ability to discover extra specs for volume types corresponding to their Storage Classes and are hindered in their ability to “do the right thing” for OpenShift users.

Cinder CSI now has a topology feature as well where it will likely make sense to be able to see RESKEY:availability_zones extra specs when setting up Storage Classes corresponding to volume types.

Proposed change

Define a set of user visible extra specs in the Cinder code and modify the REST API view for showing extra specs to reveal these in policy based request contexts. The initial set of user visible extra specs is sufficient to establish a framework, and meet the immediate needs described in the use cases. The set is intentionally small, and leverages existing extra specs. The following extra spec keys are to be treated as user visible:

multiattach
RESKEY:availability_zones
replication_enabled

The set of user visible extra specs will be a fixed list defined in the Cinder code, and can be extended but only by enhancing the code. Future updates can support additional volume characteristics that can be expressed with an extra spec. For example, it might be useful for users to know whether a volume type provides volumes that support online extend, but there currently is no extra spec associated with that feature.

The REST API behavior will be policy based, and not require a new microversion. The existing policies for accessing extra specs will remain, but the default values will be changed to grant access to any authorized user. A new policy will determine whether access is granted to all extra specs, or just user visible extra specs. The policies will allow users to view extra specs, but the view will be restricted to the extra specs that are considered user visible. A new policy will control access to all extra specs, including the ones that are user visible.

The current policies (and their default values) for accessing extra specs are:

Policy	Default
volume_extension:access_types_extra_specs	admin-only
volume_extension:types_extra_specs:index	admin-only
volume_extension:types_extra_specs:show	admin-only

The proposed policies, and their new default values, will be:

Policy	Default
volume_extension:access_types_extra_specs	any authorized user
volume_extension:types_extra_specs:index	any authorized user
volume_extension:types_extra_specs:show	any authorized user
volume_extension:types_extra_specs:read_sensitive	admin-only

The new volume_extension:types_extra_specs:read_sensitive policy will govern whether access is granted to all extra specs, or is limited to just user visible extra specs. Hereafter in this spec, the new policy will be abbreviated as read_sensitive.

Alternatives

Build a new API to return user visible capabilities and features from volume types without exposing them as “extra specs.” Although this alternative would also solve the problem, it requires more code to write, test, and document.

Control exposure of each individual extra spec entirely by policy. It is not clear how to do this without significant changes to the existing REST API. Nor in our opinion is it desirable. There are clear cut examples of backend independent capabilities and features that are properly the business of any user who is authorized to create volumes, so users should have a reasonable expectation that these can be discovered without burdening the cloud administrator with the task of constructing policies for them one by one.

The new volume_extension:types_extra_specs:read_sensitive policy is introduced, but the existing policies remain defaulting to admin-only. The downside is this would require cloud administrators to modify the default policies for the user visible extra specs to be available to non- administrative API requests. However, the entire premise is that user visible extra specs are, by definition, intended to be visible to users. Cloud administrators shouldn’t need to opt-in to reap the benefits of this feature. Cloud administrators who wish to retain the current behavior, where users are not authorized to see any extra specs, may do so by explicitly setting these policies to admin-only (the previous default value).

volume_extension:access_types_extra_specs
volume_extension:types_extra_specs:index
volume_extension:types_extra_specs:show

Data model impact

None

REST API impact

The REST API affected by this spec are:

/v3/{project_id}/types
/v3/{project_id}/types/{volume_type_id}
/v3/{project_id}/types/{volume_type_id}/extra_specs
/v3/{project_id}/types/{volume_type_id}/extra_specs/{key}

The following examples document the behavior for a volume type with two extra specs:

multiattach (a user visible extra spec)
volume_backend_name (which will be visible only when the context satisfies the read_sensitive policy)

GET /v3/{project_id}/types

This REST API returns a list of volume types, and the following example shows just a single volume type. When the context doesn’t satisfy the read_sensitive policy, the extra_specs field is included but only user visible entries are present.

Policy	Context
volume_extension:access_types_extra_specs	True
volume_extension:types_extra_specs:read_sensitive	False

{
    "volume_types": [
        {
            "id": "6685584b-1eac-4da6-b5c3-555430cf68ff",
            "qos_specs_id": null,
            "name": "vol-type-001",
            "description": "volume type 0001",
            "os-volume-type-access:is_public": true,
            "is_public": true,
            "extra_specs": {
                "multiattach": "<is> True"
            }
        }
    ]
}

When the read_sensitive policy is satisfied (by default, this will be the case only for administrators) the full list of extra_specs is returned.

Policy	Context
volume_extension:access_types_extra_specs	True
volume_extension:types_extra_specs:read_sensitive	True

{
    "volume_types": [
        {
            "id": "6685584b-1eac-4da6-b5c3-555430cf68ff",
            "qos_specs_id": null,
            "name": "vol-type-001",
            "description": "volume type 0001",
            "os-volume-type-access:is_public": true,
            "is_public": true,
            "extra_specs": {
                "multiattach": "<is> True",
                "volume_backend_name": "SecretName"
            }
        }
    ]
}

GET /v3/{project_id}/types/{volume_type_id}

The REST API shows the details for the specified volume type, and the behavior is similar to the to the /v3/{project_id}/types API.

The extra_specs field will contain any user visible extra specs.
When the read_sensitive policy is satisfied, the full list of extra_specs is returned.

GET /v3/{project_id}/types/{volume_type_id}/extra_specs

The policy will be updated to allow any authorized user access to this API. But, similar to above, for non read_sensitive requests the list of extra_specs will be limited to those that are user visible.

Policy	Context
volume_extension:types_extra_specs:index	True
volume_extension:types_extra_specs:read_sensitive	False

{
    "extra_specs": {
        "multiattach": "<is> True"
    }
}

If there are no user visible extra specs defined for the specified volume type, then an empty dictionary will be returned.

{
    "extra_specs": {}
}

When made by an administrator, the response will be the complete set of extra specs (no change from current behavior).

Policy	Context
volume_extension:types_extra_specs:index	True
volume_extension:types_extra_specs:read_sensitive	True

{
    "extra_specs": {
        "multiattach": "<is> True",
        "volume_backend_name": "SecretName"
    }
}

GET /v3/{project_id}/types/{volume_type_id}/extra_specs/{key}

When the specified extra-specs {key} is in the user visible set, the value will be returned regardless of whether the requester satisfies the read_sensitive policy.

Policy	Context
volume_extension:types_extra_specs:show	True
volume_extension:types_extra_specs:read_sensitive	N/A

{
    "multiattach": "<is> True"
}

The response to requests for extra specs that are not user visible will depend on the context. If the request satisfies the read_sensitive policy then the appropriate response is returned.

Policy	Context
volume_extension:types_extra_specs:show	True
volume_extension:types_extra_specs:read_sensitive	True

{
    "volume_backend_name": "SecretName"
}

But if the request does not satisfy the read_sensitive policy then the API will return 404 NOTFOUND. Returning 404 (instead of 403) prevents leakage of the names of read_sensitive extra spec keys.

Policy	Context
volume_extension:types_extra_specs:show	True
volume_extension:types_extra_specs:read_sensitive	False

{
    "itemNotFound": {
        "code": 404
        "message": "Volume Type 6685584b-1eac-4da6-b5c3-555430cf68ff has no extra specs with key volume_backend_name."
    }
}

Security impact

None. The set of user visible extra specs is hard coded and does not include extra specs that reveal back end details.

Active/Active HA impact

None. REST API layer change with no locking or exclusion or service restart implications.

Notifications impact

None – no asynchronous operations involved.

Other end user impact

Cinderclient and OSC and Horizon should not need changes. Regular users will see some new information but no new fields are required.

Performance Impact

None

Other deployer impact

No impact on OpenStack deployment tools or deployment planning/design.

Developer impact

No impact on OpenStack development outside the scope of this change.

Implementation

Assignee(s)

Primary assignee: abishop

Work Items

Define list of user visible extra specs.
Define the read_sensitive policy, and modify the existing default policy rules that govern access to extra specs to allow access to any authorized user.
Update the API responses per this spec.
Modify unit tests to check for exposure of exactly and only the user visible extra specs without read_sensitive authorization.
Tempest tests

Dependencies

Testing

New tempest tests in cinder-tempest-plugin

Documentation Impact

Add section to Cinder Administration that defines and lists user visible extra specs. Describe the new read_sensitive policy, and explain the policy settings that will give an operator “legacy” behavior.
Update API reference

References

None

NVMeoF Connection Agent

Tue, 15 Jun 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/nvmeof-connection-agent

Daemon that monitors NVMeoF connections and MDRAID arrays created by the new NVMeoF connector. It reports initiator-side events to the storage orchestrator, identifies faulted volume replicas, requests new replicas and replaces faulted replicas with newly assigned ones.

Problem description

When the NVMe connector connects a client-replicated volume, OpenStack will see it as one volume, and has no way of monitoring managing and healing the replicas in these MDRAID arrays. This agent will take care of that.

Currently there’s no mechanism to monitor and heal these replicated volumes. We cannot do it only on the Cinder driver side because currently there is no integrated mechanism to detect initiator connection events and carry out replica replacement on the compute node.

For target-side volume replication (traditional approach), it is the storage backend that takes care of monitoring and self healing. The NVMe + MDRAID approach moves the data replication responsibility from the storage backend to the consuming initiator (ie. compute node).

So the monitoring and healing needs to be on the initiator / compute side.

With this approach, the agent will monitor the NVMeoF connections and report changes to the storage orchestrator / provisioner. It will monitor MDRAID arrays and reconcile their physical state on the host with expected state from the volume provisioner, replacing broken legs.

Finally, orchestration decisions / optimizations will be carried by the volume orchestrator / provisioner using reported information from agent monitoring. Though this is outside the scope of the agent (it is storage backend implemented functionality) - it is useful to mention here that it will handle cases such as avoid using faulty replicas during re-attachment scenarios, because in this design approach only the initiator node can detect the replicas’ sync states of its MDRAID arrays.

Use Cases

When working with replicated NVMeoF volumes that are attached to an instance for a long time, one of the replicas may go faulty. This agent will detect it and attempt to replace it, i.e., self heal the MDRAID array, without the need to detach and re-attach the entire volume from the instance.

Additionally, the agent will detect and report connection and replica sync state events to the storage orchestrator (or potentially other endpoints that can make use of it) - which is gathered by the storage backend for making storage provisioning / orchestration decisions, as well as for telemetry.

Proposed change

Add an agent entry point code to os-brick, such as: os_brick/os-brick/cmd/agent.py

Add an entry_points console_scripts entry in os-brick’s setup.cfg

The agent main function will first initialize the agent by reading access information to the volume orchestor / provisioner from a pre-defined config file (such as /etc/nvme-agent/agent.conf ?)

Vendor specific params will be used and prefixed by the vendor prefix, such as: kioxia_provisioner_ip kioxia_provisioner_port kioxia_provisioner_token kioxia_cert_file

Once initialized, the agent will start a periodic task that will do the following:

Host probe / heartbeat to the storage orchestrator / provisioner
Pull volume metadata (connection and replication state) from provisioner
Monitor NVMeoF devices and MDRAID arrays belonging to it
Detect connection and replication state changes and report to provisioner
Request replacements for faulty replicas
Reconcile replica states from provisioner (carry out the replacements)

Typical self healing flow:

volume replica goes faulty
agent notices faulty replica, reports to provisioner
provisioner marks replica as bad (so it wont be used later unless synced)
agent keeps pulling volume information from provisioner
certain grace period passes, agent sees no state changes of faulty replica from provisioner, so it sends explicit request to replace replica
provisioner replaces replica and updates volume information
agent pulls volume replica information, notices a replica has changed
agent carries out replica replacement

Alternatives

Operator could use some own scripts to monitor connections and replicated arrays states, report detected events, and carry out replica replacements manually.

Data model impact

None

REST API impact

None

Security impact

Sudo executions of nvme and mdadm Needs access for reading of root filesystem paths such as: /sys/class/nvme-fabrics/… /sys/class/block/…

Active/Active HA impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

If configured to run by the operator, this will be a new process running on the compute node. Though it will spend most of its time sleeping, it will wake up every 30 seconds to do its periodic tasks: probe the storage provisioner and inspect nvme connections and mdraid states.

These tasks are not compute intensive, with time mostly spent waiting for a response from the storage provisioner, and the nvme and mdraid operations will only have time complexity linear to the number of devices under the control of the agent (which can be treated as constant due to a low upper limit per host). And finally, the performance effect on the network will also be small, since it will only be sending/receiving small amounts of (meta)data across the network.

Other deployer impact

None

Developer impact

To allow multiple vendor implementations, the specific methods / logic for:

probing / heartbeating the storage provisioner
pulling / parsing volume metadata from provisioner
reporting state changes to provisioner
requesting provisioner to replace replica

These all involve communication with and functionality carried out by the storage backend provisioner / orchestrator, will need to be implemented on a per vendor basis.

The architecture is such that the agent will be a generic daemon that will define the interface, and the kioxia implementation will be the first example of a vendor-specific implementation.

Implementation

Assignee(s)

Zohar Mamedov: zoharm

Work Items

Agent entry point and initialization.

Agent periodic tasks:

Host probe / heart beat
Monitoring (connection and replication event detection)
Report events to storage provisioner
Connection and replication state re-conciliation

Dependencies

None

Testing

We should be able to accept this with just unit tests.

Documentation Impact

Document that with this feature os-brick will be coming with a console-script that is used to launch this agent.

Document how to configure the agent for usage.

References

Presentation slides with architectural diagram on slide 2 https://docs.google.com/presentation/d/1lPU8mQ7jJmr9Tybu5gXkbE7NC1ppkMnoBS4cgSFhzWc

Temporary Resource Tracking

Thu, 10 Jun 2021 00:00:00

https://blueprints.launchpad.net/cinder/+spec/temp-resources

Improve Cinder’s temporary resource tracking to prevent related quota issues.

Problem description

Cinder doesn’t currently have a consistent way of tracking temporary resources, which leads to quota bugs.

In some cases temporary volumes use the temporary key in the admin metadata table to mark them and in other cases we determine a volume is temporary based on its migration_status field, there are even cases where volumes are not being marked as temporary. Due to this roundabout way of marking temporary volumes and having multiple options makes our Cinder code error prone, as is clear by the number of bugs around it.

As for temporary snapshots, Cinder doesn’t currently have any way of reliably tracking them, so the code creating temporary resources assumes that everything will run smoothly and the deletion code in the method will be called after successfully completing the operation. Sometimes that is not true, as the operation could fail and leave the temporary resource behind, forcing users to delete them manually, which messes up the quota, since the REST API delete call doesn’t know it shouldn’t touch the quota.

When we say that we don’t have a reliable way of tracking snapshots we refer to the fact that even though snapshots have a name that helps identify them, such as [revert] volume %s backup snapshot and backup-snap-%s, these are also valid snapshot names that a user can assign, so we cannot rely on them to differentiate temporary snapshots.

Use Cases

There are several cases where this feature will be useful:

Revert to snapshot is configured to use a temporary snapshot, but either the revert fails or the deletion of the temporary volume fails, so the user ends up manually deleting the snapshot, and the quota is kept in sync with reality.
Creating a backup of an in-use volume when backup_use_temp_snapshot is enabled fails, or the deletion of the temporary resource failed, forcing the user to manually deleting the snapshot, and the user wants the quota to be kept in sync with reality.
A driver may have some slow code that gets triggered when cloning or creating a snapshot for performance reasons but that would not be reasonable to execute for temporary volumes. An example would be the flattening of cloned volumes on the RBD driver.

Proposed change

The proposed solution is to have an explicit DB field that indicates whether a resource should be counted towards quota or not.

The field would be named use_quota and it would be added to the volumes and snapshots DB tables. We currently don’t have temporary backups, so no field would be added to the backups DB table.

This would replace the temporary admin metadata entry and the migration_status entry in 2 cycles, since we need to keep supporting rolling upgrades where we could be running code that doesn’t know about the new use_quota field.

Alternatives

An alternative solution would be to use the temporary key in the volumes’ admin metadata table like we are doing in some case and create one such table for snapshots as well.

With that alternative DB queries could become more complex, unlike with the proposed solution where they would become simpler.

Data model impact

Adds a use_quota DB field of type Boolean to both volumes and snapshots tables.

It will have an online data migration to set the use_quota field for existing volumes as well as an updated save method for Volume and Snapshot OVOs that sets this field whenever they are saved.

REST API impact

There won’t be any new REST API endpoint since the use_quota field is an internal field and we don’t want users or administrators modifying it.

But since this is useful information we will add this field to the volume’s JSON response for all endpoints that return it, although with a more user oriented name consumes_quota:

Create volume
Show volume
Update volume
List detailed volumes
Create snapshot
Show snapshot
Update snapshot
List detailed snapshots

Security impact

None.

Active/Active HA impact

None, since this mostly just affects whether quota code is called or not when receiving REST API delete requests.

Notifications impact

None.

Other end user impact

The change requires a patch on the python-cinderclient to show the new returned field consumes_quota.

Performance Impact

There should be no performance detriment with this change, since the field would be added at creation time and would not require additional DB queries.

Moreover performance improvements should be possible in the future once we remove compatibility code with the current temporary volume checks, for example not requiring writing to the admin metadata table, making quota sync calculations directly on the DB, etc.

Other deployer impact

None.

Developer impact

By default Volume and Snapshot OVOs will use quota on creation (set use_quota to True) and when developers want to create temporary resources that don’t consume quota on creation or release it on deletion will need to pass use_quota=False at creation time.

Also when doing quota (adding or removing) new code will have to check this field in Volumes and Snapshots.

It will no longer be necessary to add additional admin metadata or check the migration_status, which should make it coding easier and reduce the number of related bugs.

Implementation

Assignee(s)

Primary assignee:: Gorka Eguileor (geguileo)

Work Items

DB schema changes.
DB online migration and OVO changes.
Update existing operations that mark volumes as temporary to use the new use_quota field.
Update operations that are not currently marking resources as temporary to do so with the new use_quota field.
REST API changes to return the use_quota field as consumes_quota.
Cinderclient changes.

Dependencies

None.

Testing

No new tempest test will be added, since the case we want to fix is mostly around error situations that we cannot force in tempest.

Unit tests will be provided as with any other patch.

Documentation Impact

The API reference documentation will be updated.

References

Proposed Cinder code implementation:

Proposed python-cinderclient code implementation:

https://review.opendev.org/c/openstack/python-cinderclient/+/787407

Proposed code to leverage this new functionality in the RBD driver to not flatten temporary resources:

https://review.opendev.org/c/openstack/cinder/+/790492

Snapshotting attached volumes w/o force

Fri, 19 Mar 2021 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/fix-snapshot-create-force

Cinder requires passing the “force” flag to a snapshot create call to create a snapshot from a volume while it is attached to an instance. This is unnecessary, as snapshotting attached volumes results in crash-consistent snapshots, which are useful, sufficient, and one of the most common cases of how snapshots are used in the real world.

Problem description

Most users and other software that create Cinder snapshots actually want crash-consistent snapshots of attached volumes, so making this an exception case is not productive. Code is written to always use “force”, and users learn that it is needed to create snapshots.

In most virtualization platforms for many years, creating crash-consistent snapshots of online disks is not an exception case, it is a normal operation. It should be in Cinder too.

Use Cases

easier for end users
less surprising snapshot API for developers

Proposed change

Introduce a new microversion that no longer uses a “force” flag to control when snapshots can be created for a volume.

This means that snapshot creation succeeds for volumes that are in the “available” or “in-use” state. The “force” parameter is no longer needed, but “force=true” is accepted to reduce code changes required for users who are currently passing this flag from their code.

Snapshot creation with “force=false” will be rejected as invalid after this new microversion.

Alternatives

None

Data model impact

None

REST API impact

Introduce a new microversion for this change
Snapshot creation will succeed for in-use volumes w/o force flag added.
Passing force=True will succeed for in-use volumes as it does currently, but this parameter is no longer needed for this case.
Passing force=False for snapshot creation is not allowed after the new microversion. This is presumably rarely used and removing it reduces ambiguity about what “force=False” would mean when in-use volumes can be snapshotted by default.

Security impact

None

Active/Active HA impact

None

Notifications impact

None

Other end user impact

Minimal cinderclient changes

Performance Impact

Fewer snapshot create calls that return HTTP 400 resulting in the user issuing a second snapshot create call w/ “force” added

Other deployer impact

Developer impact

Implementation

Assignee(s)

Primary assignee:: eharney

Work Items

Fix snapshot-create
New API microversion
Tempest Tests
Look at backup-create (similar problems)

Dependencies

Testing

New tempest tests in cinder-tempest-plugin

Documentation Impact

Minimal

References

Include encryption_key_id in volume and backup details

Thu, 17 Dec 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/include-encryption-key-id-in-details

This spec adds the encryption_key_id field to the volume and backup details for volumes that are encrypted.

Problem description

The current API supplies volume details that include a simple boolean that indicates whether a volume is encrypted. However, when a volume is encrypted, the details do not include the associated Key Manager (e.g. Barbican) encryption_key_id. This makes it difficult to correlate encryption keys stored in Barbican with their associated volumes. Any operator workflow that might benefit from knowing a volume’s encryption_key_id has to read the value from the Cinder database.

A similar situation applies to backups of an encrypted volume. These backups include a clone of the volume’s encryption key, and the backup’s encryption_key_id is not available via the API.

Use Cases

1. An operator wishes to back up and export Cinder volumes and Barbican secrets for disaster recovery (DR), and later restore the volumes. Knowing each encrypted volume’s encryption_key_id will facilitate restoring the correct encryption secret.

2.1 A user or admin wants to identify the Barbican encryption keys used by Cinder volumes and/or backups. This proposal will allow them to iterate over the list of items, and note the encryption_key_id when present in its details.

2.2 A user or admin wishes to know whether a Barbican secret is used by Cinder. This is similar to 2.1, but more from Barbican’s perspective. Given a list of Barbican secrets, the user or admin may want to now which service is using the secret.

Proposed change

This proposal would add a microversion to include the encryption_key_id field in the volume and backup details. The field would be included in the datails only when relevant:

The encryption_key_id is set (not null)
It isn’t the all-zeros value used by the obsolete fixed-key ConfKeyManager

Alternatives

Operators continue to resort to accessing the Cinder database whenever they need to know a volume’s or backup’s encryption_key_id.
The proposed change could be enhanced to require admin privileges.

Data model impact

None

REST API impact

A new microversion will be created, and the encryption_key_id will be added to the volume and backup details.

Security impact

On one hand, the proposed change will allow users to see the Key Manager (Barbican) ID where the encryption secret is stored. But it’s important to bear these factors in mind:

The encryption_key_id is just a UUID, and access to the actual secret is protected by Barbican’s security policies.
A volume’s encryption_key_id is already present in the connection_info returned by the volume attachment API.

Active/Active HA impact

None

Notifications impact

None

Other end user impact

The feature will not affect users beyond the fact that displaying the details of an encrypted volume or backup will include the encryption_key_id field.

The feature does not require changes in cinderclient, openstackclient or openstack-sdk.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Alan Bishop <abishop@redhat.com>

Work Items

Add a new microversion.
Update the API documentation and code.
Add unit tests.

Dependencies

None

Testing

Unit tests provide sufficient coverage, and there are no plans for tempest changes.

Documentation Impact

Update the API documentation.

References

None

NVMe Connector Support MDRAID replication

Fri, 11 Dec 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/nvme-of-add-client-raid-1

NVMe connector replicated volume support via MDRAID. Allow OpenStack to use replicated NVMe volumes that are distributed on scale-out storage.

Problem description

When consuming block storage, resilience of the block volumes is required. This can be achieved on the storage backend with NVMe and iSCSI though the target will remain a single point of failure. And with multipathing, the HA of paths to a target does not handle volume data replication.

A storage solution exposing high performance NVMeoF storage would need a way to handle volume replication. We propose achieving this by using RAID1 on the host, since the relative performance impact will be less significant in such an environment and will be greatly outweighed by the benefits.

Use Cases

When resiliency is needed for NVMeoF storage.

Taking advantage of NVMe over a high performance fabric, gain benefits of replication on the host while maintaining good performance.

In this case volume replica failures will have no impact on the consumer, and will allow for self healing to take place seamlessly.

For developers of volume drivers, this will allow support for OpenStack to use replicated volumes that they would expose in their storage backend.

End users operating in this mode will benefit from performance of NVMe with added resilience due to volume replication.

Proposed change

Expand the NVMeoF connector in os-brick to be able to take in connection information for replicated volumes.

When connect_volume is called with replicated volume information, NVMe connect to all replica targets and create a RAID1 over the devices.

Alternatives

Replication can also be actively maintained on the backend side, which is the way it is commonly done. However, with NVMe on a high performance fabric, the benefits of handling replication on the consumer host can outweigh the performance costs.

Adding support for this also increases the type of backends we fully support, as not all vendors will chose supporting replication on the backend side.

And we can support both without any kind of impact on each one.

Data model impact

NVMe Connector’s connect_volume and disconnect_volume param connection_properties can now also hold a volume_replicas list, which will contain necessary info for connecting to and identifying the NVMe subsystems for then doing MDRAID replication over them.

Each replica dict in volume_replicas list must include the normal flat connection properties.

If volume_replicas has only one replica, treat it as non-replicated volume.

If volume_replicas is ommitted, use the normal flat connection properties for NVMe as in the existing version of this NVMe connector.

non-replicated volume connection properties:

{'uuid': '96e25fb4-9f91-4c88-ab59-275fd354777e',
 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:...'
 'portals': [{
     'address': '10.0.0.101',
     'port': '4420',
     'transport': 'tcp'
 }],
 'volume_replicas': None}

replicated volume connection properties:

{'volume_replicas': [
     {'uuid': '96e25fb4-9f91-4c88-ab59-275fd354777e',
      'nqn': 'nqn.2014-08.org.nvmexpress:uuid:...'
      'portals': [{
          'address': '10.0.0.101',
          'port': '4420',
          'transport': 'tcp'
      }]},
     {'uuid': '12345fb4-9f91-4c88-ab59-275fd3544321',
      'nqn': 'nqn.2014-08.org.nvmexpress:uuid:...'
      'portals': [{
          'address': '10.0.0.110',
          'port': '4420',
          'transport': 'tcp'
      }]},
]}

REST API impact

None

Security impact

Requires elevated priviliges for managing MDRAID. (Current NVMe connector already does sudo executions of nvme cli, so this change will just add execution of mdadm)

Active/Active HA impact

None

Notifications impact

None

Other end user impact

Working in this replicated mode will allow for special case scenarios where for example a MDRAID array with 4 replicas loses connection to two of the replicas, keeps writing data to two remaining ones. Then, after a re-attach from a reboot or a migration, for some reason now has access to only the two originally lost replicas, and not the two “good” ones, then the re-created MDRAID array will have old / bad data.

The above can be remedied by storage backend awareness of devices going faulty in the array. This is enabled by the NVMe monitoring agent, which can recognize replicas going faulty in an array and notify the storage backend, which will mark these replicas as faulty for the replicated volume.

Multi attach is not supported for NVMe MDRAID volumes.

Performance Impact

Replicated volume attachments will be slower (need to build MDRAID array). It’s a fair tradeoff, slower attachment for more resiliency.

Other deployer impact

NVMe and MDRAID and their CLI clients (nvme and mdadm) need to be available on the hosts for NVMe connections and RAID replication respectively.

Developer impact

Gives option for storage vendors to support replicated NVMeoF volumes via their driver.

To use this feature, volume drivers will need to expose NVMe storage that is replicated and provide necessary connection information for it when using this feature of the connector.

This would not affect non-replicated volumes.

Implementation

Assignee(s)

Zohar Mamedov: zoharm

Work Items

All done in NVMe connector:

In connect_volume parse connection information for replicated volumes.
Connect to NVMeoF targets and identify the devices.
Create MD RAID1 array over devices.
Return symlink to MDRAID device.
disconnect_volume destroy the MDRAID.
extend_volume grow the MDRAID.

Dependencies

NVMe and MDRAID and their CLI clients (nvme and mdadm) need to be available on the hosts for NVMe connections and RAID replication respectively. Fail gracefully if they are not found.

Testing

In order to properly test this in tempest, programmatic access will be needed to the storage backend. For example, to fail one of the drives of a replicated volume.

We could also slide by with just a check of connected NVMe subsystems (nvme list) and scan of MDRAID arrays (mdadm -D scan) to see that multiple NVMe devices were connected and a RAID was created.

In either case tempest will need to be aware that the storage backend is configured to use replicated NVMe volumes and only then do these checks.

Aside from that, running tempest with NVMe replicated volume backend will still fully test this functionality, it’s just the specific assertions (nvme devices and RAID info) that would be different.

Finally, we will start with unit tests, and have functional tests in os-brick as a stretch goal.

Documentation Impact

Document that NVMe connector will now support replicated volumes and that connection information of replicas is required from the volume driver to support it.

References

Architectural diagram https://wiki.openstack.org/wiki/File:Nvme-of-add-client-raid1-detail.png

Store volume format info in Cinder

Mon, 02 Nov 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-support-store-volume-format-info

Problem description

Currently, there are several filesystem-based drivers in Cinder which rely on auto detection of formats. Cinder does not store the actual format of volume and supposes all volumes are “raw” format.

One of such problems is seen when using glance cinder store with nfs as cinder backend and the file to be copied is a qcow2 image. When performing the extend operation, qemu-img uses the resize command which detects the volume format as qcow2 by reading the image’s qcow2 header written into the volume. This leads to unsuccessful creation of the image. This is a problem because current code depends on auto detection of format and if the user has written a qcow2 image into the volume, the driver thinks it is a qcow2 volume (rather than qcow2 image stored in a raw volume) hence causing complication. Thus we need to store format info instead of auto detection.

Proposed change

The “format” info will be added to “volume_admin_metadata” of volumes. The “format” will be only set / updated for filesystem-based drivers, other drivers will not contain this metadata. The “format” field will be created by FS drivers at the time of volume creation and is set to the format which the driver is configured to use. The “format” info will be returned in the attachment get API in the connection_info dict. When cloning a volume, this field will be updated in the new volume’s metadata. For existing volumes, we check if format exists in admin_metadata and if not, we can perform qemu-img info to get the format and store it in the admin_metadata at that point.

Alternatives

Some of the fs type driver uses “qemu-img info” command to judge the format of volume. However, this auto-detection method has many possible error / exploit vectors. Because if the beginning content of a “raw” volume happens to be a “qcow2” disk, auto detection method will judge this volume to be a “qcow2” volume wrongly. This is the same issue which cases glance to fail an image create operation when using cinder as a store.

Data model impact

For volumes on filesystem-based drivers, there will be a related record in volume_admin_metadata table. For this record, the key column is “format” and the value is the volume’s format in volume_admin_metadata table.

REST API impact

The “format” info will returned in attachment get API:

/v3/{project_id}/attachments/{attachment_id}

{
    "attachment": {
        "id": "3b8b6631-1cf7-4fd7-9afb-c01e541a073c",
        ...
        "connection_info": {
            "format": "raw",
            ...
        }
    }
}

Security impact

Implements the proper/complete fix for bug 1350504.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Fixes problem where a user can cause a volume to become inoperable by writing a qcow2 header into it.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: whoami-rajat

Work Items

To support this feature, the work items are:

Store format info in volume_admin_metadata table when creating or cloning a volume in fs type drivers.
Modify volume format when doing a snapshot delete (blockRebase) operation on an attached volume.
Pass format info when doing an extend operation.
Return format field in the attachment get API in connection_info dict.

Dependencies

None

Testing

Related unit and functional tests.

Documentation Impact

Add documentation regarding the format field visible in the attachment get API for fs drivers.

References

None

Specify availability_zone and volume_type for backup restore API

Fri, 21 Aug 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/availabilityzone-and-volumetype-for-backup-restore

Add two optional parameters to specify availability_zone and volume_type for cinder backup restore API to better support backup and restore across availability zones.

Backup across availability_zones is a typical scenario for disaster recovery: say, backup driver located in AZ-1 is configured to backup data to backup backend located in AZ-2. When AZ-1 crashes, user tries to restore volume from the backup data located in AZ-2. It would be useful to provide parameters for user to specify the availability_zone and the volume_type of the restore volume.

The availability_zone parameter will enable user to create a restore volume to a specified availability_zone where the backup data is actually located. The geographical locality will enhance the performance for restore copy.

While with the volume_type parameter, the backup restore API can provide more options for user to create a restore volume, say, with rbd backup backend, compared with LVM, user could choose to restore to a rbd volume, where restore will do diff copy instead of full copy and thus enhance restore efficiency.

Problem description

Currently, when user makes a request to restore a backup without volume_id, the backup-restore API will create a restore volume with default volume_type and default availability_zone.

As the availability_zone is none, cinder scheduler will randomly pick up one availability_zone to create this restore volume. In one typical disaster recovery scenario described above, restore volume with randomly selected availability_zone may locate in an availability_zone geographically far away from backup data. In this way, restore might fail if the restore side backup driver could not access backup data across availability_zones. Even if backup driver could access backup data remotely, cross-wan IO will cause a performance penalty. No matter for the sake of correctness or efficiency, user would like to have control of specifying the availability_zone of restore volume.

Without volume type option, the restore volume will be created with default volume type and thus default volume backend. Since the default volume backend may not understand the backup data format, the backup driver may not be able to take the most efficient way to restore from the backup data but will do a full copy. It is better to give user more options to enhance restore performance.

Use Cases

There are customers who want to restore a backup in a certain backend and availability_zone without any restore volume. The customers can specify the volume_type and availability_zone to restore this backup.

Proposed change

In order to solve this problem, we add two optional parameters to specify volume_type and availability_zone for the backup-restore API. The volume_type parameter is to specify the volume backend of the restore volume. The availability_zone parameter is to specify the availability_zone where the restore volume would locate in. Thus, user could choose restore volume type according to backup backend and choose restore volume location geographically close to the backup data location. This change will give user more options to ensure restore correctness and enhance restore performance.

Alternatives

None

Data model impact

None

REST API impact

Modify REST API to restore backup:

POST /v3.x/{tenant_id}/backups/{id}/action
Add volume_type and availability_zone in request body

JSON request schema definition:

'backup-restore': {
    'volume_id': volume_id,
    'volume_type': volume_type,
    'availability_zone': availability_zone,
    'name': name}

Security impact

None

Notifications impact

Ensure type and AZ are included in the restore.start notification.

Other end user impact

End user will be able to restore a backup with specified volume_type and availability_zone.

Performance Impact

This might allow cross-az restore scenarios that weren’t possible before, which might cause network performance degradation.

Other deployer impact

The deployer will be able to restore a backup with specified volume_type and availability_zone.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: alan-bishop
Other contributors:: <None>

Work Items

TBD

Dependencies

None

Testing

Unit tests and tempest tests will be provided.

Documentation Impact

None

References

None

Native Rest Block Driver

Tue, 09 Jun 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/restblock-driver

We, at scality, are working on a Linux native kernel driver: Rest Block Driver. (it is on github, but still private as we want to reach a certain point before releasing it). This is a kernel driver that is able to talk to a REST-based storage service (we plan to eventually support multiple rest backends), in order to attach the files from that storage as native block devices on the host linux. One of the key points is that it allows a REST-based volume storage to take advantage of linux’s block driver cache.

Problem description

As a company, we are interested in providing an Openstack integration to our clients, and as such, we want to provide a new driver for cinder to drive and make use of the aforementionned Rest Block Driver.

This block driver currently offers the following simple features:

Mirror-server list management (failover and load balancing)

Provisioning (create/extend/delete)

Automatic attach/detach of volumes (practical for use within cinder)

Also, we have plans for a few features that should come shortly:

Storage Pool management to provide volumes from different storage pools. Each pool will have their own list of mirrors to choose from.

Real failover implementation

Multiple back-ends support (for multiple REST-like protocols implementations, ours being a CDMI implementation)

Native snapshot management

Use Cases

Proposed change

The Rest Block Driver is mostly controlled through the /sys file system. Thus, most of the control operations will be implemented using the process utils module. For the general feature implementation, here is how we plan to support the multiple required features of a Cinder driver:

Provisioning (create/extend/delete) : natively supported by the kernel driver

Automatic attach of volumes at setup-time: Natively supported by the kernel driver

Snapshots (create/delete) : Supported through LVM tool classes/functions

Copy To/From Image: Supported thanks to image_utils

Volume Cloning (from volume or from snapshot): Supported through copy utils

Please not that this reflects the current state of the kernel driver, and might evolve.

In the future it will provide a way to manage the snapshots from the driver itself, but it is not yet supported. Thus, the driver will use the lvm code tools to manage thin snapshot provisioning and support snapshots for the volumes exposed by our Rest Block Driver.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

Since this driver is controlled through the /sys file system, it requires administrative privileges to be controlled (ie. to write/read from /sys files).

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

As this is a 3rd-party provided driver, it implies setting up the said software first. Other than that, only the new driver-specific configuration values will be added.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <dav.pineau@gmail.com>
Other contributors:: None

Work Items

Provide a cinder driver leveraging both LVM for non-implemented features and the driver itself for the natively supported features.

Dependencies

None

Testing

The driver does not bring any modification to the existing APIs and behaviors. As such, no new tempest tests should be required in our understanding.

As this driver relies on a vendor-specific software, the gate obviously cannot test the driver. We are currently setting up third-party CI testing for both our previous driver (Scality Sofs) and this new one.

Documentation Impact

None

References

None

Support volume local cache

Wed, 20 May 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-volume-local-cache

This blueprint proposes to add support of volume local cache in cinder and os-brick. Cache software such as open-cas [5] can use fast NVME SSD or persistent memory (configured as block device mode) to cache for slow remote volumes.

Problem description

Currently there are different types of fast NVME SSDs, such as Intel Optane SSD, with latency as low as 10 us. What’s more, persistent memory which aim to be SSD size but DRAM speed gets popupar now. Typical latency of persistent memory would be as low as hundreds of nanoseconds. While typical latency of remote volume for a VM can be at the millisecond level (iscsi / rbd). So these fast SSDs or persistent memory can be mounted locally on compute nodes and used as a cache for remote volumes.

In order to do the cache, three projects need to be changed. These are cinder, os-brick, and Nova. Meanwhile related hardware should be added in system, e.g. high performance SSD or persistent memory. The mechanism is similar to volume encryption where dm-crypt is used. Cache software support would be added in os-brick and Nova will call os-brick when it is trying to attach a volume, os-brick then calls the cache software to setup the cache for the volume. After that, a new virtual block device would be created and laying upon the original block device. os-brick will expose this new virtual block device to Nova, meanwhile the original block device mount point will remain unchanged. All of the cache setup and teardown would be handled within os-brick.

A property would be added in the extra specs of the volume type. If a volume type has extra-spec of “cacheable”, then it means the related volumes can be cached in compute node locally.

Like all the local cache solution, multi-attach cannot be supported. This is because cache on node1 doesn’t know the changes made to backend volume by node2. So cinder should guarantee these two properties cannot be set at the same time.

Considering VM migration, cache software should not format the source volume (the volume to be cached). So cache software such as bcache cannot be used. This spec only supports open-cas. open-cas is easy to use, you just need to specify a block device as the cache device, and then can use this device to cache other block devices. This is transparent to upper layer and lower layer. Regarding upper layer, guest doesn’t know it is using an emulated block device. Regarding lower layer, backend volume doesn’t know it is cached, and the data in backend volume will not have extra change because of cache. That means even if the cache is lost for some reason, the backend volume can be mounted to other places and become available immediately. open-cas supports the cache modes below:

Write-Through (WT): write to cache and backend storage at the same time. So the data is fully synced in cache and backend storage. There will not be any data corruption when cache becomes invalid.
Write-Around (WA): like Write-Through, but only cache for volume blocks that are already in cache.
Write-Invalidate (WI): write to backend storage and invalidate cache
Write-Back (WB): write to cache and lazy write to backend storage. This mode has better write performance but is possible to lose data because the latest data is in cache, but may not be flushed to the backend storage.
Write-Only (WO): like write-back, but only cache write, will not cache read. So this mode also has the possibility to lose data.

The first three modes are suitable for scenarios that data integrity must be guaranteed, every write io is both written to cache and backend storage. In these modes it is just like read-only cache feature in ceph. No operation would be blocked. e.g. VM live migration, snapshot taking, volume backup and others can be done as usual. Cache software can also be changed to others rather than open-cas at any time because there’s no dirty data in cache at any time. This is suitable for read intensive scenario, but it will be no benefit for write io because every write io need to go to backend storage everytime for data integrity.

The last two modes are suitable for scenarios that have high requirements for both read/write performance, but low requirements for data integrity. E.g. some test environment. In these two modes, all operations that depend on the backend volume containing full data would not work safely. So VM live migration, snapshot, volume backups, consistency groups, etc, cannot works safely because there may be still dirty data in cache. At least, operator need to stop the disk io by some way and flush the cache(via casadm) before doing these operations. Cache software also cannot be changed to others except all dirty data has been flushed to backend.

Cache mode can be got from cache instance ID via cache admin tool. Nova passes available cache instances ID list to os-brick, so os-brick knows the cache mode of the cache instances. This spec would make os-brick refuse to attach volume with last two cache modes. But cache mode is set outside of OpenStack, Cinder / os-brick cannot control cache mode being modified after volume attached. But it would be well documented that Write-Back (WB) and Write-Only (WO) mode is dangerous and operators should fully know what they are doing when trying to set to these cache modes.

Some storage backends may support discard/TRIM functionality, but cache software doesn’t have sense of this. Cache software evicts data based on policy like ‘lru’(this is the default policy of open-cas).

Some storage client, compute node, may enable multipath for volumes, e.g. iscsi/fc volumes. Volume local cache would not work with multipath, with the same reason of multi-attach. os-brick detects multipath via function get_volume_paths() and would not set cache for volume with multipath.

No restrictions would be introduced on retyping a ‘cacheable’ volume. This is because retyping includes two steps: 1) attach new volume 2) detach old volume. So old volume would be released from caching, new volume would be cached.

This is the shared cache, and the number of volumes to be cached is unlimited. A volume be cached does not mean it will occupy space in cache. The volumes with hot IO will consume more cache space, meanwhile volumes with no IO will not occupy any space in cache devce. The data in cache would be evicted when the data getting cold or other volume’s IO getting hot.

Use Cases

In read intensive scenarios, e.g. AI training, VM volume local cache will significantly boost the storage performance (throughput, and especially disk io latency)

Proposed change

In order to do volume local cache:

In Nova, end user selects a flavor that is advertised as having a volume-local-cache so guest can be landed in a server with cache capability. The end user should expect different performance based on what server flavor was chosen. More information such as error handling, user message, etc, are out of scope of this spec and would be defined in Nova spec [4].
In Cinder, volume type that is ‘cacheable’ should be selected.

It is Cinder which determines and sets the ‘cacheable’ property. A volume marked as “cacheable” doesn’t mean it must be cached, it just means it is eligible to be cached. Nova calls os-brick to set cache for the volume only when the volume has the property of ‘cacheable’.

Cache mode is bound to cache instance. So different cache instances can have different cache modes. All cached volumes share same cache mode if they are cached by the same cache instance. The operator can change cache mode dynamically, using cache software management tool. So cache mode setting is out of OpenStack and is not controlled by OpenStack. Operator should not change cache to unsafe mode. os-brick just accepts the cache name and cache instance IDs from Nova.

cache_name identifies which cache software to use, currently it only supports ‘opencas’. Nova knows what cache name is, based on which cache software is enabled in compute node.

Each compute node can have more than one cache instances. os-brick can weight each cache instance passed in, by e.g. total cache size, how many free space, etc, via cache admin tool(casadm), and select the best one.

Some storage types support “extend volume” which triggered from cinder side. e.g. via command “cinder extend …”. It works normally when the volume is not “in-use”. But if the volume is attached and “in-use”, os-brick would not support to extend and just raise NotImplementedError for the volume with cacheable volume_type. This is because open-cas don’t support volume extending dynamically in current release. But “Resize Instance” feature which triggered from Nova still can work, because volume will be detached and then re-attached during “”Resize Instance”.

The final solution would be like:

                       Compute Node

+---------------------------------------------------------+
|                                                         |
|                        +-----+    +-----+    +-----+    |
|                        | VM1 |    | VM2 |    | VMn |    |
|                        +--+--+    +--+--+    +-----+    |
|                           |          |                  |
+---------------------------------------------------------+
|                           |          |                  |
| +---------+         +-----+----------+-------------+    |
| |  Nova   |         |          QEMU Virtio         |    |
| +-+-------+         +-----+----------+----------+--+    |
|   |                       |          |          |       |
|   | attach/detach         |          |          |       |
|   |                 +-----+----------+------+   |       |
| +-+-------+         | /dev/cas1  /dev/cas2  |   |       |
| | osbrick +---------+                       |   |       |
| +---------+ casadm  |        open cas       |   |       |
|                     +-+---+----------+------+   |       |
|                       |   |          |          |       |
|                       |   |          |          |       |         Storage
|              +--------+   |          |    +-----+----+  | rbd   +---------+
|              |            |          |    | /dev/sdd +----------+  Vol1   |
|              |            |          |    +----------+  |       +---------+
|        +-----+-----+      |          |                  |       |  Vol2   |
|        | Fast SSD  |      |    +-----+----+   iscsi/fc/...      +---------+
|        +-----------+      |    | /dev/sdc +-------------+-------+  Vol3   |
|                           |    +----------+             |       +---------+
|                           |                             |       |  Vol4   |
|                     +-----+----+    iscsi/fc/...        |       +---------+
|                     | /dev/sdb +--------------------------------+  Vol5   |
|                     +----------+                        |       +---------+
|                                                         |       |  .....  |
+---------------------------------------------------------+       +---------+

Changes would include:

Add “cacheable” property in extra-spec of volume type
- Volume local cache cannot work with multiattach. So when adding “cacheable” to extra-spec, cinder should check if “multiattach” property exists or not. If “multiattach” exists, then cinder should refuse to add “cacheable” property to volume type, and vice versa.
- Fill “cacheable” property in connection_info. So os-brick can know whether a volume can be cached or not.
Add a common framework for different cache software in os-brick. This framework should be flexible to support different cache software.

1) A base class - CacheManager would be added and the main functions would be:
- __init__()
  
  This function would accept the parameters from Nova. Parameters include:
  
  root_helper - used for cache software management tools.
  
  connection_info - containing device path
  
  cache_name - specify the cache software name, currently only support ‘opencas’
  
  instance_ids - specify cache instances that can be used. os-brick chooses a best one among these instances
- attach_volume()
  
  This function would be called by Nova (in function _connect_volume) to setup cache for a volume when it is trying to attach the volume.
- detach_volume()
  
  This function would be called by Nova (in function _disconnect_volume) to release the cache when it is trying to detach a volume.
2) In __init__.py, a map of cache software and its python class would be added. So os-brick can find the correct class based on cache name.

CACHE_NAME_TO_CLASS_MAP = {
“opencas”: ‘os_brick.caches.opencas.OpenCASEngine’, …

}

Meanwhile a function like _get_engine() would be added to go through the map to find the correct class.
Add the support for open-cas in os-brick.

Implement functions attach_volume/detach_volume for open-cas.

Code work flow would be like:

            Nova                                        osbrick

                                              +
         +                                    |
         |                                    |
         v                                    |
   attach_volume                              |
         +                                    |
         |                                    |
         +                                    |
       attach_cache                           |
             +                                |
             |                                |
             +                                |
 +-------+ volume_with_cache_property?        |
 |               +                            |
 | No            | Yes                        |
 |               +                            |
 |     +--+Host_with_cache_capability?        |
 |     |         +                            |
 |     | No      | Yes                        |
 |     |         |                            |
 |     |         +-----------------------------> attach_volume
 |     |                                      |        +
 |     |                                      |        |
 |     |                                      |        +
 |     |                                      |      set_cache_via_casadm
 |     |                                      |        +
 |     |                                      |        |
 |     |                                      |        +
 |     |                                      |      return emulated_dev_path
 |     |                                      |        +
 |     |                                      |        |
 |     |         +-------------------------------------+
 |     |         |                            |
 |     |         v                            |
 |     |   replace_device_path                |
 |     |         +                            |
 |     |         |                            |
 v     v         v                            |
                                              |
attach_encryptor and                          |
rest of attach_volume                         +

Volume local cache lays upon encryptor would have better performance, but expose decrypted data in cache device. So based on security consideration, cache should lay under encryptor in Nova implementation.

Alternatives

Assign local SSD to a specific VM. VM can then use bcache internally against the ephemeral disk to cache their volume if they want.

The drawbacks may include:
- Can only accelerate one VM. The fast SSD capability cannot be shared by other VMs. Unlike RAM, SSD normally is in TB level and large enough to cache for all the VMs in one node.
- The owner of the VM should setup cache explicitly. But not all the VM owners want to do this, and not all the VM owners have the knowledge to do this. But they for sure want that the volume performance to be better by default.
Create a dedicated cache cluster. Mount all the cache (NVME SSD) in the cache cluster as a big cache pool. Then allocate a certain amount of cache to a specific volume. The allocated cache can be mounted on compute node through NVMEoF protocol. Then use cache software to do the same cache.

But this would be the compete between local PCIe and remote network. The disadvantage of doing it like this is: the network of the storage server would be a bottleneck.
- Latency: Storage cluster typically provides volumes through iscsi/fc protocol, or through librbd if ceph is used. The latency would be at the millisecond level. Even with NVME over TCP, the latency would be hundreds of microseconds, depending on the network topology. In contrast, the latency of NVME SSD would be around 10 us, take Intel Optane SSD p4800x as example.
Cache can be added in backend storage side, e.g. in ceph. Storage server normally has its own cache mechanism, e.g. using memory as cache, or using NVME SSD as cache.

Similiar with above solution, latency is the disadvantage.

REST API impact

None

Data model impact

None

Security impact

Cache software will remove the cached volume data from cache device when volume is detached. But normally it would not erase the related sectors in cache device. So in theory the volume data is still in cache device before it is overwritten. Unless the cache device is plugged out, otherwise it is acceptable because the volume itself is also mounted and visible on host OS. Volume with encryption doesn’t have this issue if encryption laying upon volume local cache.

Notifications impact

None

Other end user impact

None

Performance Impact

Latency of VM volume would be reduced

Other deployer impact

Need to configure cache software in compute node

Developer impact

The support for other cache software can be added by other developers later

Implementation

Assignee(s)

Primary assignee:: Liang Fang <liang.a.fang@intel.com>

Work Items

Implement a common framework for supporting different cache software
Support open-cas
Unit test be added

Dependencies

None

Testing

Unit-tests, tempest and other related tests will be implemented.
Test case in particular: leverage DRAM to simulate fast ssd, act as the cache for open-cas; Use fio to do the 4k block size rand read test; Compare the result of volume with / without cache. The expected behavior is: cached volume would get lower latency.

Documentation Impact

Documentation will be needed. User documentation on how to use cache software to cache volume.

References

[1] https://review.opendev.org/#/c/663549/ [2] https://review.opendev.org/#/c/663542/ [3] https://review.opendev.org/#/c/700799/

Support modern compression algorithms in cinder backup

Fri, 08 May 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-modern-compression-algorithms-in-cinder-backup

This blueprint proposes to add support of modern compression algorithms like zstd to cinder backup.

Problem description

Cinder backup currently supports only gzip and bzip2 to compress/decompress the volume data during backup and restore operation.

It should support other compression algorithms as well because efficiency of compression entirely depends on cloud provider’s environment and choice. Some may want fast compression compromising the compression ratio, so to provide more choices to cloud provider, it will be good to add support of other algorithm like zstd.

Zstandard(zstd) gives high compression ratio(comparable to zlib) with very much fast speed. It’s now used as package compression algorithm for arch linux.

https://facebook.github.io/zstd/

Some general benchmark result is available on the website introduced above, or lzbench website.

https://github.com/inikep/lzbench/

Use Cases

Cinder backup with high-speed compression algorithm

Proposed change

add zstd algorithm package details in requirements.txt.
add the support of zstd algorithm in get compressor routine of cinder backup.

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

Cinder backup and restore operation gets faster.
Size of backup data may be bigger than zlib or bz2 compression algorithm (depends on content of data volume)
Simple performance test result I have done
- Server spec: 8core CPU, 16GB memory
- Backup store is NFS-backend on localhost (i.e. small network bottleneck)
- Backup/restore a 16GB volume created from a Windows image (qcow2 12001017856 bytes)
- Measured backup/restore time backup data size, and size ratio versus size of qcow2 (considering as approximate amount of data in the volume) The smaller the better for all values.
- CPU usage for algo other than zstd is about 100% to 110% (i.e. Using only one core.) where max usage is 800% (= 8cores * 100%).
- CPU usage for zstd is about 80% to 400% (utilizes multi cores).
- Famous algorithm like gzip may performs better if offloaded to hardware.

Algorithm	backup time(sec)	restore time(sec)	size(bytes)	v.s. qcow2(%)
none	205.8	109.2	17218167818	143.5
gzip	743.2	141.5	6658990044	55.5
bz2	2102.5	742.2	6480248772	54.0
zstd(to be added)	196.2	100.2	6286227236	52.4
lz4(for reference)	211.0	113.0	7903490364	65.9

Other deployer impact

The python module zstd is required to use zstd compression algorithm.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee: Kazufumi Noto

Other contributors: None

Work Items

Add the package details to cinder/requirements.txt.
Implement the code to add support of zstd compression.
Documents need to be updated to reflect the added support.
Add test cases to create backup using zstd compression.

Dependencies

None

Testing

Unit tests will be added to create backup using zstd compression.

Documentation Impact

The following documents need to be updated to reflect the added support:

cinder sample configuration file in configuration reference.

References

Backup Multi-Backends Configuration

Wed, 11 Mar 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/backup-backends-configuration

Add a new config section for backup configuration options to unify the way how we do it with volume drivers and support multiple backup drivers in one service.

Problem description

Right now we’ve got all backup-specific configuration options defined in the [DEFAULT] section:

[DEFAULT]
backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
backup_ceph_user = cinder-backups
backup_ceph_pool = volume-backups
...

It leads to mix of backup-specific config options and Cinder general configuration. This approach also doesn’t support backups multi-backend. Operators can configure multiple backup backends only by configuring several cinder-backup services.

Current approach blocks Generic backup implementation because drivers self.configuration won’t be able to see required configuration which are defined in the [DEFAULT] section.

Use Cases

The main use-case here is to decouple backup-specific configuration from the [DEFAULT] section and introduce a backup_type.

Proposed change

The new config option called ‘enabled_backup_backends’ will contain a list of enabled backup backends. This will implement the same approach as we have for volumes with ‘enabled_backends’ config option.

All backup-related configuration will be moved into the new backend-specific section:

[DEFAULT]
enabled_backup_backends = ceph_backup, ceph_backup_ssd
default_backup_type = ceph-backup-type
...

[ceph_backup]
backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
backup_ceph_user = cinder-backups
backup_ceph_pool = volume-backups

[ceph_backup_ssd]
backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
backup_ceph_user = cinder-backups
backup_ceph_pool = volume-backups-ssd

The old-style backup configuration will be supported at least for one release to follow all deprecation policies.

With multi-backups configuration we want to introduce backup types too. It will help us to use all multi-backend advantages. Backup types will follow the same approach as volume types have:

__DEFAULT__ backup type will be introduced and all existing backups will be migrated to this backup type.
default_backup_type config option will be introduced.
Backup types will have own extra specs.

Alternatives

Leave everything as-is and introduce some hacks in the code to get volume drivers working as a backup drivers in the scope of Generic Backup feature.

Data model impact

New tables for backup types will be introduced. We’ll follow the same approach as for volume types which works well for Cinder:

REST API impact

New API endpoints will be implemented for backup types. It should be similar like volume type API.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Backup creation procedure will contain few more database calls but it should not affect performance a lot.

Other deployer impact

Operators should configure backups using a new mechanism and migrate from old-style configuration.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: e0ne

Work Items

Add a new ‘enabled_backup_backends’ config option and deprecate old-style config
Modify the backup manager to honor new configuration
Add some unit tests
Devstack should be able to configure cinder backups in a new way
Operator documentation should be updated
API reference should be updated

Dependencies

None

Testing

Unit tests
Existing tempest tests will cover new functionality

Documentation Impact

Update documentation to describe new config option
New REST API endpoints will be documented

References

Add backup id to volume’s metadata

Fri, 31 Jan 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-volume-backup-id

Problem description

Currently, the end user can see the source (source by snapshot or image etc.) from the new volume created. But when the end user has restored a backup volume, in the volume show response, we cannot see its source. This can cause a lot of confusion for end users.

Use Cases

As an end user, I would like to know the restored volume comes from which backup resource.

Proposed change

Add the property src_backup_id to the volume’s metadata, to record from which backup the new volume was created from. When restoring from a chain of incremental backups, src_backup_id is set to the last incremental backup used for the restore.

Once added to the volume metadata, the src_backup_id will appear on any API response that displays volume metadata:

the volume-show response (GET /v3/{project_id}/volumes/{volume_id})
the volume-list-detail response (GET /v3/{project_id}/volumes/detail)
the volume-metadata-show response (GET /v3/{project_id}/volumes/{volume_id}/metadata)
the volume-metadata-show-key response, only if the key is src_backup_id (GET /v3/{project_id}/volumes/{volume_id}/metadata/{key})

Vendor-specific changes

None

Alternatives

None

Data model impact

None

REST API impact

None. The volume-show, volume-list-detail, and volume-metadata-show responses are currently defined to contain a metadata element that is either JSON null or a JSON object consisting of a list of key/value pairs. The src_backup_id will appear in this list for appropriate volumes, but this respects the current API and does not require a new microversion.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Xuan Yandong

Work Items

Add src_backup_id to the volume metadata

Dependencies

None

Testing

Add related unit tests
Add related functional test
Add tempest tests

Documentation Impact

Release note should point out that since this is stored in the volume metadata, it can be modified or removed by end users, so operators should not rely upon it being present for administrative or auditing purposes.

Add a similar note somewhere in the admin docs, probably a page about volume metadata written by Cinder (which may not currently exist). In addition to reminding admins that end users can overwrite volume metadata, should explain how to read the src_backup_id (particularly the part about what id is used when restoring from incremental backups).

References

None

Support to query cinder resources filter by time comparison operators

Mon, 13 Jan 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-to-query-cinder-resources-filter-by-time-comparison-operators

Support users can query resources by specifying the time comparison operators along with created_at or updated_at, and cinder will return all which matches the time condition.

Problem description

Cinder (also other projects, like heat and neutron) API only support filtering resources by discrete values, even though cinder resources have timestamp fields, users can only query resources operated at given time, not during given period. Users may be interested in resources operated in a specific period for monitoring or statistics purpose but currently they have to retrieve and filter the resources by themselves. This change can bring facility to users and also improve the efficiency of timestamp based query.

Use Cases

In large scale environment, lots of resources were created in system, for tracing the change of resource, user or manage system only need to get those resources which was creatded or changed from some time point, instead of querying all resources every time to see which was changed.

Proposed change

Introduce six time comparison operators along with created_at or updated_at fields for retrieving resources more flexible. User needs to specify the operator first, a colon (:) as a separator, and then the time. One step closer, we will also support multi-operators querying at once, a comma (,) as the separator.
Operator ‘gt’: Return results more recent than the specified time.
Operator ‘gte’: Return any results matching the specified time and also any more recent results.
Operator ‘eq’: Return any results matching the specified time exactly.
Operator ‘neq’: Return any results that do not match the specified time.
Operator ‘lt’: Return results older than the specified time.
Operator ‘lte’: Return any results matching the specified time and also any older results.

Alternatives

As discussed in ‘Problem description’ section, user can retrieve and then filter resources by themselves, but this approach is neither convenient nor efficient. Leaving filtering work to database can utilize the optimization of database engine and also reduce data transmitted from server to client.

Data model impact

None

REST API impact

List API will accept new query string parameters. User can pass the operator and time to the list API url to retrieve resources created or updated since or prior to a specific time. This changes also need to bump the microversion of API to keep forward compatibility.

GET /v3/{project_id}/volumes/{detail}?updated_at=gt:2016-01-01T01:00:00,lt:2016-12-01T01:00:00

Security impact

None

Notifications impact

None.

Other end user impact

Python client may add help to inform users this new filter. Python client support dynamic assigning search fields so it is easy for it to support this new filter.

Performance Impact

As discussed in ‘Alternatives’ section, performance can be improved for timestamp based query by utilizing database engine. Additional, it also can add index to improve querying performance.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: wanghao<sxmatch1986@gmail.com>

Work Items

Add API filter
Add querying support in sql
Add related test

Dependencies

None

Testing

Unit test to test if those filters can be correctly applied.
Tempest test if change filter work correctly from API perspective.

Documentation Impact

The cinder API documentation will need to be updated to reflect the REST API changes.

References

[1]: https://developer.openstack.org/api-ref/image/v2/?expanded=list-images-detail#v2-comparison-ops

Support Glance multiple stores

Mon, 13 Jan 2020 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-glance-multiple-backend

This blueprint proposed to support Glance multiple stores.

Problem description

In Rocky and Stein, Glance has added the ability to configure multiple stores as an EXPERIMENTAL feature [1]. This feature is on schedule to be fully supported in the Train cycle. This way an operator can configure more than one of similar or different kind of stores and use one as a default store. If a store is not specified at the time of uploading an image then the image will be stored in default store.

In case of Cinder upload-volume-to-image, if no changes are made to Cinder, even if multiple stores are configured, the volume image will always be uploaded to the default store. This does not cause any issues, but does not allow an operator to take advantage of Glance multiple stores.

Use Cases

An operator is offering multiple Glance image stores distinguished by some desirability metric (as explained to end users in each store’s description) and wants a user to have the option to select which of these stores will be used to store an image created from a Cinder volume.

Proposed change

Define a field named ‘image_service:store_id’ in the volume-type extra-specs. Using ‘image_service’ as the namespace for the ‘store_id’ key signals to the scheduler that this field should be ignored when scheduling a volume build.

The value is the store ‘id’ as indicated in the Image Service API v2 stores discovery response:

GET $IMAGE_API_URL/v2/info/stores

{
    "stores": [
        {
            "id":"reliable",
            "description": "Has a good transfer speed/price ratio"
        },
        {
            "id":"fast",
            "description": "Fast store with short transfer time",
            "default": true
        },
        {
            "id":"cheap",
            "description": "Inexpensive store for seldom used images"
        }
    ]
}

There was discussion at the Train Forum and PTG concerning the validation of extra_specs [2], so the question arises of how that would impact the proposed usage of extra_specs for the Glance store information. If the validation implementation follows the Kilo spec [3], validation will happen in the REST API. The validation code could filter out the ‘image_service:store_id’ field before validating the other fields with the appropriate Cinder backend driver. Additionally, the code could validate the ‘image_service:store_id’ setting by making the Image Service API stores-info call and verfiying that value occurs as an element of the list given by the JSONPath $.stores[*].id in the response. (This would be ["reliable", "fast", "cheap"] in the example above.)

An objection to this proposal is that the Glance store should be associated directly with the image-type, and not hidden in with the extra-specs. Consider, however, how this would look to end users. If the new field is made part of the volume_type, it would be handled like the ‘qos_specs_id’, and the type-show response would look something like this:

{
    "volume_type": {
        "description": "A new type",
        "extra_specs": {},
        "id": "9f319ace-a53c-40cb-8171-f12c4547baaf",
        "is_public": true,
        "name": "a_new_type",
        "os-volume-type-access:is_public": true,
        "qos_specs_id": null,
        "image_service:store_id": null
    }
}

A null value here means that the default Glance image store will be used, but this is confusing to end users because a natural interpretation of the null value is that this volume_type does not allow images to be uploaded to Glance. If we use the extra_specs, however, a volume type using the default Glance image store would look like:

{
    "volume_type": {
        "description": "Volume type using Glance default store",
        "extra_specs": {
            "volume_backend_name": "lvmdriver-1"
        },
        "id": "686cc9f0-f4ca-4a5a-b113-eeb57d3977fd",
        "is_public": true,
        "name": "lvmdriver-1",
        "os-volume-type-access:is_public": true,
        "qos_specs_id": null
    }
}

Which is exactly the same as we have currently, while a volume type that wants to specify a store would look like this:

{
    "volume_type": {
        "description": "Volume type specifying a Glance store",
        "extra_specs": {
            "volume_backend_name": "lvmdriver-1",
            "image_service:store_id": "cheap"
        },
        "id": "686cc9f0-f4ca-4a5a-b113-eeb57d3977fd",
        "is_public": true,
        "name": "lvmdriver-1",
        "os-volume-type-access:is_public": true,
        "qos_specs_id": null
    }
}

Which, as you’d expect, only brings the Glance store ID to your attention if something different than the usual case is being specified.

Additionally, this proposal would require no changes to the Cinder data model or the REST API, though these considerations are secondary to the usability implications discussed above.

Vendor-specific changes

None

Alternatives

Make the ‘image_service:store_id’ a property of the volume_type instead of an extra_specs field. While this would respect the intention of extra_specs describing properties of the Cinder backend, it has the usability issues mentioned in the Proposed change section and would require a new API microversion and a database change.
Add a new Cinder configuration option ‘store’ under the ‘glance’ section to upload all the volume images to a specific glance store. If this option is not present then all the volume images will be uploaded to default glance store. This solution will be efficient if operator doesn’t want to expose the use of uploading volume image to specific store to end user. This will be a very simple change and doesn’t require any Block Storage API change or change in python-cinderclient.

This alternative has the downside that all images of Cinder volumes must be stored in the same store. If an operator has taken the trouble to configure multiple stores in Glance, presumably there is end-user demand to use different stores for different purposes, and presumably such an operator will want block storage users to be able to take advantage of the Glance multiple stores feature.
Add a new microversion to the volume-upload-to-image API to support uploading a volume image to a specific glance store of an end user’s choice. This could be done by adding a new input request parameter ‘store’ to the volume-upload-to-image API. If the ‘store’ option is not specified in a request then the image will be uploaded to the default store. This has the advantage that an end user will be able to choose any of the available Glance stores.

This alternative has the disadvantage that the end user will have to make the GET /v2/info/stores call to the Image API to discover the store identifiers in use in a particular cloud. It also has the disadvantage that it works differently from Cinder’s support of multiple back ends, where the backend a volume is stored in is determined by the volume type. It would be better to use a workflow more familiar to Cinder users.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None. User experience will be consistent with current practice, where an end user selects a volume type based on specific features exposed by the cloud operator.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishek-kekane
Other contributors:: brian-rosmaita

Work Items

Add ‘image_service:store_id’ handling to the Cinder create-image-from-volume workflow
Add related tests

Dependencies

None

Testing

Add related unittest
Add related functional test
Add tempest tests

Documentation Impact

Operators documentation should be updated according to spec implementation.

References

Datera Driver

Fri, 27 Dec 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/datera-driver

Datera storage driver in Cinder.

Problem description

Integration for Datera storage is not available in OpenStack.

Use Cases

Proposed change

Add a Cinder driver that can allow OpenStack services communicate with Datera storage for both vHost and ISCSI.

Alternatives

n/a

Data model impact

n/a

REST API impact

n/a

Security impact

n/a

Notifications impact

n/a

Other end user impact

n/a

Performance Impact

n/a

Other deployer impact

The deployer needs to set the cinder.conf to the right volume_driver.

volume_driver=cinder.volume.drivers.datera.DateraDriver

ISCSI would require setting up san_ip, san_login and san_password appropriately.

vHost would have dependencies on using Linux-IO with the vHost fabric module. The target_helper in the cinder.conf needs to be set to lio_vhost.

Developer impact

n/a

Implementation

Assignee(s)

Primary assignee:: thingee

Work Items

Write driver with ISCSI support.
Write unit tests for ISCSI support.
Provide cert tests with ISCSI support.
Write driver with vhost support.
Write unit tests for vhost support.
Provide cert tests with vhost support.
Provide CI with ISCSI and/or vhost support.

Dependencies

Need vHost connector [1].

Testing

Unit tests
CI testing

Documentation Impact

n/a

References

[1] - https://review.opendev.org/#/c/103048/

Support clone_image from Glance cinder backend

Fri, 27 Dec 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/clone-image-in-glance-cinder-backend

When Glance cinder store is enabled, the glance image can store locations of cinder volume. When a raw image is cinder volume backed, we can efficiently create a new volume from the image by cloning the backing volume, instead of downloading the image from Glance. We can also make upload-to-image from a volume efficient by creating a cloned volume and add its location to an image.

Note that downloading the cinder-backed image from Glance is not currently supported, but this feature is proposed. [1]

Problem description

Creating a new volume from an image, and uploading a volume to an image take long time due to image transfer.
Sharing volume data among tenant is not currently supported in Cinder. To share the volume data, we need to upload it to Glance once.

Use Cases

Share volume templates (e.g. operating system volumes) among tenants via cinder volume backed images.
Each projects can create new instances via the image rapidly. If storage array supports thin-provisioning, the disk space could also be efficient.

Proposed change

When the allowed_direct_url_schemes in cinder.conf contain ‘cinder’, on creating a volume from an image:

Check if the image metadata satisfy the following conditions:
- The image format is raw.
- The image has locations in the format of ‘cinder://<volume-id>’
- Any of the specified volume is on the same host as a new volume.
- The image owner and the volume owner is the same (to avoid accessing volumes not shared by the image owner).
If all the conditions are met, clone the specified volume. When the requested size is larger than the image, extend the new volume.
Otherwise, download the image to a new volume.

On uploading a volume to an image:

If image format is raw and image_upload_use_cinder_backend is enabled in cinder.conf, clone the volume and register its location to the image [2].
- If image_upload_use_internal_tenant is set to True, the cloned volume is placed in the internal tenant.
Otherwise, upload the volume data to Glance.

Alternatives

Generic image cache has the same effect in some situations. However, the proposed feature and generic image cache can co-exist to cover the different situations.

Image cache does not speed up uploading, but this feature make it efficient.

image cache can handle non-raw format images (it stores the converted image in image volumes), but this proposed feature doesn’t handle it so far.

Data model impact

None

REST API impact

None

Security impact

When the cloned volume is stored in the internal tenant, it could potentially be a risk if someone was able to get a hold of its credentials or access the image-volumes. Care will have to be taken to ensure it isn’t accessible by normal users.

Notifications impact

None

Other end user impact

None

Performance Impact

This improves the performance of uploading and download of the image, especially storage array can provide efficient volume cloning.

Other deployer impact

Some configuration options need to be set to enable this feature.

glance_api_version = 2
allowed_direct_url_schemes must contain ‘cinder’
image_upload_use_cinder_backend = True (new)
image_upload_use_internal_tenant = True / False (new)

In Glance, cinder backend must be enabled and show_multiple_locations must be set to True.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: tsekiyama
Other contributors:: None

Work Items

Efficient download from cinder volume backed image
Efficient upload to image

Dependencies

None

To enabling the other components to download the cinder volume backed images, Glance should have the changes [1].

Testing

Unit tests for download from cinder backed image
Unit tests for upload to image

Documentation Impact

Documentation about how to enable this feature should be added.

References

glance-specs: https://review.opendev.org/#/c/183363
glance patch(adding rootwrap): https://review.opendev.org/
glance_store patch: https://review.opendev.org/#/c/166414

https://review.opendev.org/#/c/201754/

Copy volume to image in multiple stores of glance

Fri, 22 Nov 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/copy-image-in-multiple-stores

This specs propose to send base image reference to glance on volume upload to image API.

Problem description

In Train, Glance has added the ability to configure multiple stores [1]. This way an operator can configure more than one of similar or different kind of stores and use one as a default store. If a store is not specified at the time of uploading an image then the image will be stored in default store.

In Train, I have proposed a spec [2] where cinder can specify the glance store via the volume-type. The store specified in volume-type will be passed to glance using x-image-meta-store header. Even operator specifies the glance store identifier in volume-type it will only allow volume image to be uploaded to single specified store and does not allow for co-location of the image. Also, it depends on the operator configuring the volume-type extra-specs to define a preferred Glance store; if the operator doesn’t take this step, all uploads go to the default Glance store.

Use Cases

At the moment, if cloud provider decided to upgrade their cloud to Train release to use the ability of glance of configuring multiple store and the base image from which volume is created resides in multiple stores of glance, then if the operator has not defined a glance store in the volume-type extra specs, the image created from volume will always be uploaded to default store and there is no way to replicate/copy these image bits to all the stores where base image is uploaded. As a result, operators today need to perform a number of manual steps in order to replicate/copy image bits on glance stores despite using the ‘enabled_backends’ configuration option. For this purpose, he/she need to manually copy the newly created image from volume in all stores and register these others locations URL with the image using the glance API.

Proposed change

When a volume is created from an image, Cinder currently stores the image_uuid in the volume_image_metadata. When Cinder upload volume to image is requested cinder should pass the ‘image_uuid’ as a header ‘x-openstack-base-image-ref’ to glance, so that glance can identify the store(s) in which the original image is located. Cinder is also storing this glance store information in volume types [2], where the store identifier associated with volume-type will be passed as a header ‘x-image-meta-store’ to glance to upload the image created from volume to that specific store.

If a volume is created from an image and Cinder passes both the ‘x-image-meta-store’ and ‘x-openstack-base-image-ref’ headers, Glance will ignore the ‘x-openstack-base-image-ref’ header and will use the store_id specified in the ‘x-image-meta-store’ header.

If volume is not bootable and store information is available in volume type then glance will use store associated with volume type to upload the image created from volume.

If volume is not bootable and store information is not available in volume type then glance will use ‘default store’ to upload the image created from volume.

Alternatives

Continue with passing store information using volume types [2].

Data model impact

None

REST API impact

None

Security impact

None

Active/Active HA impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: whoami-rajat
Other contributors:: abhishek-kekane

Work Items

Pass image uuid stored in ‘volume_image_metadata’ as header to glance
Add related tests

Dependencies

Depends on implementation of ‘Support Glance multiple stores’ [2].

Testing

Add related unittest
Add related functional test
Add tempest tests

Documentation Impact

Operators documentation should be updated according to spec implementation.

References

Dynamic Reconfiguration after Modfications of cinder.conf

Wed, 28 Aug 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/dynamic-reconfiguration/

Problem description

As it stands, whenever changes are made to the Cinder configuration file all cinder services need to be restarted in order to pick up the new configuration setup. In a real world environment, a user does not want to be dealing with services shutting down and spinning back up when some aspect of configuration gets changed. In order to stay consistent with this, constantly running services that can be dynamically updated as changes are made provides a much more realistic view of how users interact with services.

Use Cases

Allows developers testing a new configuration option to change what the option is set to and test without having to restart all the Cinder services.
Allows developers to test new functionality implemented in a driver by enabling and disabling configuration options in cinder.conf without restarting Cinder services each time the option value is changed from ‘disabled’ to ‘enabled’.
Allows admins to manage ip addresses for various backends in cinder.conf and have the connections dynamically update.

Proposed change

SIGHUP Approach: The administrator will send the SIGHUP signal to be caught by the Cinder processes. The processes would then be instrumented to catch the signal, drain in progress actions and reload the configuration file. This will also take care of re-initializing the drivers. It still requires the administrator to take an additional manual step to send the SIGHUP after changing the configuration file. This approach requires deployers to update their sysinit scripts to support reloading the processes. Overall, this is a cleaner approach because everything is shut down, the memory is cleaned up, and then the sysinit scripts will take care of reloading the processes.

Note: This is all done on a per host basis; in a multinode setup, the signal will need to be sent to each node. Currently in Active/Active environments this is a sizable issue that needs to be investigated.

Alternatives

Manual Approach: Continue to manually restart Cinder services when changes are made to settings in the cinder.conf file. This is the current approach and is what we are trying to get around.
Reload & Restart Approach: First each process would need to finish the actions that were ongoing. Next the db and RPC connections would need to be dropped and then all caches and all driver caches would need to be flushed before reloading. This approach adds a lot of complexity and lots of possibilities for failure since each cache has to be flushed- things could get missed or not flushed properly.
File Watcher: Code will be added to the cinder processes to watch for changes to the cinder.conf file. Once the processes see that changes have been made, the process will drain and take the necessary actions to reconfigure itself and then auto-restart. This capability could also be controlled by a configuration option so that if the user didn’t want dynamic reconfiguration enabled, they could just disable it in the cinder.conf file. This approach is dangerous because it wouldn’t account for configuration options that are saved into variables. To fix these cases, there would be a sizeable impact for developers finding and replacing all instances of configuration variables and in doing so a number of assumptions of deployment, configuration, tools, and packaging would be broken.

Data model impact

None

REST API impact

None

Security impact

In order to edit the cinder.conf file, the user must have root authority. So, there is not an additional security impact.

Notifications impact

There would need to be notifications of when the reload starts and ends.

Other end user impact

None

Performance Impact

None

Other deployer impact

Update sysinit scripts to support the new ‘cinder-<service> reload’ command.

Developer impact

Test functionality with vendor’s drivers to ensure correct behavior.

Implementation

Assignee(s)

Primary assignee:: kendall-nelson
Secondary assignee:: jacob-gregor

Work Items

Implement handling of SIGHUP signal - Ensure caches are cleared - Dependent vars are updated - Connections are cleanly dropped
Write Unit tests
Testing on a variety of drivers
Update devref to describe SIGHUP/reload process

Dependencies

Testing

Documentation Impact

We will need to update documentation to describe the new capability.

References

Weekly Meeting Logs[1]

[1]http://eavesdrop.openstack.org/meetings/cinder/2016/cinder.2016-03-16-16.01.log.txt

Generic backup implementation

Wed, 28 Aug 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/generic-backup-implementation

Generic backup implementation will allow us to use any supported volumes backend as backup storage. So we won’t need to implement specific backup driver for supported backends.

Problem description

We’ve got different backup and volume drivers now. That’s why we have to re-implement the part of volume driver as backup driver if we want to use the same storage for backups.

Use Cases

Be able to use any cinder-supported storage backend for backups.

Proposed change

Implmement generic backup implementation in a base volume driver like we did for volume migrations. After this vendors could easily implement storage-assistant backups for their drivers.

We will have base backup drivers class for storages like Swift, Google Cloud Storage, etc which will implement only backup-related features.

It will allow a volume backed up to storage A to be restored to a volume on any storage B.

Cinder should not allow to use the same storage both for volume and backups. In such case, backup driver initialization should fail. If storage supports different pools Cinder will allow to create backup on the same storage but in the different pool.

We don’t need to have ‘backup and volumes in the same storage’ feature in Cinder even configurable because we can use snapshots or clone volume for it. Backups should be in different storage or at least in another pool.

As a generic implementation, cinder will use the same mechanism as generic volume migraion:

create volume on a destination storage
attach both source and destination volumes to a cinder node
use ‘dd’ tool to copy volume data
detach both volumes from a cinder node

In case if volume is ‘in-use’ we’ll create temporary snapshot and do backup from it.

We can’t use ‘clone volume’ feature between different storages.

Vendor-specific changes

Vendors or drivers maintainers could implement vendor-specific backup implementation to use storage API for faster backup process.

Alternatives

Follow the current approach with separate volumes and backups drivers.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Backups are likely to be faster to block backends than to swift. It means backup to block storage could be faster than a backup to object storage.

Other deployer impact

Operator should be able to configure backups storage and cinder backup driver.

Developer impact

Volume drivers could implement vendor-specific backup implementation

Implementation

Assignee(s)

Primary assignee:: Ivan Kolodyazhny <e0ne@e0ne.info>
Other contributors:: Volume Driver maintainers

Work Items

TDB

Dependencies

None

Testing

Unit tests
Tempest tests should be implemented in a new feature group

Documentation Impact

Operators documentation should be updated according to spec implementation.

References

http://eavesdrop.openstack.org/meetings/cinder/2016/cinder.2016-08-10-15.59.html

Volume Rekey

Wed, 12 Jun 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/volume-rekey

Cinder supports volume encryption with keys stored in Barbican.

This spec tracks some improvements we can make in how we handle encryption keys.

Problem description

When cloning volumes, cinder clones the encryption key as well, so multiple volumes are unlockable with the same encryption key.

We can make this encryption scheme more robust by changing the encryption key upon volume clone, so that cloned volumes are not unlockable with the key of the source volume.

This is possible since we encrypt volumes using LUKS. Changing the key does not require re-encrypting the volume.

Use Cases

Security hardening for volume encryption.

Proposed change

When a volume is cloned, attach it as part of the clone process, and change the encryption key using LUKS tools.

The rest of the clone process continues as normal afterward.

My current implementation does this by calling a new rekey_volume() method in the create_volume flow, which uses “cryptsetup luksChangeKey”. This should work for any iSCSI/FC drivers, which already must perform a similar attachment when creating a volume from an image.

Some work (planned for after Train) is still needed to make this work for RBD, because there does not seem to be a qemu-img tool that can change encryption keys, and cryptsetup requires a local block device. This leaves us with two options for RBD: a) use krbd mapping to get a block device b) use rbd-nbd to get a block device

NBD is not widely supported in relevant OSes, so krbd looks like the choice there.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

Volume encryption is better hardened against threats due to compromise of a single encryption key.

Active/Active HA impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Slightly more time to clone encrypted volumes.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: eharney

Work Items

Implement this for iSCSI/FC drivers
Test with the LVM driver

In a later release… * Implement this for RBD - Requires some additional effort * Consider additional cases where this concept would be useful - Volume transfer - Backup restoration (?)

Dependencies

None

Testing

Will be on by default and therefore tested by tempest tests that clone encrypted volumes.

Documentation Impact

None

References

None

Generalized filtered listing on resources

Thu, 23 May 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/generalized-filtering-for-cinder-list-resource

Problem description

We continue adding various filter options to the Cinder List API’s. Things like “filter results by x=y”. This results in a good deal of code being added for one-off preferences or user convenience. It also means that we continue to bump micro-versions for such additions and create more versions of the API than we probably care to document, test and maintain.

What’s annoying about this is that it’s not even that we’re adding things to the server side, we’re instead just popping things off of the params in the request body based on what micro-version is specified.

The biggest area of churn for this sort of thing is in the client side of things.

Use Cases

Filters are convenient for operators doing things with large clouds. It’s certainly a nice-to-have feature to be able to do things like just ask Cinder for a list of all volumes with an error status, or that are bootable or the latest, suggestion all those that are attached to host xyz.

Proposed change

In order to add these today, we add a specific arg to the clients, and then we add specific arg parsing on the Cinder API server to interpret and parse these args. These parsed arguments are then just used to build a filter that we pass in to the sql call to get_volumes (or get_<resource>).

Rather than continuing to add these for the various resources in an ad-hoc manner, this spec proposes a generalized filtered-list mechanism. Rather than add a specific filter by name over and over, instead provide a single filter argument that supplies Key Value pairs to be used to build the db query filter. There’s likely some concern that people won’t know what valid filter keys are, for that we can also provide a list-filters command that could be used in an introspective sort of way, querying the object model for the valid filter items and returning it to the caller.

The list-filters component is derived from an Admin provided list of valid and enabled filters for end users to access. This obviously serves two purposes:

Advertise what filters are enabled/available to the end user

Gives the Cloud Operator the ability to pick what filters are enabled

It’s important to note that not all clouds will want to expose filters like backend-host and other internals that shouldn’t typically be leaked to the end user.

What’s interesting about this is from the API Servers perspective we don’t even really care about any of this. We can already pass in filters via the params in the request body and this just works. If you’re using curl this just works already as is. If you are using the client you’d just need a way to expose it properly: (https://gist.github.com/j-griffith/f455e51d3e597af96bfe7022974b5bd9)

Alternatives

We could certainly continue doing what we’re doing now and just adding specific filter arguments to the clients and the API, but I don’t think that’s sustainable.
We could provide a generalized “filters” (pick a better name) controller that could be called directly and then call the specified resource item with filters. This would mean you would do things like: cinder list-filters –resource volume to get a list of valid filter items for a volume (volumes should be the default).

cinder list-filters –resource snapshot for snapshots, etc.

But then have something like a cinder [<resource>-]list –filters xyz=abc which would go to a resource list controller and then call the resource controller appropriately.

This might be a good way to reduce some duplicate code and churn even further but it may require just as much work and code as just having each resource have it’s own “–filters” options to the list commands.

Data model impact

None

REST API impact

Other than adding the ability to view the valid filter keys we don’t even need to change anything on the Cinder server side currently.

Security impact

None that I’m aware of

Notifications impact

Other end user impact

Life would be better

Performance Impact

None

Other deployer impact

None

Developer impact

Life would be better

Implementation

Assignee(s)

Primary assignee:: john-griffith

Work Items

Add the filters arg to the client
Add the ability to query the API server for valid filters for a resource

Dependencies

Testing

Documentation Impact

We will need to update documentation to describe the new capability, although I expect anybody not using the client may already be exposing this on their own.

References

Leverage compression accelerator

Sun, 14 Apr 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/leverage-compression-accelerator

This blueprint proposes to leverage hardware compression accelerator to accelerate the image compression when uploading volume to glance as image.

Problem description

When trying to upload volume to glance as image, currently all the format transformation is done by software and the performance is not good.

For example, when uploading a volume of size 30GB with 12GB data resides, select qcow2 as the target image format, currently the tool used to do the transformation is qemu-img. This has below problems:

Low compression rate After transforming, the output size is 12GB.
High CPU consuming During the transforming, CPU usage is about 100%-170% (~ two cores).
Long duration The time to transform the volume is about 23 seconds.

As a contrast, if using hardware accelerator to do the format transformation, choose ‘gzip’ as the target format, the performance is much better. Take Intel QAT as the accelerator for example, performance data would be:

Compression rate After transforming, the output size is 2.3GB.
CPU consuming During the transforming, CPU usage is about 97% (one core).
Duration The time to transform the volume is about 9 seconds.

So the compressed size is only a quarter of its original size, but takes less than half the time.

Use Cases

User wants to upload volume to glance as a compressed image

Proposed change

Add a new container format ‘compressed’

Cinder don’t have an enum to control container format list, rather than disk format. But cinder will have specific steps for the ‘compressed’ container format. Refer to below changes.

The disk format remains unchanged by this spec, it can be any type in enum in cinder.api.schemas.volume_actions.volume_upload_image.
Add an option to enable/disable the image compression support

If there’s no hardware accelerator exists, then it will fallback to software solution to do the compression. But this will consume CPU resource. So this spec will add an option to control if the compression feature be enabled or not.
Add compression tool of famous hardware accelerators into file volume.filters

The compression tool such as ‘qzip’ will be added in volume.filters so that it can be run by root wrapper
Show ‘compressed’ in the container format list when selecting a volume to upload

This need to be done in cinder client and horizon.
Check if there are some hardware accelerators exist in the system

Go through the accelerator tool list to find if any hardware accelerator exist. If none exist, failover to software solution if compressed format is selected.

This will be done in ./cinder/image/accelerator.py, and at last, will call the driver of accelerator to check the existence.

Refer to: https://review.opendev.org/#/c/668825/1/cinder/image/accelerator.py Function: is_engine_ready https://review.opendev.org/#/c/668825/1/cinder/image/accelerators/qat.py Function: is_accel_exist
Do image compression after convert_image but before uploading to glance

Call the existing accelerator’s tool to do the compression. e.g. if Intel QAT device is identified, then call qzip here to do the compression. If no accelerator identified, then fallback to software solution, using the tool gzip to do the decomression.
Do image decompression after downloading from glance but before convert_image

When creating volume from image, call hardware accelerator to do the decompression. If hw accelerator doesn’t exist, then fallback to software solution, using the tool gzip to do the decomression

Alternatives

Another way is to use the improved processor with software solution only, such as gzip. But the cost would be high vs dedicated hardware accelerator, and have no performance benefit.

REST API impact

Add the image type of “compressed” when uploading volume to image

Data model impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Less CPU comsuming if hardware accelerator available
Less conversion time if hardware accelerator available
Smaller image size

Other deployer impact

Need to install the driver and utility tool of the accelerator
Can configurate the option to enable/disable compression capability

Developer impact

Developers that want to add support for other accelerators need to add their tool to root wrap

Implementation

Assignee(s)

Primary assignee:: Liang Fang <liang.a.fang@intel.com>

Work Items

During image conversion, switch to hardware accelerator if possible
Implement a common framework for hw leverage in image conversion
Implement a typical hw accelerator in image conversion
Unit test be added

Dependencies

None

Testing

Unit-tests, tempest and other related tests will be implemented.
Test case in particular: when creating a volume from an image with container_format == ‘compressed’, but the image contains some format other than what Cinder supports. The expected behavior is to fail gracefully.

Documentation Impact

Documentation will be needed. User documentation on how to use accelerator and developer documentation on how to add an additional accelerator

References

[1] https://review.opendev.org/#/c/668825/ [2] https://review.opendev.org/#/c/668943/

Untyped volumes to default volume type

Wed, 10 Apr 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/untyped-volumes-default-volume-type

This blueprint proposes to use a default volume type instead of allowing users to create untyped volumes.

Problem description

Currently a user is able to create a volume without any volume type, since a volume’s characteristics are defined by a volume type, creating untyped volumes shouldn’t be allowed.

Use Cases

Most users use volume types, and our code is simpler and less buggy if we just always attach volume types to volumes, so we should just force all deployments to use volume types.

Proposed change

This spec proposes the following changes :

Create a default volume type during cinder database migration or on cinder services start time. The default volume type will have no extra specs assigned to it and will be named __DEFAULT__
If a volume type named __DEFAULT__ already exists, the deployer needs to rename or remove it before upgrading
Add an online migration to convert all untyped volumes and snapshots to __DEFAULT__ volume type
Set a default value __DEFAULT__ for default_volume_type config option so the default value is picked when it is unset in cinder.conf
Don’t allow deletion of the __DEFAULT__ volume type via type-delete
Updating of volume type (__DEFAULT__) will be handled by MANAGE_POLICY which defaults to admin-only

Alternatives

Mandate specifying volume type while creating volumes.
Do this as a behind-the-scenes DB migration rather than relying on manual intervention with upgrade checkers.
Do this as a best-effort-if-it’s-safe operation in the volume manager, which would migrate most deployments, and skip those that are ruled out for whatever reason.

REST API impact

None

Data model impact

None

Security impact

None

Notifications impact

None

Other end user impact

All volumes created without specifying the volume-type parameter will be associated with the default volume type.
Untyped Volumes and Snapshots will be assigned __DEFAULT__ volume type
Users will no longer be able to create a volume with None volume type.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Rajat Dhasmana <rajatdhasmana@gmail.com>
Other assignee:: Eric Harney <eharney@redhat.com>

Work Items

By restricting untyped volumes, we need to do the following changes:

Add an upgrade check to verify that current deployment doesn’t contain any volume type named __DEFAULT__ else the upgrade will fail
Create __DEFAULT__ volume type at the time of cinder db migration or service start time
Add upgrade check to verify no type named __DEFAULT__ exists before upgrading
Add an online migration to convert all untyped volumes and snapshots to __DEFAULT__ volume type
Related code changes to associate default_volume_type to volumes if no volume type is specified by user

Dependencies

None

Testing

Unit-tests, tempest and other related tests will be implemented.

Documentation Impact

Need to update volume type related docs.

References

None

Add project_id attribute to group and group snapshot response

Thu, 21 Feb 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-project-id-to-group-groupsnapshot-response

This blueprint proposes to add project_id attribute to the response body of list groups with detail, list group snapshots with detail, show group detail and show group snapshot detail APIs.

Problem Description

Currently, the group and group snapshot show response doesn’t include project_id. It is an important parameter while differentiating between multiple groups and multiple group snapshots in horizon.

Use Cases

As horizon is adding tabs for groups and group snapshots listing with detail in the admin panel[1], it requires project_id as a response parameter from the groups and group snapshots API. It is similar to what is implemented in volumes and snapshots list.

Proposed change

This spec proposes to add project_id attribute to the response body of list groups with detail, list group snapshots with detail, show group detail and show group snapshot detail APIs.

Add a new microverion API to add project_id attribute to the response body of list groups with detail, list group snapshots with detail, show group detail and show group snapshot detail APIs:

List groups with detail GET /v3/{project_id}/groups/detail
List group snapshots with detail GET /v3/{project_id}/group_snapshots/detail
Show group detail GET /v3/{project_id}/groups/{group_id}
Show group snapshot detail GET /v3/{project_id}/group_snapshots/{group_snapshot_id}

Alternatives

None

REST API impact

Add a new microversion in Cinder API.

List groups with detail:

GET /v3/{project_id}/groups/detail
Response BODY:
{
    "groups": [{
        ...
        "project_id": "7ccf4863071f44aeb8f141f65780c51b"
    }]
}

List group snapshots with detail:

GET /v3/{project_id}/group_snapshots/detail
Response BODY:
{
    "group_snapshots": [{
        ...
        "project_id": "7ccf4863071f44aeb8f141f65780c51b"
    }]
}

Show group detail:

GET /v3/{project_id}/groups/{group_id}
Response BODY:
{
    "group": [{
        ...
        "project_id": "7ccf4863071f44aeb8f141f65780c51b"
    }]
}

Show group snapshot detail:

GET /v3/{project_id}/group_snapshots/{group_snapshot_id}
Response BODY:
{
    "group_snapshot": [{
        ...
        "project_id": "7ccf4863071f44aeb8f141f65780c51b"
    }]
}

Calling this method shows project_id for a group or group snapshot. It is intended for admins to use, which is used to display the project_id to which the group or group snapshot belongs, and controlled by GROUP_ATTRIBUTES_POLICY or GROUP_SNAPSHOT_ATTRIBUTES_POLICY.

Data model impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Rajat Dhasmana <rajatdhasmana@gmail.com>

Work Items

Add a new microversion.
Add project_id to the response body of list groups with detail, list group snapshots with detail, show group detail and show group snapshot detail APIs.
Add the related unit tests.
Update related list groups with detail, list group snapshots with detail, show group detail and show group snapshot detail api doc.

Dependencies

None

Testing

Unit-tests and other related test should be implemented

Documentation Impact

None

References

[1] https://blueprints.launchpad.net/horizon/+spec/cinder-generic-volume-groups

Enable multiattach of volumes

Wed, 16 Jan 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/multi-attach-v3-attach

The ability to attach a volume to multiple hosts/servers simultaneously is a use case that has been asked for over the years. The implementation of the Attach workflow in Cinder however made this somewhat difficult and less than robust.

With the new Cinder Attachment API’s [1] this is a bit easier because Attachments are treated as independent objects.

This spec proposes a set of changes to enable and control the use of multiattach volumes from the Cinder side.

Problem description

Currently we refuse to attach a volume that is already attached (in-use). There are situations where a cloud user may have deployed a clustered file-system on a Cinder volume and they would like the ability to attach a volume to multiple hosts/servers at the same time.

Note that Cinder is not providing any sort of Filesystem checks or sharing overlays on the volume, it’s completely up to the user to handle this and determine if they can in fact attach a volume to multiple locations without incurring filesystem/data corruption.

The problem that we are proposing to solve with this spec is simply to enable the ability to attach a volume to multiple locations, and provide the cloud operator with policies to enable/disable the functionality for the cloud. We do NOT offer any mechanism to keep a user from corrupting their data or doing something terrible.

Terminology

Multi-Attach RO:: The ability to attach a volume as one would normally do (Read Write access) and then to attach that volume to another host(s)/server(s) in Read Only mode. Note, this is actually tricky because this typically will require that the consumer (ie Nova/KVM) knows how to set and enforce Read Only on an attached volume.
Multi-Attach RW:: The ability to attach a volume as one would normally do (Read Write access) and then to attach that volume to another host(s)/server(s). In this case there is no differentiation between the initial attachment and any subsequent attachments. They are all treated as independent items and are all Read Write volumes with no real distinction beyond what’s provided with a single attachment.

The ability to set Read Only options on attachments will be handled as seperate work in the from of an argument to attachment-create. That feature is not considered part of the scope of this spec and will be handled independently.

Multi-Attach Capable Backend:: Some backend devices just flat out may not support this. Those that do are considered to be “Multi-Attach Capable” backend devices and need to report this in their capabilities.

Example Volume Types to communicate Multi Path info:

{ ‘multiattach’: ‘<is> True’, }

It will be up to the Cinder scheduler to determine if the policy allows these types of volumes to be created, and it will then be up to the driver to form the corresponding Connection and Volume attributes appropriately in it’s model returns.

Again NOTE we will NOT allow retype of multiattachment setting for an in-use volume.

Use Cases

There are products that can be used like Oracle RAC that would make things like H/A databases via a shared Block device possible which is something of interest.
The other use case that’s been proposed is the ability to have things like passive stand-by servers/volumes.
Analyzing/Fleecing an Instance from another Instance or host

Proposed change

MultiAttach policy

Create a Cinder policy that enables/disables the ability to multi-attach a volume. This policy is a global policy for the endpoint and simply either enables or disables the capability.

Additionally we’ll want a specific policy to enable/disable the ability to multiattach bootable volumes (volume with the bootable parameter set True).

MultiAttach capable volume type

In order to ensure that a volume is created on a backend that supports multiple attachments, it will need to be created on a backend that allows this. The only way to safely control this is be requiring that a type of “multiattach” is created and used on volume creation.

If an additional attach request is made to a volume that is NOT of the type multi-attach enabled the request should fail as the volume is already in-use, and either a new volume should be created of the correct type, or the volume should be retyped. Of course if the policy allows and the type is correct the additional attachment request should be processed and the existing attachment(s) are left active and NOT interrupted.

To avoid various corner cases and confusion for users in what’s allowed and what’s not, this spec proposes we set a hard-code rule that disallows the retyping of attachment parameters of an in-use volume. Regardless of what those changes are; multi–>non-multi, non-multi–>multi.

Any retype command should inspect the multiattach keys and if there is any change being requested in said keys the retype should fail immediately at the API service.

NOTE that the default will be multiattach:False

There is an additional required task, the –allow-multiattach parameter included with volume create needs to be deprecated and it needs to be very explicit that that flag is NOT relevant to the implementation of multi-attach that is being proposed in this spec and implemented in Cinder API microversion xyz.

Special considerations for force-detach; recent changes have been added that treat things like detach of attachments with no connector as force detach operations. This should be fine in a multi-attach scenario because the force detach is ONLY issued against that instantiation of an attachment. In other words the only thing force detach does is ignore state checks and disconnects the specified session, leaving others intact. The process on the Cinder side should follow the standard multi-attach detach process with the exception of ignoring the status of the volume.

FWIW that call rarely works as users expect anyway, so maybe this will be a good time to revisit it. Perhaps deprecate it, and include a force parameter to the attachment-delete API call itself?

Alternatives

Be cloudy and write clustered apps
Use things like independent data instances
Get over it, you probably don’t really need this

Data model impact

N/A All of the needed changes to the data model should already be in place. Inparticular the existing multiattach column on the volume object, which will signal the consumer that they’re working with a multiattach capable volume. So, if a volume of type multiattach:True is requested/created, it’s multiattach column is set to True, else False.

To elaborate on the create flow: Admin creates a volume type multiattach with extra-specs: { ‘multiattach’: ‘<is> True’ }

If a user desires a multiattach volume, he/she issues a create call specifying the multiattach type: cinder create –volume-type multiattach –name my-mavol 20

If there are NO backends reporting multiattach: True then scheduling will fail. Unfortunatly this process includes the task-flow retries and the object will be created, the caller will get a 202 response, but after taskflow and the scheduler finish retries and fail to find a backend the volume status will be set to error. It may be possible to speed this up if we need to and put capability checks into the API layer so that we could respond immediately with an Invalid Request. We do provide an API call now that pulls capabilities from the system, so we could make that check upon receipt of the create request and make sure it’s possible to fulfill the request. It would probably be wise to cache this info based on the periodic capability reporting instead of fetching it each time.

REST API impact

This has a number of impacts on the API, the most obvious of which is the fact that you can attach a volume multiple times. The other changes that may not be so obvious are things like representing a volume’s attachment status, and managing state changes when a secondary attachment is processed or removed.

The current representation of volume-status is likely to be insufficent once multiple attachments are enabled.

Security impact

N/A

Notifications impact

N/A

Other end user impact

User can potentially attach a volume to multiple servers, and corrupt their data.

Performance Impact

N/A

Developer impact

Drivers will need to add a capabilities field “multiattach: True/False”, and do any special handling on their end for connecting/disconnecting volumes in this category.

Implementation

Assignee(s)

Primary assignee:: None
Other contributors:: None

Work Items

Update Cinder attach/detach to accomodate shared targets (most of this is being done independently, see Dependencies section for more info)
Implement policy changes
Implement changes to API to allow/ignore existing attachments on attachment-create calls.
Update the attach/detach volume-status transitions to be multiattach aware and make sure they reflect the correct values.
Update the detail volume view and possibly the summary view to clearly inform the user when a volume is attached to multiple servers.

Dependencies

Handling of disconnects for devices using shared targets Initial work is under review here: https://review.openstack.org/#/c/520676/ Additionally, the API will need a micro version bump and additions to the response views for volumes.
Required patches for service-uuid have already merged https://review.openstack.org/#/c/519025/

Testing

https://review.openstack.org/#/c/266605/

New unit tests will be added to test the changed code and functional testing will need to be added as well.

Documentation Impact

This will require updates to both deployment guides as well as end-user guides.

References

Promote Replication Target (Cheesecake)

Wed, 09 Jan 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/replication-backend-promotion

Problem description

After failing backend A over to backend B, there is not a mechanism in Cinder to promote backend B to the master backend in order to then replicate to a backend C. We also lack some management commands to help rebuild after a disaster if states get out of sync.

Current Workflow

Setup Replication
Failure Occurs
Fail over
Promote Secondary Backend
1. Freeze backend to prevent manage operations
2. Stop cinder volume service
3. Update cinder.conf to have backend A replaced with B and B with C
4. Hack db to set backend to no longer be in ‘failed-over’ state. This is the step this spec is concerned with. Example:
```
update services set disabled=0,
                    disabled_reason=NULL,
                    replication_status='enabled',
                    active_backend_id=NULL
  where id=3;
```
5. Start cinder volume service
6. Unfreeze backend

Use Cases

There was a fire in my data center and my primary backend (A) was destroyed. Luckily, I was replicating that backend to backend (B). After failing over to backend B and repairing the data center, we installed backend C to be a new replication target for B.

There is also a case where, for whatever reason, the replication state gets out of sync with reality. In a situation where the status and active backend id need to be adjusted manually by the cloud admin while recovering from a disaster.

Proposed change

To handle the case where the Cinder DB just needs to be synchronized with the real world we will add the following cinder manage commands to reset the active backend for a host. Similar to reset-state for volumes this will just do DB operations. The assumption being that the cinder.conf has already been updated, and the volume service is stopped, in a disabled state, and probably frozen. The manage command will verify that the service is stopped, disabled, and frozen.

cinder-manage reset-active-backend replication_status=<status> \
    <active_backend_id> <backend-host>

Equivalent to:

update services set replication_status='<status>',
                    active_backend_id='<id>',
   where host='<backend-name>';

Where the defaults for status will be disabled and the active_backend_id will be None. The target for this could also be a cluster in A-A deployments.

Note: It will be up to the Admin to re-enabled the service.

That gives us the ability to avoid having an admin go manually run DB commands, but does NOT allow for doing this in an “online” recovery. The volume service must be offline for this to work safely.

Making this change will require drivers to make adjustments to their replication states upon initialization. As-is we don’t really have much definition around what is allowed to change in cinder.conf and how much the driver should support for changes to replication targets and things like that. There is sort of an implied contract that when __init__ or do_setup is called the driver should make their current replication state match what they are given in the config. This is a little bit problematic today though as there isn’t a mechanism for them to update DB entries like replication_extended_status or replication_driver_data. To solve that gap, and allow for this new officially supported method of modifying replication status when offline, we will introduce a new driver api method

update_host_replication(replication_status, active_backend_id, volumes, groups)

This method will be called immediately following do_setup and will return a model update for the service as well as a list of updates for the volumes and groups if the driver supports replication groups. The group parameter will default to None if not defined.

The drivers will need to be able to take appropriate steps to get the system into the desired state based on the current DB state (which was theoretically modifed before startup by the new cinder-manage command) and the current cinder.conf.

If not implemented things will continue to work as they do today, and require that the admin potentially take more drastic measures to recover after performing a failover. The goal here is to not break any existing functionality, but to add enough infrastructure for drivers to behave better.

When we do this we might require some way of fencing to prevent multiple A-A driver instances from doing the setup at the same time as it will more than likely be problematic for some backends, and could risk data loss if more than one is modifying replication states at the same time. As a simple solution we could use a new replication_status like updating for the service, and only allow them to call setup_replication if the status is not set to that. This status can also be beneficial for an admin to know what is happening if the process of updating via the driver takes noticeable amounts of time.

Doing it this way should also allow for later on making “online” updates where we can utilize that same driver hook to modify replication states. This spec and initial implementation does not aim to cover that scenario.

Alternatives

We could add admin API’s to Cinder. Those API’s could do the DB updates and ping the drivers. The downside is that it requires the API and volume service to be online, which may be problematic in the scenario that you are picking up pieces after a disaster.

Later on we can look into doing “online” promotions where the volume service does not need to be offline. Similar code in the drivers would be required, but the complexity gets increased rapidly by trying to support this.

There was also discussion about using new admin API’s which would modify a db state that tracks replication info. The downside to this is that we will move into a scenario where the running config and state doesn’t match the configured state.

Following on that path of tracking replication state in the db, we could go to the extreme and move all of the replication configuration to be done via API’s. We can then track state, and provide drivers with diffs as the state changes. In the longer term that addresses the runtime vs config state disparity, but it will be a significant change in workflow and deployments. Not to mention would require somewhat major changes to drivers implementing replication.

Data model impact

A new status for the service/cluster will be added.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Volume service startup will probably take a performance hit, depending on the backend and how many replicated volumes need to be modified and updated.

Other deployer impact

None

Developer impact

Driver maintainers will need to potentially implement this new functionality, and be aware of the implications of how/when replication configuration and status can be adjusted.

Implementation

Assignee(s)

Primary assignee:: jbernard

Work Items

Implement cinder-manage reset-active-backend command
Implement volume manager changes to allow for update_host_replication to be used at startup by drivers.
Open a bug against each backend that supports replication and needs an update as a result of this change.

Dependencies

None

Testing

None

Documentation Impact

Documentation in the Admin guide for how to perform a backend promotion, and updating the devref for driver developers to explain the expectations of drivers implementing replication.

References

None

Affinity and anti-affinity scheduler filter

Mon, 07 Jan 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/affinity-antiaffinity-filter

To add scheduler filter to Cinder that allows scheduler to make placement decision based on affinity relationship between existing volumes and new volume (the one being scheduled). The affinity relationship here means the location of volumes (‘host’ of volume).

Problem description

Cinder has done a good job hiding the details of storage backends from end users by using volume types. However there are use cases where users who build their application on top of volumes would like to be able to ‘choose’ where a volume be created on. How can Cinder provide such capability without hurting the simplicity we have been keeping? Affinity/anti-affinity is one of the flexibility we can provide without exposing details to backends.

The term affinity/anti-affinity here is to describe the relationship between two sets of volumes in terms of location. To limit the scope, we describe one volume is affinity with the other one only when they reside in the same volume back-end (this notion can be extended to volume pools if volume pool support lands in Cinder); on the contrary, ‘anti-affinity’ relation between two sets of volumes simply implies they are on different Cinder back-ends (pools).

This affinity/anti-affinity filter filters Cinder backend based on hint specified by end user. The hint expresses the affinity or anti-affinity relation between new volumes and existing volume(s). This allows end users to provide hints like ‘please put this volume to a place that is different from where Volume-XYZ resides in’. Below are two use cases where this new filter can be useful.

DB team builds MySQL master onto one volume, they’d prefer to put new volumes for slave DBs to different backends from where the master DB resides in, for the sake of high availability.
Log processing project would like to have fast storage as possible, so they create soft RAID across multiple volumes. They want to put these volumes as close to each other as possible, ideally on the same storage backend, for the sake of performance.

Use Cases

Proposed change

Add two new filters to Cinder - AffinityFilter and AntiAffinityFilter. These two filters will look at the scheduler hint provided by end users (via scheduler hint extension) and filter backends by checking the ‘host’ of old and new volumes see if a backend meets the requirement (being on the same backend as existing volume or not being on the same backend(s) as existing volume(s)).

Alternatives

There had been one proposal to allow admin user to directly specify the backend for new volumes. It doesn’t really provide similar functionality as affinity filter ‘cos it was admin only and it itself has a few drawbacks (security concern, for example).

Data model impact

None

REST API impact

None

Security impact

Although this change involves using or parsing user-provided - scheduler hints, which is already part of Cinder. This doesn’t put Cinder in any more danger as it is now.

Notifications impact

None

Other end user impact

None

Performance Impact

New filters would query DB once per request, it only adds slightly latency to the system and the latency has nothing to do with the size of the system.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: zhiteng-huang (winston-d on IRC)

Work Items

Filter implementation
Internal DB API modifition

Dependencies

None

Testing

Test against AffinityFilter:

Create one volume A;
Create another volume B with uuid of A and ‘same_backend’ as hint;
Checks if B is created on same backend as A;

Test against AntiAffinityFilter:

Create one volume A;
Create another volume C with uuid of A and ‘different_backend’ as hint;
Checks if C is created on different backend as A;

Documentation Impact

Need to document the usage of new filters.

References

Nova has been offering simliar feature called SameHostFilter and DifferentHostFilter since Diablo.

https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py

Backup State Reset API

Mon, 07 Jan 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/support-reset-state-for-backup

Provide an API to reset the state of a backup being stucked in creating or restoring.

Problem description

Since there are volume reset-state function and snapshot reset-state function, backup also needs reset-state as well. When creating or restoring backup, it may leave backup stucked in creating or restoring status due to database down or rabbitmq down, etc.. Currently we could only solve these problems by restarting cinder-backup. This bp is to another mean for administrators to solve these problems by calling backup reset state API.

Resetting status from creating/restoring to available

restoring –> available Directly change the backup status to ‘error’, because the backup data is already existed in storage backend.

creating –> available Use backup-create routine as an example to illustrate what benefit we can get from backup-reset function. Backup-create routine first backup volume and metadatas, and then update the status of volume and backup. If database just went down after update the volume’s status to ‘available’, leaving the backup’s status to be ‘creating’ without having methods to deal with through API.

If we have reset-state API and resetting status from creating to available, we first verify whether the backup is ok on storage backend. If so, we change backup status from creating to available. If not, we throw an exception and change backup status from creating to error.

2. Resetting status from creating/restoring to error Directly change the backup status to ‘error’ without restart cinder-backup.

Use Cases

Proposed change

A new API function and corresponding cinder command will be added to reset the status of backups.

The initial proposal is to provide a method for administrator to handle the backup item stucked in status like creating or restoring.

Alternatives

update backups set status='some status' where id='xxx-xxx-xxx-xxx';

Data model impact

None

REST API impact

Add a new REST API to reset backup states:

POST /v2/{tenant_id}/backups/{id}/action

JSON request schema definition:

'backup-reset_status': {
    'status': 'available'
}

Normal http response code:: 202
Expected error http response code:: 400

Security impact

None

Notifications impact

None

Other end user impact

A new command, backup-reset-state, will be added to python-cinderclient. This command mirrors the underlying API function.

Resetting the status of a backup can be performed by: $ cinder backup-reset-state –state <state> <backup>

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ling-yun

Work Items

Implement REST API
Implement cinder client functions
Implement cinder command

Dependencies

None

Testing

None

Documentation Impact

The cinder client documentation will need to be updated to reflect the new command.

The cinder API documentation will need to be updated to reflect the REST API changes.

References

None

New Core APIs for Discovering System Capabilities

Sun, 06 Jan 2019 00:00:00

https://blueprints.launchpad.net/cinder/+spec/discovering-system-capabilities

There needs to be a way for Horizon to discover the capabilities of the remote block storage deployment so that it can enable/disable the UI widgets that exercise those capabilities. Consider the volume backup functionality which may or may not be supported by a cinder deployment. If Horizon could programmatically detect the presence of this capability, it can enable the volume backup actions in the UI. This will obviate the need for the operator to manually set appropriate fields in Horizon’s config to signal the availability of this feature, which is the case today. We propose adding a set of APIs that would return the “capabilities” available to the current user. These can be used by Horizon and other clients to configure themselves without the need for config files that are manually kept in sync.

This blueprint grew out of the design discussion in [1] and [2] for the bug #1334856 [3] filed by jgriffith.

Problem description

Horizon allows users to create volume backups via the UI, if the underlying cinder deployment supports the same. Not all cinder deployments support this functionality, so Horizon needs a way to know when it is supported so that it can show the UI elements for backup.

Today, Horizon uses [5] a config setting “enable_backup” for controlling the enable status of volume backup operations in the UI. This needs to be manually set/unset by the operator based on the status of the cinder-backup service. This is not only cumbersome but also prone to operator error.

The volume backup API is implemented on the server side by the cinder-backup service. This service is absent when the cinder deployment does not support volume backups. This is the case, e.g., for devstack with default config.

The “os-services” API extension shows the state of all backend services. This suggests a way by which Horizon could get to know the existence of the cinder-backup service. However this has two problems. The first is that the os-services API is admin-only whereas the backup operation is available to even non-admin users. The second and more important problem is that not all features have corresponding services or extension APIs that could be used to compute the support for that feature in a similar way.

This problem raises its head in non UI domains also. Consider the OpenStack Heat project. Heat templates allows the option of backing up a volume upon the deletion of the corresponding volume resource in the template. However, as there is no way to know whether backup capability is supported by a Cinder deployment, this can result in late failures during a volume delete. This ultimately prevents interoperability of orchestration templates across clouds.

It is easy to see that the above problem is not limited to just the backup feature but is much more general. Horizon needs a way to programmatically know which capabilities are supported by the remote block storage service so that it can enable only those UI widgets that deal with these capabilities and disable others.

Use Cases

The solution proposed in this spec allows building a dynamic, self-configuring Horizon UI. This unburdens the operator from manually configuring Horizon and constantly keeping it in sync with the capabilities of the Cinder deployment.

This solution can be immediately applied to solve the volume backup enable status problem in Horizon. No longer will the operator need to set the “enable_backup” in Horizon config; Horizon will programmatically get the status using the proposed API.

Other capability which can similarly gain is the replication capability, which is only supported by certain volume types (actually “volume backends”, but normally volume backends are more or less tied to volume types). Note that feature will be listed by a different capability API than the one that lists the backup feature. See details below.

The proposed set of APIs can also be used in custom UIs or user scripts for the same reasons that we envision Horizon using it for.

Proposed change

A new, public Block Storage microversion API that returns the set of all capabilities for a particular user and resource. Note that the service (or the deployment) itself is considered as a resource (the root resource).

The capabilities of a system can be defined at multiple levels. The top-level capabilities such as backup capability are defined at the service level (or the root resource level). So, the question “Does this cinder service instance have backup capability?” makes sense by itself. Lower level capabilities are defined on specific (REST) resources. For example, replication feature is defined on a volume type. So, it doesn’t make sense to ask “Does this cinder service instance have the replication capability?”; instead, we need to ask “Does the volume type xyz have the replication capability?”.

While we can define a capabilities API for every resource, in practice it will be required only a small number of resources. For the Cinder API, it is only required today for the service and volume type resources.

It is not possible to avoid multiple levels of capability APIs. The lower level capability APIs require input (the resource type) which is not available at the top level.

The different levels of capability APIs also align with the natural organization of any UI (e.g. Horizon). On the home page, the UI only shows the widgets corresponding to top level capabilities. It is possible to navigate to deeper levels by selecting widgets on the home page. For example, selecting a particular volume type can lead a page showing widgets that correspond to the capabilities of that particular volume type. By making the calls to appropriate level capabilities API, Horizon can programmatically configure itself at every level.

Note that a cinder instance may implement a particular feature but may not allow a particular user to access the same. So, the capabilities API should not only consider whether the service implements the particular feature but also if the current user is privileged to access the same. The “privilege” information is already available in the policy.json [4] file that maps different operations to the users that can access them. So, any capapbility API implementation must make use of this file.

The detection of whether a particular deployment/resource implements a particular capability varies by the capability itself. For the backup capability, it could be the presence of the cinder-backup service. For replication feature, it could be statically set when the volume type is created or it could be fetched from the driver somehow.

It is not possible to implement the capabilities API based solely on the info in the policy.json [4] file as the policy file does not allow defining rules per resource. For example, we can allow replication operation for all users but we cannot constrain it to a specific volume type (which is deal breaker since not all volume types support replication).

It is easy to see that all capabilities APIs should be public, i.e. accessible by any user.

The key contribution of this BP is identifying and proposing an API pattern, namely “one capability API per resource” (of course if some resource doesn’t need it, we don’t need to define a capability API for it). This idea is simple but powerful and also reusable across all OpenStack projects.

While this BP proposes an API pattern, we will only implement the top level capabilities API. This will return only “volume-backup” for now but can be augmented as new capabilities are added to Cinder at the service level. For now, the detection of the backup capability would be similar to the os-services extension. It will check if the cinder-backup service is enabled or not by checking the services table from cinder database and if so return “volume-backup” in the response.

A brief note on the presence of capabilities like APIs in Cinder today: There is a “volume capabilities” extension API [6], but it is defined at the service level (/v2/{tenant_id}/capabilities/{hostname}) rather than at the volume type level. The “show volume type details” API [8] stuffs the capabilities in a catch-all “extra_specs” property. If this BP is approved, we will need to rationalize these existing APIs with the new capabilities APIs that will be defined. This is not in scope for this sepc.

Alternatives

Make the existing os-services extension public: This will expose the private cloud internals to the tenant. This is a security hole and hence makes this alternative infeasible. Also, there may not be a 1:1 correspondence between capabilities and services.
Split the existing os-services extension API into public and private halves, with the public part exposing limited information.

We could modify the services:index action to take a details=true/false parameter: http://{cinder-endpoint}/v2/{tenant-id}/os-services?details=false

And define different policies for detail=true (admin_api) and detail=false (”” i.e. unrestricted).
- “volume_extension:services:index_with_details”: “rule:admin_api”
- “volume_extension:services:index_without_details”: “”
It is not clear if this can be implemented in a backward compatible way and also whether there is precedence for splitting the policy of a single API call based on parameters. Also, as mentioned above, there may not be a 1:1 correspondence between capabilities and services.
Re-use the existing “list extensions” public API [7]. This was proposed by dulek in [2]. First, there may not be a 1:1 correspondence between capabilities and extensions (although it is true for the volume backup case). Even if it were always true, the operator would need to prune cinder.conf (manually!) so that it lists only those extensions that are actually supported. As explained in [2], there is no easy way to do that. Also, as noted by duncant, this breaks the existing semantics - many deployments have the API extensions enabled (as it comes by default) without the service being actually running. So the check return value would mean different things on different systems.

Data model impact

None. As explained, we use the existing services table for the volume backup capability detection. Future capability additions may use different resources and algorithms.

REST API impact

We give instances of the “capabilities API pattern” for three resources, including the service itself.

GET /v3.x/{tenant id}/capabilities

Returns the set of capabilities of the underlying block storage deployment at the service level.

Normal http response code(s): 200

Response is a list of capabilities. Each capability is a simple noun or hyphenated compound noun. E.g:

{
  "capabilities": [
    {
      "name": "volume-backup",
      "description": "Allows creating backups of volumes."
    },
    {
      "name": "other-capability",
      "description": "Other capability description."
    },
  ]
}

GET /v3.x/{tenant_id}/types/{volume_type_id}/capabilities

Returns the set of capabilities of a particular volume type.

Normal http response code(s): 200

Response is a list of capabilities. E.g:

{
  "capabilities": [
    {
      "name": "replication",
      "description": "Allows replication of volumes."
    },
    {
      "name": "other-capability",
      "description": "Other capability description."
    },
  ]
}

General example:

GET /v3.x/{tenant_id}/<some-resource-collection>/{some-resource-instance}/capabilities

Returns the set of capabilities of some-resource-instance.

Security impact

None. Exposing an abstract set of system capabilities should be safe. These capabilities can be gleaned from the available actions in the UI in any case (e.g. backup UI widget is visible implies volume backup capability exists).

Notifications impact

None

Other end user impact

This change is transparent to the user although the user can use this API in a similar way as Horizon for custom UIs or management scripts.

Performance Impact

These are new APIs and should not affect any existing APIs or code paths.

Other deployer impact

Operator no longer has to manually set “enable_backup” in Horizon config settings file. Back-compat story for this Horizon config change is out of scope for this spec.

Developer impact

The developer will need to be aware of the capabilities API pattern and evaluate if any new optional functionality he/she plans to add to a Cinder service or a lower level resource (e.g. volume type) may benefit from being exposed via this API. The developer may first need to define a capability API for that resource.

Implementation

Assignee(s)

Primary assignee:: dramakri

Work Items

Implement the proposed public microversion capabilities API at the service level
Implement at least the backup capability detection
Add test cases
Update API docs

Dependencies

None

Testing

Unit and functional test cases need to be added to validate this new API.

Documentation Impact

New API and client call in Cinder needs to be documented.
Changes to Horizon config setting needs to be documented.

References

Add user_id attribute to backup response

Wed, 19 Dec 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-user-id-attribute-to-backup-response

This blueprint proposes to add user_id attribute to the response body of list backup with detail and show backup detail APIs.

Problem description

Currently, there is a user_id field in the backups table. These fields are very useful for admin to manage the backup file, but this is not returned in response body. So it is difficult to manage the resources under the project. If there are multiple users under one project, it is impossible to distinguish which user the backup file belongs to.

Use Cases

In large scale environment, lots of backups resources were created in system, that we can only see the project to which the backup file belongs, but we cannot know to which user the backup belongs.

Administrators would like the ability to identify the users that have created backups.

Proposed change

This spec proposes to add user_id attribute to the response body of list backup with detail and show backup detail APIs.

Add a new microverion API to add user_id attribute to the response body of list backup with detail and show backup detail APIs:

List backups with detail GET /v3/{project_id}/backups/detail
Show backup detail GET /v3/{project_id}/backups/{backup_id}

Alternatives

The admin/user could get user_id from the context as a log print, but it’s difficult to find it out easily, especially when the user wants to find a very old backup file.

REST API impact

Add a new microversion in Cinder API.

List backups with detail:

GET /v3/{project_id}/backups/detail
Response BODY:
{
    "backups": [{
        ...
        "user_id": "515ba0dd59f84f25a6a084a45d8d93b2"
    }]
}

Show backup detail:

GET /v3/{project_id}/backups/{backup_id}
Response BODY:
{
    "backups": [{
        ...
        "user_id": "515ba0dd59f84f25a6a084a45d8d93b2"
    }]
}

Calling this method shows a user_id for volume backup. It is intended for admins to use, which is used to display the user to which the backup file belongs, and controlled by BACKUP_ATTRIBUTES_POLICY.

Data model impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Brin Zhang <zhangbailin@inspur.com>

Work Items

Add a new microversion.
Add user_id to the response body of list backup with detail and show backup detail APIs.
Add the related unit tests.
Update related list backup with detail and show detail api doc.

Dependencies

None

Testing

Unit-tests, tempest and other related test should be implemented

Documentation Impact

None

References

None

Driver reinitialization after failure

Thu, 13 Dec 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/driver-initialization-after-fail

This spec proposes support for reintialization of volume drivers after it fails during startup.

Problem description

During Cinder initialization, for many reasons, the storage backend might not be ready and responding. In this case, the driver will not be loaded even if the array becomes available right after.

As there is no retry in Cinder volume service, even later the backend storage is ready, Cinder volume service can’t recover by itself. It needs users to restart the volume service manually.

Use Cases

When a Cinder volume service starts, sometimes its corresponding storage services are not ready. But later the storage services become ready. As a result the volume service can’t work properly and can’t recover by itself. But the administrators probably prefer Cinder to automatically recover from the temporary failures without manual intervention of restarting the service.

Proposed change

The proposal is to

Allow reinitialization of a volume driver when it failed to initialize.
Provide a configuration to set the maximum retry numbers.
The interval of retry will exponentially backoff. Every interval is the exponentiation of retry count. The first interval is 1s, second interval is 2s, third interval is 4s, and so on.
Retry will be handled in init_host.

For this, the following additional config option would be needed:

‘reinit_driver_count’ (default: 3)
Set the maximum times to reintialize the driver if volume initialization fails. Default number is 3.

Alternatives

We also can differentiate whether it should retry with an exception. Like import error, config error, it may not retry. But the benefit is not very impressive, and implementing the differentiation needs work in every driver. As drivers don’t differentiate such errors from backend storage errors.

Data model impact

None

REST API impact

None.

Cinder-client impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

Users don’t need to restart volume service when the initialization of drivers fail on recoverable errors.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Lisa Li (xiaoyan.li@intel.com)

Work Items

Add the option ‘reinit_driver_count’.
Retry to initialize volume drivers when it fails.
Add related unit test cases.

Dependencies

None

Testing

Add unit tests to cover this change.

Documentation Impact

Add administrator documentation to advertise the option of ‘reinit_driver_count’ for driver reinitialization and explain how this should be used.

References

None

Improve volume transfer records

Mon, 10 Dec 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/improve-volume-transfer-records

This blueprint proposes to improve volume transfer records by adding source_project_id and destination_project_id, accepted fields to transfer table and related api responses, makes it easier for users to trace the volume transfer history.

Problem description

Currently, the volume transfer record does not include the destination project_id after transferring and the source project_id before transferring. These fields are very useful for admins and operators to trace the transfer history.

And also once the transfer is deleted, the user can’t determine if this transfer had been accepted or not before it was deleted.

It is bad for admins and users to trace volume historical track between project and audit the volume records.

Use Cases

In order to trace the volume transfer history, the admin wants to know who was the volume owner before transferring.
The admin wants to know whether a deleted transfer had been accepted or not.

Proposed change

This spec proposes to do

Add three new fields to transfer table:
- source_project_id, this field records the source project_id before volume transferring.
- destination_project_id, this field records the destination project_id after volume transferring.
- accepted, this field records if this transfer was accepted or not.
Add a new microverion API to add above fields to the response of follow API:
- Create a volume transfer POST /v3/{project_id}/volume-transfers
- Show volume transfer detail GET /v3/{project_id}/volume-transfers
- List volume transfer and detail GET /v3/{project_id}/volume-transfers/detail
And the response of “List volume transfer (non-detail)” API will not include these fields.

Alternatives

The admins could find part of volume transferring info from log, but it’s difficult to find it out easily, especially when the user wants to audit a very old volume transfer.

REST API impact

A new microversion will be created to add these new added fields to transfer related API responses.

Data model impact

None

Security impact

None

Notifications impact

Notifications will be changed to add these new added fields.

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Yikun Jiang <yikunkero@gmail.com>

Work Items

Add source_project_id and destination_project_id, accepted fields to transfer table
Add source_project_id and destination_project_id, accepted fields to related API.
Implement changes for python-cinderclient to support list transfer with --detail.
Update related transfer api doc.

Dependencies

None

Testing

Unit-tests, tempest and other related test should be implemented

Documentation Impact

None

References

None

Object deletion from DB without driver interaction

Tue, 16 Oct 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-delete-from-db

Problem description

It’s impossible to delete a volume, a backup or a snapshot when the service is not available. There’s no way to bypass driver interaction.

Use Cases

Sometimes, OpenStack admins are pulling the plugs off some storage backends to use a new one. They realize later that they need to cleanup the various objects.

Proposed change

With cinder-manage, we will have a –db-only switch under the volume delete command. The snapshots are deleted in cascade. We will also implement backup subcommand that acts like the volume subcommand.

Alternatives

The only known workaround is to manually update multiple tables and set the status to ‘deleted’ to various objects.

Data model impact

None

REST API impact

None

Security impact

None

Active/Active HA impact

None

Notifications impact

None

Other end user impact

These commands will be available to operators:

cinder-manage volume delete [--db-only] <volume-id>
cinder-manage backup delete [--db-only] <backup-id>

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: David Vallee Delisle <dvd@redhat.com>

Work Items

The following changes will be done under cinder-manage command: * Add a –db-only switch to the volume delete command. * Add a backup delete subcommand with –db-only support. * The snapshots are deleted in cascade when a volume is deleted.

The following changes will be done in the rpcapis: * Add a db_only argument in the delete function

The following changes will be done in the manages: * Add a db_only argument in the delete function

Dependencies

None

Testing

Create tests to validate the delete volume and the delete backup are really deleting.

Documentation Impact

Man page for cinder-manage and any associated documentation will be updated.

References

None

Transfer snapshots with volumes

Tue, 19 Jun 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/transfer-snps-with-vols

This spec aims to extend transfer function that can transfer snapshots with volumes at same time.

Problem description

Currently, Cinder can transfer volumes to another project’s owner, but if this volume has snapshots before transferring, after this operation, new owner can’t delete this volume successfully since it has snapshots.

Use Cases

When transferring volume with snapshots, a user will now have a choice whether or not to transfer those snapshots along with a volume. By default, snapshots are transferred, but selecting this option will allow snapshots not to be transferred to another project. Note those snapshots can be deleted by the remote user if necessary.

Proposed change

In normal deleting, Cinder will disallow the request since the volume has snapshots, but unfortunately, Cinder has cascade deleting operation now, the request will be passed down to driver, some driver will raise exception since snapshot is still existing. So it also should be changed in cascade deletion process.

Add an optional argument “–no-snapshots” in transfer API and CLI. If user didn’t specify it, cinder will transfer snapshots that a volume has by default.
Add a new field in transfer DB model to record this option.
Update snapshot’s information in DB like user id, project id, etc.
Check if the volume still has some snapshots in other project when cascade deleting.

Alternatives

Another option is cinder transfer the snapshots if user specifies an option argument ‘–with-snapshots’.

This option can be minimal with the change, in order to make the client code more simple.

Data model impact

Add a new field “with_snapshots(boolean)” in transfer model.

REST API impact

New microversion in Cinder API.

Add a new V3 API and an optional argument “no_snapshots”:

POST /v3/{project_id}/volume-transfers
RESP BODY: {"transfer": {
                         ...
                         no_snapshots: [True/False],
                        }
           }

Security impact

If users didn’t transfer snapshots with volume, there could be kind of security impact that remote users may be able to act upon the untransferred snapshots. For example, by leveraging COW, that remote user can change the snapshot size in backend.

Notifications impact

Add ‘with_snapshots’ information to transfer notificaiton.

Other end user impact

None

Performance Impact

There could be a db performance issue if a lot of snapshots associated with a given volume since cinder need to change those snapshots’ project id, user id, etc.

Other deployer impact

None

Developer impact

Drivers that implement some form of volume ownership tracking on the backend will need to be fixed to track this change too.

Implementation

Assignee(s)

Primary assignee:: wanghao<sxmatch1986@gmail.com>
Other contributors:: None

Work Items

Implement feature in Cinder tree.
Update cinderclient to support this functionality.
Add change to API doc.

Dependencies

None

Testing

Both unit and Tempest tests are needed to be created to cover the code change.

Documentation Impact

The cinder API documention will need to be updated to reflect the REST API changes.

Devref entry on the volume transfer driver entry point should be created.

References

None

Update backup’s size when backup is created

Thu, 14 Jun 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/report-size-when-backup-created

This blueprint proposes to update backup with the size reported by backup backend when backup is created.

Problem description

Most clouds provide end user a convenient way to protect their data by creating a backup for their volume. Users can create any backup copy at anytime, and the cloud providers charge them by the backup size in total. As the basic volume service, Cinder supports creating, restoring and deleting backups plus the backup quotas, but our backup size isn’t correct at present, the backup’s size always equals to the original volume’s size even if the incremental backup doesn’t have any content in it.

Use Cases

Backup’s size attribute will reflect the actual size on the backend, the quota and charge would be more accurate.

Proposed change

This spec proposes that the backup driver should report the actual size that object holds at the backend when backup is created. So, when user requests to create a backup for a volume with 10G, Cinder will still generate the record with 10G as well as consume 10G for quota reservation, but when the actual size is reported, we will update both the object and the quotas.

Note: The actual size here stands for the final size at the backend, that means if we create backup for a 1G volume which only has 200mb data and after compression only 100mb is used at disk, the actual size is 100mb, not 200mb or 1G.

As Cinder has the unified unit G for every resource, that size should be rounded up to G too, for example, if we create an incremental backup whose actual size is only 12mb at backend, driver should still report 1 G to Cinder service:

def backup(self, backup, volume_file, backup_metadata=False):
    """Start a backup of a specified volume.

    Driver should return the size that the backup object holds
    in the backend, with the unit of G. For instance:

    .. code-block:: python

        {
            'size': 2
        }
    """
    return

Cinder’s backup service would use this size to update backup model as well as commit portion of the reservation (Instead of commit the reservation record, Cinder will update the reservation and then commit the updated record after backup is actually created in the background to cover this case ).

In order to report the actual size that backup holds, drivers need to record or calculate the object size. Take our chunkeddriver for instance, the total size is accumulated by the objects’ size:

size = object1_size + object2_size....

If compression feature is enabled, the object’s size should be:

size = compressed_object1_size + compressed_object2_size...

Now Cinder has the attribute size for backup object which only reflects the volume’s size when created. In order to support updating the actual size of the backup, we will add one more attribute here.

1. volume_size: This is used to record the original volume’s size and can be used to create new volume when restoring, we can not directly link to the original volume size here because we could resize the volume after the volume is backed up.

2. size: This will be used to store the reported value from driver rather the original volume size. And when backup is first created, this value would be equal to the volume_size and will be updated when backup is created at backend.

Alternatives

Keep using volume’s size to generate the backup’s size as well as quota usage record in database.

Data model impact

None

REST API impact

None

Cinder-client impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There would be a slight performance impact when committing quota reservations.

Other deployer impact

None

Developer impact

Developer should report backup’s size in backup method when adding new backup driver.

Implementation

Assignee(s)

Primary assignee:: tommylikehu(tommylikehu@gmail.com)

Work Items

Update cinder to support update the backup size.
Update existing driver to report backup’s actual size.
Add related unit testcases.

Dependencies

None

Testing

Add unit tests to cover this change.

Documentation Impact

Update the base backup driver’s interface.
Update developer’s documentation to advertise this change.

References

None

Cinder Certificate Validation

Tue, 05 Jun 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/certificate-validate

OpenStack now supports signature verification for signed images. However, it does not support strong certificate validation for certificates used to generate image signatures. Specifically, cinder has no mechanism to identify trusted certificates. While cinder verifies the signature of a signed image using cursive, there is no way to determine if the certificate used to generate and verify that signature is a certificate that is trusted by the user. This change will introduce an addition to the cinder API allowing the user to specify a list of trusted certificates when creating volume from images. These trusted certificates will be used to conduct certificate validation in concert with signature verification in cursive, providing the user confidence in the identity and integrity of the image being created.

Problem description

Cinder is capable of verifying the signature of a signed image by using cursive, the OpenStack signature verification library [1]. Signature verification ensures that unmodified image data is retrieved from glance. However, the validation of the certificate used to generate the signature of the signed image is currently limited to a timestamp validity check, ensuring only that the certificate is valid for use at the time of signature verification. There is no mechanism to ensure that the certificate used is one approved by the end user. An attacker with access to glance could replace a user’s signed image with a modified, malicious image signed with the attacker’s certificate, stored in the OpenStack deployment’s certificate manager. If asked to create volume from the modified image, cinder would retrieve the image and its corresponding certificate, verify the image signature, and build a volume using the malicious image data. Providing support for certificate validation in cursive helps cinder detect this attack scenario and take steps to alert the user of the potential compromise.

Note that this threat model considers glance to be untrusted and does not include threats to the integrity, availability, or confidentiality of cinder. It assumes that: (1) an attacker has access to the certificate manager and is able to store certificates for use with image signing, and (2) this attacker is unable to access arbitrary certificate public/private key pairs belonging to other users. An attacker with such access would be able to impersonate the user, replacing signed images and perfectly updating the corresponding image signatures and metadata as needed to conceal the attack.

Use Cases

Cinder users want to ensure that they are creating from images they trust by controlling the set of certificates used to sign their images.

With this change, a cinder user can specify the identities of trusted certificates when creating volumes from a signed image if signature verification and certificate validation are enabled. One of these trusted certificates is expected to be the signing certificate of the certificate used to generate the image signature.

With certificate validation enabled, image signature verification will only succeed if the image signing certificate was generated by a trusted certificate. This allows users to place themselves in-the-loop of the signature verification process, requiring valid certificate information for boot to succeed. Note that if the deployment is configured to trust certain certificates that the user isn’t aware of, the deployer could still manipulate images in a way that the user wouldn’t detect with this feature.

Proposed change

Supporting certificate validation requires several changes. The initial change adds a pair of new configuration options. The first configuration option, enable_certificate_validation, will enable the use of the cursive certificate validation routine when conducting image signature verification (see the fifth change below). This option will only be used if verify_glance_signatures is set to enabled and will default to False. Certificate validation will be performed if this option is set to True.

The second configuration option, default_trusted_certificate_ids, will contain a list of certificate IDs that are designated as trusted by the cinder deployment. This list of trusted certificate IDs will only be used if certificate validation is enabled and if no trusted certificate IDs were provided by the user (see the second and third changes below). The list should be defined by an administrator as it will be the default set of trusted certificate IDs for the cinder deployment. The default value of this option will be an empty list, requiring a user-provided set of trusted certificate IDs if left empty. If the user does provide a list of trusted certificate IDs, the list of default trusted certificate IDs will be ignored.

The second change adds a parameter, trusted_image_certificates, to the volume create command of the cinder API. The value of the parameter is an array containing the string IDs of the trusted certificates used to validate the signed image’s signing certificate. These IDs are assigned by the certificate manager upon upload of the trusted certificates. Multiple IDs are allowed here to provide flexibility for the user. It may not be feasible for the user to know which specific certificate corresponds to their image. Allowing the user to define a set of trusted certificates removes the need for an image/certificate mapping, simplifying the user experience. When provided, cinder will pass these values to the certificate validation routine in cursive, overriding the default list of trusted certificate IDs (see the second configuration option above). Cursive will use them to fetch the trusted certificates using castellan. If provided, the value of the parameter will be saved in image metadata.

The third change integrates the certificate verification routine into the signature verification workflow. When the certificate verification routine fetches the image’s signing certificate, it builds the verification context using the trusted certificates provided by the user. It then passes the signing certificate through the context for certificate validation. If validation succeeds, signature verification can proceed as normal. If validation fails, signature verification fails as well, the volume is placed in an ERROR state.

NOTE: The corresponding feature [2] has been merged into Cursive already.

It is possible a silent fail scenario could occur if (1) cinder is not configured to conduct certificate validation, and (2) the user provides trusted certificate IDs expecting certificate validation to occur. In this case, cinder would not conduct certificate validation and would boot the instance, causing the user to believe certificate validation succeeded even though it never happened. To prevent this from happening, the create workflow will be updated to conduct signature and certificate verification if trusted certificate IDs are associated with the create request. This matches the user’s expectations and prevents a silent fail scenario from ever occurring. Note here that this override only occurs if the user specifies certificate IDs. The default list of certificate IDs is only used if the feature is enabled and therefore will never trigger the override.

The fourth change updates the cinderclient to support the trusted_image_certificates parameter for the volume create commands. This includes support for a new environment variable, OS_TRUSTED_CERTIFICATE_IDS, that can be used to define a comma-delimited list of trusted certificate IDs. If the trusted_image_certificates parameter is not used, the client will pull the value of the environment variable and use it instead. This value will be converted into a list before being passed on.

If the user does not provide a value for the trusted_image_certificates parameter, either explicitly or through the OS_TRUSTED_CERTIFICATE_IDS environment variable, cinder will pull the list of trusted certificate IDs from the default_trusted_certificate_ids configuration option. If this option is left as an empty list, there is no way for nova to obtain a trusted certificate for certificate validation. In this case there would be no way to determine if the image’s signing certificate is trusted so signature verification would fail, in turn failing volume creation.

To be clear, we will verify the image’s signature only when image is downloaded from glance and content is copied to volume on host. So when image volume is created via clone_image or clone_image_volume we will skip this verification process, in order not to confuse end users, we will also add a flag in volume image metadata to indicate whether we validated the certificates during volume creation, and the service will generate an user message if the certificate validation fails in the backend.

Note that these trusted certificates are stored in a certificate manager independent of the volume service. For this work, a certificate manager is any service backend supported by castellan that provides management operations for certificates objects. Certificate management is often a subset of the functionality provided by generic key managers, which are capable of managing different types of cryptographic secrets (e.g., encryption keys, passwords). As of the Pike release, barbican (the OpenStack key management service) is the only OpenStack service that satisfies the requirements for a certificate manager. In the future, any OpenStack or third-party service that is supported by castellan and provides certificate management could be used instead of barbican.

Alternatives

An alternative approach to certificate validation here would be to support certificate trust stores, collections of trusted certificates associated with individual users or projects. When creating a new server, the user would specify their trust store as a source of trusted certificates, replacing the list of certificates provided in the trusted_image_certificates parameter. There are many ways to support trust stores, including: a filesystem directory trust store containing trusted certificate files stored locally on the compute host, a metadata/managed resource approach supported under services like nova or keystone, and a container-based secret storage approach supported by services like barbican. While useful in defining collections of trusted certificates, a trust store approach would need to scale for large cloud deployments which may be difficult from a management and maintenance perspective. Trust stores also introduce a new construct that must be trusted by the user, especially if the user is not directly responsible for maintaining their trust store. These restrictions may not be feasible for some cloud deployments.

An alternative to the user providing trusted certificates, or storing trusted certificates in a trust store(or just a database table), would be to dynamically fetch certificates using information stored in the Private Internet Extension of the signed certificate being validated. This approach allows deployers and users to use signing certificates without needing to pre-fetch all of the root and intermediate certificates required to complete the certificate validation process. However, this approach requires the compute service have persistent network access to all possible certificate repositories where root and intermediate certificates may be stored. In many cases, this will include network access to the public Internet which may not be feasible for a generic deployment.

An enhanced certificate validation routine would include certificate revocation, supporting commonly used approaches like certificate revocation lists (CRLs) and/or the Online Certificate Status Protocol (OCSP). Supporting certificate revocation would allow the compute service to dynamically determine when certificates become invalid in real time due to compromise, further improving the security of booting signed images. However, supporting certificate revocation involves dynamically fetching and trusting network resources, often under the control and authority of third-parties. This may not be feasible for some deployments. It is possible that certificate revocation could be integrated outside of the compute service, for example within the certificate manager or through another third-party service. This would grant nova the benefits of timely revocation without complicating the signature verification and certificate validation features in nova itself.

It should be noted here that support for certificate revocation is intended to be added in future work for this feature.

Data model impact

None

REST API impact

Add a new microversion to support specifying trusted_image_certificates when creating volume:
```
POST: /V3/{tenant_id}/volumes
```

{
  "volume": {
      "size": 1,
      "name": "image_volume",
      "imageRef": "CAB56EC2-4BDC-45D4-9898-3F88A7003261",
      "trusted_image_certificates": [
          "00000000-0000-0000-0000-000000000000",
          "11111111-1111-1111-1111-111111111111",
          "22222222-2222-2222-2222-222222222222"
      ],
  }

Note that while in these examples the values in trusted_image_certificates are UUIDs they are not guaranteed to be so. Certificate managers use different ID allocation schemes; while some use strict UUIDs, others use simple incrementing integers or raw hex strings. For this feature, the type of trusted_image_certificates will be an array containing zero or more JSON string values.

The following is a JSON schema description of the trusted_image_certificates parameter:

{
    "type": "array",
    "minItems": 0,
    "maxItems": 50,
    "uniqueItems": true,
    "items": {
        "type": "string"
    }
}

Note the upper and lower bounds for the number of certificate IDs included in the trusted_image_certificates parameter. If an API call is made for a signed image and exceeds the maximum number of allowed certificate IDs, then the API call will fail.

API version bump will be required for these changes.

Security impact

With the added verification step provided by this feature when enabled, the security of the signed image verification feature is improved.

Notifications impact

None

Other end user impact

With support being added for the OS_TRUSTED_CERTIFICATE_IDS environment variable, users are encouraged to set the variable with the list of trusted certificate IDs through their openrc file, alongside their authentication credentials. The value of the OS_TRUSTED_CERTIFICATE_IDS environment variable is a comma-delimited string of trusted certificate IDs, which will be converted into a list of certificate IDs for the trusted_image_certificates parameter.

An example openrc file is shown below, using the same trusted certificate IDs as those used in the API example (see REST API Impact above):

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=projectName
export OS_AUTH_URL=https://identityHost:portNumber/v2.0
export OS_TRUSTED_CERTIFICATE_IDS=00000000-0000-0000-0000-000000000000,
11111111-1111-1111-1111-111111111111,22222222-2222-2222-2222-222222222222

Performance Impact

Cinder will load the user’s trusted certificates via cursive every time signature verification is performed. Depending upon the size and number of certificates, and the frequency of signature verification, this could introduce a performance burden on the volume service, such as consuming images from the Cinder image cache. To alleviate this, see “Alternatives” above regarding a persistent certificate trust store and dynamically loading certificates from remote storage.

Other deployer impact

The inclusion of two new configuration options, enable_certificate_validation and default_trusted_certificate_ids, will smooth the transition for deployments looking to enable this feature.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Yikun Jiang(yikunkero@gmail.com)
Other assignee:: tommylikehu(tommylikehu@gmail.com)

Work Items

Add two new configuration options, enable_certificate_validation and default_trusted_certificate_ids. The first will enable the use of certificate validation if signature verification is enabled. The second will provide a default list of trusted certificate IDs that can be used if no trusted certificate IDs are provided with the volume request.
Update the existing signature verification workflow in cinder to incorporate certificate validation, using the verify_certificate routine in cursive to validate the signing certificate.
Add a new cinder API parameter, trusted_image_certificates, to the volume create command. The value of this parameter will need to be passed through to the signature verification step when downloading the image from glance.
Update cinderclient to support the trusted_image_certificates parameter.
Update cinderclient to pull the value of the OS_TRUSTED_CERTIFICATE_IDS environment variable when the trusted_image_certificates parameter is not provided by the user.

Dependencies

None

Testing

Unit tests will be included to test the functionality implemented in cinder, cinderclient. Tempest tests will also be implemented to test the end-to-end feature across glance and cinder.

Documentation Impact

Documentation on the trusted_image_certificates API parameter and the two new configuration options will need to be added, as will instructions defining the OS_TRUSTED_CERTIFICATE_IDS environment variable and its usage.

References

[1] “Cursive.” Online: https://launchpad.net/cursive

[2] “Add certificate validation.” https://review.openstack.org/#/c/357202/

API Validation

Tue, 01 May 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/json-schema-validation

Currently, Cinder has different implementations for validating request bodies. The purpose of this blueprint is to track the progress of validating the request bodies sent to the Cinder server, accepting requests that fit the resource schema and rejecting requests that do not fit the schema. Depending on the content of the request body, the request should be accepted or rejected consistently.

Problem description

Currently Cinder doesn’t have a consistent request validation layer. Some resources validate input at the resource controller and some fail out in the backend. Ideally, Cinder would have some validation in place to catch disallowed parameters and return a validation error to the user.

The end user will benefit from having consistent and helpful feedback, regardless of which resource they are interacting with.

Use Cases

As a user or developer, I want to observe consistent API validation and values passed to the Cinder API server.

Proposed change

One possible way to validate the Cinder API is to use jsonschema similar to Nova, Keystone and Glance (https://pypi.org/project/jsonschema). A jsonschema validator object can be used to check each resource against an appropriate schema for that resource. If the validation passes, the request can follow the existing flow of control through the resource manager to the backend. If the request body parameters fails the validation specified by the resource schema, a validation error wrapped in HTTPBadRequest will be returned from the server.

Example: “Invalid input for field ‘name’. The value is ‘some invalid name value’.

Each API definition should be added with the following ways:

Create definition files under ./cinder/api/schemas/.
Each definition should be described with JSON Schema.
Each parameter of definitions(type, minLength, etc.) can be defined from current validation code, DB schema, unit tests, Tempest code, or so on.

Some notes on doing this implementation:

Common parameter types can be leveraged across all Cinder resources. An example of this would be as follows:

from cinder.api.validation import parameter_types
# volume create schema
<snip>
     create = {
        'type': 'object',
        'properties': {
            'volume': {
                'type': 'object',
                'properties': {
                    'size': parameter_types.positive_integer,
                    'availability_zone': parameter_types.availability_zone,
                    'source_volid': {
                        'format': 'uuid',
                    },
                    'description': parameter_types.description,
                    'multiattach': parameter_types.boolean,
                    'snapshot_id': {
                        'format': 'uuid',
                    },
                    'name': parameter_types.name,
                    'imageRef': {
                        'format': 'uuid',
                    },
                    'volume_type': {
                        'format': 'uuid',
                    },
                    'metadata': {
                        'type': 'object',
                     },
                    'consistencygroup_id': {
                        'format': 'uuid',
                    },
                },
            'required': ['size'],
            'additionalProperties': False,
        }
        'required': ['server'],
        'additionalProperties': False,
    }
}

create_v312 = copy.deepcopy(create)
create_v312['properties']['volume'][
    'properties']['group_id'] = {'format': 'uuid',},

parameter_types.py:

name = {
    'type': 'string', 'minLength': 0, 'maxLength': 255,
}

positive_integer = {
    'type': ['integer', 'string'],
    'pattern': '^[0-9]*$', 'minimum': 1, 'minLength': 1
}

description = name
availability_zone = name

# This registers a FormatChecker on the jsonschema module.
# It might appear that nothing is using the decorated method but it gets
# used in JSON schema validations to check uuid formatted strings.
from oslo_utils import uuidutils

@jsonschema.FormatChecker.cls_checks('uuid')
def _validate_uuid_format(instance):
    return uuidutils.is_uuid_like(instance)

boolean = {
    'type': ['boolean', 'string'],
    'enum': [True, 'True', 'TRUE', 'true', '1', 'ON', 'On', 'on',
             'YES', 'Yes', 'yes',
             False, 'False', 'FALSE', 'false', '0', 'OFF', 'Off', 'off',
             'NO', 'No', 'no'],
}

The validation can take place at the controller layer using below decorator:

from cinder.api.schemas import volume

@wsgi.response(http_client.ACCEPTED)
@validation.schema(volume.create, "3.0")
@validation.schema(volume.create_v312, "3.12")
def create(self, req, body):
    """creates a volume.

     version 3.12 added groupd_id to the volume request body.
     If user has requested volume create with version v2 then the
     'validation.schema' decorator will ignore its schema validation and
     will pass the request as it is in the create method."""

Initial work will include capturing the Block Storage API Spec for existing resources in a schema. This should be a one time operation for each major version of the API. This will be applied to the Block Storage V3 API.
When adding a new extension to Cinder, the new extension must be proposed with its appropriate schema.

Alternatives

Before the API validation framework, we needed to add the validation code into each API method in ad-hoc. These changes would make the API method code dirty and we needed to create multiple patches due to incomplete validation.

If using JSON Schema definitions instead, acceptable request formats are clear and we don’t need to do ad-hoc works in the future.

Pecan Framework: Pecan Some projects(Ironic, Ceilometer, etc.) are implemented with Pecan/WSME frameworks and we can get API documents automatically from the frameworks. In WSME implementation, the developers should define API parameters for each API. Pecan would make the implementations of API routes(URL, METHOD) easy.

Data model impact

None

REST API impact

API Response code changes:

There are some occurrences where API response code will change while adding schema layer for them. For example, On current master ‘services’ table has ‘host’ and ‘binary’ of maximum 255 characters in database table. While updating service user can pass ‘host’ and ‘binary’ of more than 255 characters which obviously fails with 404 ServiceNotFound wasting a database call. For this we can restrict the ‘host’ and ‘binary’ of maximum 255 characters only in schema definition of ‘services’. If user passes more than 255 characters, he/she will get 400 BadRequest in response.

API Response error messages:

There will be change in the error message returned to user. For example, On current master if user passes more than 255 characters for volume name then below error message is returned to user from cinder-api:

Invalid input received: name has <actual no of characters user passed> characters, more than 255.

With schema validation below error message will be returned to user for this case:

Invalid input for field/attribute name. Value: <value passed by user>. ‘<value passed by user>’ is too long.

Security impact

The output from the request validation layer should not compromise data or expose private data to an external user. Request validation should not return information upon successful validation. In the event a request body is not valid, the validation layer should return the invalid values and/or the values required by the request, of which the end user should know. The parameters of the resources being validated are public information, described in the Block Storage API spec, with the exception of private data. In the event the user’s private data fails validation, a check can be built into the error handling of the validator to not return the actual value of the private data.

jsonschema documentation notes security considerations for both schemas and instances: http://json-schema.org/latest/json-schema-core.html#anchor21

Better up front input validation will reduce the ability for malicious user input to exploit security bugs.

Notifications impact

None

Other end user impact

None

Performance Impact

Cinder will need some performance cost for this comprehensive request parameters validation, because the checks will be increased for API parameters which are not validated now.

Other deployer impact

None

Developer impact

This will require developers contributing new extensions to Cinder to have a proper schema representing the extension’s API.

Implementation

Assignee(s)

Primary assignee: Dinesh_Bhor (Dinesh Bhor <dinesh.bhor@nttdata.com>)

Work Items

Initial validator implementation, which will contain common validator code designed to be shared across all resource controllers validating request bodies.
Introduce validation schemas for existing core API resources.
Introduce validation schemas for existing API extensions.
Enforce validation on proposed core API additions and extensions.
Remove duplicated ad-hoc validation code.
Add unit and end-to-end tests of related API’s.
Add/Update cinder documentation.

Dependencies

The extension’s which are there under cinder/api/contrib/ are getting called by v2 as well as v3. So if we add schema validation for v3 then we will have to remove the existing validation of parameters which is there inside of controller methods which will again break the v2 apis.

Solution:

Do the schema validation for v3 apis using the @validation.schema decorator similar to Nova and also keep the validation code which is there inside of method to keep v2 working.
Once the decision is made to remove the support to v2 we should remove the validation code from inside of method.

Testing

Tempest tests can be added as each resource is validated against its schema. These tests should walk through invalid request types.

We can follow some of the validation work already done in the Nova V3 API:

Negative validation tests should use tempest.test.NegativeAutoTest

Documentation Impact

The cinder API documentation will need to be updated to reflect the REST API changes.
The cinder developer documentation will need to be updated to explain how the schema validation will work and how to add json schema for new API’s.

References

Useful Links:

[Understanding JSON Schema] (http://spacetelescope.github.io/understanding-json-schema/reference/object.html)
[Nova Validation Examples] (http://git.openstack.org/cgit/openstack/nova/tree/nova/api/validation)
[JSON Schema on PyPI] (https://pypi.org/project/jsonschema)
[JSON Schema core definitions and terminology] (http://tools.ietf.org/html/draft-zyp-json-schema-04)
[JSON Schema Documentation] (http://json-schema.org/documentation.html)

Cinder Volume Active/Active support - Replication

Fri, 09 Feb 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support

As it stands to reason replication v2.1 only works in deployment configurations that were available and supported in Cinder at the time of its design and implementation.

Now that we are also supporting Active-Active configurations this translates to replication not properly working on this new supported configuration.

This spec extends replication v2.1 functionality to support Active-Active configurations while preserving backward compatibility for non clustered configurations.

Problem description

On replication v2.1 failover is requested on a per backend basis, so when a failover request is received by the API it is then redirected to a specific volume service via an asynchronous RPC call using that service’s topic message queue. Same thing happens for freeze and thaw operations.

It works when we have a one-to-one relation between volume services and storage backends, but it doesn’t when we have many-to-one relationship because the failover RPC call will be received by only one of the services that form the cluster for the storage backend and the others will be oblivious to this change and will continue using the same replication site they had been using before. This will result in some operations succeeding, those going to the service that performed the failover, and some operations failing, since they are going to the site that’s not available.

While that’s the primary issue, it’s not the only one, since we also have to track the replication status at the cluster level.

Use Cases

Users want to have highly available cinder services with disaster recovery using replication.

It is not enough that individual features will be available on their own as they’ll want to have them both at the same time; so being able to use either Active-Active configurations without replication, or replication if not deployed as Active-Active, is insufficient.

They could probably make it work if they stopped all but one volume services in the cluster, issued the failover request, and once it has been completed they brought the other services back up, but this would not be a clean approach to the problem.

Proposed change

The proposed change in its core is to divide the failover operation in the driver into two individual operations, one that will do the side of things related with the storage backend, for example force promoting volumes to primary on the secondary site, and another that will make the driver perform all the operations against the secondary storage device.

As mentioned before only one volume service will receive the request to do the failover, so by splitting the operation the manager will be able to request the local driver to do the first part of the failover and once that is done it will send all volume nodes in the cluster handling that backend the signal that the failover has been completed and that they should start pointing to the failed over secondary site, thus solving the problem of some services not knowing that a new site should be used.

This will also require two homonymous RPC calls to the drivers new methods in the volume manager: failover and failover_completed.

We will also add the replication information to the clusters table to track replication at the cluster level for clustered services.

Given current use of the freeze and thaw operation there doesn’t seem to be a reason to do the same split, so these operations would be left as they are and will only be performed by one volume service when requested.

This change will require vendors to update their drivers to support replication on Active-Active configurations, so to avoid surprises we will be preventing the volume service from starting in Active-Active configurations with replication enabled on drivers that don’t support the Active-Active mechanism.

Alternatives

The splitting mechanism for the failover_host method is pretty straight forward, the only alternative to the proposed changed would be to split the thaw and freeze operations as well.

Data model impact

Three new fields related to the replication will be added to the clusters table. These will be the same fields we currently have in the services table and will hold the same meaning:

replication_status: String storing the replication status for the whole cluster.
active_backend_id: String storing which one of the replication sites is currently active.
frozen: Boolean reflecting whether the cluster is frozen or not.

These fields will be kept in sync between the clusters table and the services table for consistency.

REST API impact

A new action called failover equivalent to existing failover_host will be added, and it will support a new cluster parameter in addition to the host field already available in failover_host.
Cluster listing will accept replication_status, frozen and active_backend_id as filters,
Cluster listing will return additional replication_status, frozen and active_backend_id fields.

Security impact

None.

Notifications impact

None.

Other end user impact

The client will return the new fields when listing clusters using the new microversion and new filters will also be available.

Failover for this microversion will accept the cluster parameter.

Performance Impact

The new code should have no performance impact on existing deployments since it will only affect new Active-Active deployments.

Other deployer impact

None.

Developer impact

Drivers that wish to support replication on Active-Active deployments will have to implement failover and failover_completed methods as well as the current failover_host method since it is being used for backward compatibility with the base replication v2.1.

The easiest way to support this with minimum code would be to implement failover and failover_completed and then create failover_host based on those:

def failover_host(self, volumes, secondary_id):
    self.failover(volumes, secondary_id)
    self.failover_completed(secondary_id)

Implementation

Assignee(s)

Primary assignee:: Gorka Eguileor (geguileo)
Other contributors:: None

Work Items

Change service start to use active_backend_id from the cluster or the service.
Add new failover REST API
Update list REST API method to accept new filtering fields and update the view to return new information.
Update the DB model and create migration
Update Cluster Versioned Object
Make modifications to the manager to support the new RPC calls.

Dependencies

This work has no additional dependency besides the basic Active-Active mechanism being in place, which it already is.

Testing

Only unit tests will be implemented, since there is no reference driver that implements replication and can be used at the gate.

We also lack a mechanism to actually verify that the replication is actually working.

Documentation Impact

From a documentation perspective there won’t be much to document besides the changes related to the API changes.

References

Replication v2.1

Cinder Pagination Sorting Enhancements

Tue, 06 Feb 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-pagination

Currently, the sorting support for Cinder allows the caller to specify a single sort key and sort direction. This blueprint enhances the sorting support for the /volumes and /volumes/detail APIs so that multiple sort keys and sort directions can be supplied on the request.

Problem description

There is no support for retrieving volume data based on multiple sort keys; a single sort key and direction is currently supported and it is defaulted to descending sort order by the “created_at” key. In order to retrieve data in any sort order and direction, the REST APIs need to accept multiple sort keys and directions.

Use Case: A UI that displays a table with only the page of data that it has retrieved from the server. The items in this table need to be sorted by status first and by display name second. In order to retrieve data in this order, the APIs must accept multiple sort keys/directions.

Use Cases

Proposed change

The /volumes and /volumes/detail APIs will align with the API working group guidelines (see References section) for sorting and support the following parameter on the request:

sort: Comma-separated list of sort keys, each key is optionally appended with <:dir>, where ‘dir’ is the direction for the corresponding sort key (supported values are ‘asc’ for ascending and ‘desc’ for descending)

For example: /volumes?sort=status:asc,display_name:asc,created_at:desc

Note: The “created_at” and “id” sort keys are always appended at the end of the key list if they are not already specified on the request.

The database layer already supports multiple sort keys and directions. This blueprint will update the API layer to retrieve the sort information from the API request and pass that information down to the database layer.

All sorting is handled in the cinder.common.sqlalchemyutils.paginate_query function. This function accepts an ORM model class as an argument and the only valid sort keys are attributes on the given model class. Therefore, the valid sort keys are limited to the model attributes on the models.Volume class.

Alternatives

Multiple sort keys and directions could be passed using repeated ‘sort_key’ and ‘sort_dir’ query parameters. For example:

/volumes?sort_key=status&sort_dir=asc&sort_key=display_name&sort_dir=asc& sort_key=created_at&sort_dir=desc

However, this is not aligned with the API sorting guidelines.

Also, this additional sorting could be conditionally enabled based on the existence of a new extension; however, since sorting by a single key is already supported, then a new extension may not be needed to enhance this support to multiple keys/directions. However, an extension could be created if necessary.

Data model impact

None

REST API impact

The following existing v2 GET APIs will support the new sorting parameters:

/v2/{tenant_id}/volumes
/v2/{tenant_id}/volumes/detail

Note that the design described in this blueprint could be applied to other GET REST APIs but this blueprint is scoped to only those listed above. Once this design is finalized, then the same approach could be applied to other APIs.

The existing API documentation needs to be updated to include the following new Request Parameters:

Parameter	Style	Type	Description
sort	query	string	Comma-separated list of sort keys and optional sort directions in the form of key<:dir>, where ‘dir’ is either ‘asc’ for ascending order or ‘desc’ for descending order. Defaults to the ‘created_at’ and ‘id’ keys in descending order.

Currently, the volumes query supports the ‘sort_key’ and ‘sort_dir’ parameters; these will be deprecated. The API will raise a “badRequest” error response (code 400) if both the new ‘sort’ parameter and a deprecated ‘sort_key’ or ‘sort_dir’ parameter is specified.

Neither the API response format nor the return codes will be modified, only the order of the volumes that are returned.

In the event that an invalid sort key is specified then a “badRequest” error response (code 400) will be returned with a message like “Invalid input received: Invalid sort key”.

Security impact

None

Notifications impact

None

Other end user impact

The cinderclient should be updated to accept sort keys and sort directions, using the ‘sort’ parameter being proposed in the cross-project spec: https://review.openstack.org/#/c/145544/

Performance Impact

All sorting will be done in the database. The choice of sort keys is limited to attributes on the models.Volume ORM class – not every attribute key returned from a detailed query is a valid sort key.

Performance data was gathered by running on a simple devstack VM with 2GB memory. 5000 volumes were inserted into the DB. The data shows that the sort time on the main data table is dwarfed (see first table below) when running a detailed query – most of the time is spent querying the other tables for each item; therefore, the impact of the sort key on a detailed query is minimal.

For example, the data below compares the processing time of a GET request for a non-detailed query to a detailed query with various limits using the default sort keys. The purpose of this table is to show the processing time for a detailed query is dominated by getting the additional details for each item.

Limit	Non-Detailed (sec)	Detailed (sec)	Non-Detailed / Detailed %
50	0.0560	0.8621	6.5%
100	0.0813	1.6839	4.8%
150	0.0848	2.4705	3.4%
200	0.0874	3.2502	2.7%
250	0.0985	4.1237	2.4%
300	0.1229	4.8731	2.5%
350	0.1262	5.6366	2.2%
400	0.1282	6.5573	2.0%
450	0.1458	7.2921	2.0%
500	0.1770	8.1126	2.2%
1000	0.2589	16.0844	1.6%

Non-detailed query data was also gathered. The table below compares the processing time using default sort keys to the processing using display_name as the sort key. Items were added with a 40 character display_name that was generated in an out-of-alphabetical sort order.

Limit	Default keys (sec)	display_name key (sec)	Slowdown %
50	0.0560	0.0600	7.1%
100	0.0813	0.0832	2.3%
150	0.0848	0.0879	3.7%
200	0.0874	0.0906	3.7%
250	0.0985	0.1031	4.7%
300	0.1229	0.1198	-2.5%
350	0.1262	0.1319	4.5%
400	0.1282	0.1368	6.7%
450	0.1458	0.1458	0.0%
500	0.1770	0.1619	-8.5%
1000	0.2589	0.2659	2.7%

In conclusion, the sort processing on the main data table has minimal impact on the overall processing time. For a detailed query, the sort time is dwarfed by other processing – even if the sort time when up 3x it would only represent 4.8% of the total processing time for a detailed query with a limit of 1000 (and only increase the processing time by .11 sec with a limit of 50).

Other deployer impact

The choice of sort keys has a minimal impact on data retrieval performance (see performance data above). Therefore, the user should be allowed to retrieve data in whatever order they need to for creating their views (see use case in the Problem Description).

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: kaufer (kaufer@us.ibm.com)
Other contributors:: None

Work Items

Ideally the logic for processing the sort parameters would be common to all components and would be done in oslo; a similar blueprint is also being proposed in nova: https://blueprints.launchpad.net/nova/+spec/nova-pagination

Therefore, I see the following work items:

Duplicate the common code being proposed in nova to process the sort parameters, see https://review.openstack.org/#/c/95260/. Once both projects are using the same code then it should be moved into oslo.
Update the API to retrieve the sort information and pass down to the DB layer (requires changes to volume/api.py, db/api.py, and db/sqlalchemy/api.py)
Update the cinderclient to accept and process multiple sort keys and sort directions

Dependencies

Related (but independent) change being proposed in nova: https://blueprints.launchpad.net/nova/+spec/nova-pagination
CLI Sorting Argument Guidelines cross project spec: https://review.openstack.org/#/c/145544/

Testing

Both unit and Tempest tests need to be created to ensure that the data is retrieved in the specified sort order. Tests should also verify that the default sort keys (“created_at” and “id”) are always appended to the user supplied keys (if the user did not already specify them).

Testing should be done against multiple backend database types.

Documentation Impact

The /volumes and /volumes/detail API documentation will need to be updated to:

Reflect the new sorting parameters and explain that these parameters will affect the order in which the data is returned.
Explain how the default sort keys will always be added at the end of the sort key list

The documentation could also note that query performance will be affected by the choice of the sort key, noting which keys are indexed.

References

API Working group sorting guidelines: https://github.com/openstack/api-wg/blob/master/guidelines/ pagination_filter_sort.rst

Volume Migration Improvement

Tue, 06 Feb 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/migration-improvement

This specification proposes to improve the current volume migration in terms of implementing a better way to manage the volume migration status, adding the migration progress indication, enriching the notification system via reporting to Ceilometer, guaranteeing the migration quality via tempest tests and CI support, etc. It targets to resolve the current issues for the available volumes only, since we need to wait until the multiple attached volume functionality lands in Nova to resolve the issues related to the attached volumes. There is going to be another specification designed to cover the issues regarding the attached volumes.

Use Cases

Having the well defined capabilities will allow the deployer to see what common capabilities are shared beyond their deployed backends in Cinder.

There are three cases for volume migration. The scope of this spec is for the available volumes only and targets to resolve the issues within the following migration Case 1 and 2:

Within the scope of this spec 1) Available volume migration using the “dd” command. For example, migration from LVM to LVM, between LVM and vendor driver, and between different vendor drivers.

2) Available volume migration between two pools from the same vendor driver using driver-specific way. Storwize is taken as the reference example for this spec.

Out of the scope of the spec 3) In-use(attached) volume migration using Cinder generic migration.

Problem description

Currently, there are quite some straightforward issues about the volume migration. 1. Whether the migration succeeds or fails is not saved anywhere, which is very confusing for the administrator. The volume status is still available or in-use, even if the administrator mentions –force as a flag for cinder migrate command. 2. From the API perspective, the administrator is unable to check the status of the migration. The only way to check if the migration fails or succeeds is to check the database. 3. During the migration, there are two volumes appearing in the database record via the check by “cinder list” command. One is the source volume and one is the destination volume. The latter is actually useless to show and leads to confusion for the administrator. 4. When executing the command “cinder migrate”, most of the time there is nothing returned to the administrator from the terminal, which is unfriendly and needs to be improved. 5. It is probable that the migration process takes a long time to finish. Currently the administrator gets nothing from the log about the progress of the migration. 6. Nothing reports to the Ceilometer. 7. There are no tempest test cases and no CI support to make sure the migration truly works for any kind of drivers.

We propose to add the management of the migration status to resolve issues 1 to 4, add the migration progress indication to cover Issue 5, add the notification to solve Issue 6 and add tempest tests and CI support to tackle Issue 7.

Proposed change

At the beginning, we declare that all the changes and test cases are dedicated to available volumes. For the attached volumes, we will wait until the multiple attached volume functionality get merged in Nova.

Management of the volume migration status:

If the migration fails, the migration status is set to “error”. If the migration succeeds, the migration status is set to “success”. If no migration is ever done for the volume, the migration status is set to None. The migration status is used to record the result of the previous migration. The migration status can be seen by the administrator only.

The administrator has several ways to check the migration status: 1) The administrator can do a regular “volume list” with a filter “migration_status=<expected volume migration status>” to find all the volumes with the specified migration status. If no filter is specified, all the volumes will list the migration status. 2) The administrator can issue a “get” command for a certain volume and the migration status can be found in the field ‘os-vol-mig-status-attr:migstat’.

If the administrator issues the migrate command with the –force flag, the volume status will be changed to ‘maintenance’. Attach or detach will not be allowed during migration. If the administrator issues the migrate command without the –force flag, the volume status will remain unchanged. Attach or detach action issued during migration will abort the migration. The status ‘maintenance’ can be extended to use in any other situation, in which the volume service is not available due to any kinds of reasons.

We plan to provide more information when the administrator is running “cinder migrate” command. If the migration is able to start, we return a message “Your migration request has been received. Please check migration status and the server log to see more information.” If the migration is rejected by the API, we shall return messages like “Your migration request failed to process due to some reasons”.

We plan to remove the redundant information for the dummy destination volume. If Cinder Internal Tenant(https://review.openstack.org/#/c/186232/) is successfully implemented, we will apply that patch to hide the destination volume.

Migration progress indication:

We would like to introduce a poll mechanism to check the migration progress in a certain interval as the implementation for the migration progress indication. The poll mechanism can be realized in a loop, and the migration progress will be checked in a certain interval, which is configurable in cinder.conf. This mechanism can be running in parallel to the volume migration.

If the volume copy starts, another thread for the migration progress check will start as well. If the volume copy ends, the thread for the migration progress check ends.

A driver capability named migration_progress_report can be added to each driver. It is either True or False. This is for the case that volumes are migrated from one pool to another within the same storage back-end. If it is True, the loop for the poll mechanism will start. Otherwise, no poll mechanism will start.

A configuration option named migration_progress_report_interval can be added into cinder.conf, specifying how frequent the migration progress needs to be checked. For example, if migration_progress_report_interval is set to 30 seconds, the code will check the migration progress and report it every 30 seconds.

If the volume is migrated using dd command, e.g. volume migration from LVM to LVM, from LVM to vendor driver, from one back-end to another via blockcopy, etc, the migration progress can be checked via the position indication of /proc/<PID>/fdinfo/1.

For the volume is migrated using file I/O, the current file pointer is able to report the position of the transferred data. The migration progress can be checkedvia this position relative to EOF.

If the volume is migrated within different pools of one back-end, we would like to implement this feature by checking the stats of the storage back-end. Storwize V7000 is taken as the reference implementation about reporting the migration progress. It is possible that some drivers support the progress report and some do not. A new key “migration_progress_report” will be added into the driver to report the capability. If the back-end driver supports to report the migration progress, this key is set to True. Otherwise, this key is set to False and the progress report becomes unsupported in this case.

The migration progress can be checked by the administrator only. Since the progress is not stored, each time the progress is queried from the API, the request will be scheduled to the cinder-volume service, which can get the updated migration progress for a specified volume and reports back.

Alternatives

We can definitely use a hidden flag to indicate if a database row is displayed or hidden. However, cinder needs a consistent way to resolve other issues like image cache, backup, etc, we reach an agreement that cinder internal tenant is the approach to go.

The purpose that we plan to have a better management of the volume migration status, add migration progress indication, report the stats to Ceilometer and provide tempest tests and CI, is simply to guarantee the migration works in a more robust and stable way.

Data model impact

None

REST API impact

The REST API should be able to provide the migration status and the migration progress information for the volumes. For the migration status, it can be retrieved from the database. For the migration progress, the API request will be scheduled to the cinder volume service, where the volume is located, and cinder volume service reports the updated progress back.

Security impact

None

Notifications impact

The volume migration should send notification to Ceilometer about the start, and the progress and the finish.

Other end user impact

None

Performance Impact

None

Other deployer impact

If the back-end driver supports the migration progress indication, a new configuration option migration_progress_report_interval can be added. The administrator can decide how frequent the cinder volume service to report the migration progress. For example, if migration_progress_report_interval is set to 30 seconds, the cinder volume service will provide the progress information every 30 seconds.

Developer impact

The driver maintainer or developer should be aware that they need to add a new capability to indicate whether their driver support the progress report. If yes, they need to implement the related method, to be provided in the implementation of this specification.

If their drivers have implemented volume migration, integration tests and driver CI are important to ensure the quality. This is something they need to pay attention and implement for their drivers as well.

Implementation

Assignee(s)

Primary assignee:: Vincent Hou (sbhou@cn.ibm.com)
Other contributors:: Jay Bryant Jon Bernard

Work Items

Management of the volume migration status:

1) Change the migration_status to “error” if the migration fails; Change the migration_status to “success” if the migration succeeds. 2) Change the volume status to “maintenance” if the administrator executes the migration command with –force flag. No attach or detach is allowed during this migration. If the administrator executes the migration command without –force flag, the volume status will stay unchanged. Attach or detach during migration will terminate the migration to ensure the availability of the volume. 3) Enrich cinderclient with friendly messages returned for cinder migrate and retype command. 4) Hide the redundant dummy destination volume during migration.

Migration progress indication:

Add a loop to wrap the implementation of the poll mechanism.

The driver, which supports the migration progress report, will set migration_progress_report to True. Otherwise, set it to False.

The option migration_progress_report_interval will be used to specify the time interval, in which the migration progress is checked.

1) If the volume is migrated between LVM back-ends, or one back-end to another, the position indication of /proc/<PID>/fdinfo/1 can be checked to get the progress of the blockcopy.

2) If the volume is migrated within different pools of one back-end, we would like to check the progress report of the back-end storage in a certain time interval.

The migration percentage will be logged and reported to Ceilometer.

Notification:

Add the code to send the start, progress and end to Ceilometer during migration.

Tempest tests and CI support:

This work item is planned to finish in two steps. The first step is called manual mode, in which the tempest tests are ready and people need to configure the OpenStack environment manually to meet the requirements of the tests.

The second step is called automatic mode, in which the tempest tests can run automatically in the gate. With the current state of OpenStack infrastructure, it is only possible for us to do the manual mode. The automatic mode needs to collaboration with OpenStack-infra team and there is going to be a blueprint for it.

The following cases will be added: 1) From LVM(thin) to LVM(thin) 2) From LVM(thin) to Storwize 3) From Storwize to LVM(thin) 4) From Storwize Pool 1 to Storwize Pool 2

Besides, RBD driver is also going to provide the similar test cases from (2) to (4) as above.

We are sure that other drivers can get involved into the tests. This specification targets to add the test cases for LVM, Storwize and RBD drivers as the initiative. We hope other drivers can take the implementation of LVM, Storwize and RBD as a reference in future.

Documentation:

Update the manual for the administrators, and the development reference for the driver developers and maintainers.

Dependencies

Cinder Internal Tenant: https://review.openstack.org/#/c/186232/ Add support for file I/O volume migration: https://review.openstack.org/#/c/187270/

Testing

Depending on ability to parse the required information for the LVM driver, the following scenarios for available volumes will taken into account: 1) Migration using Cinder generic migration with LVM(thin) to LVM(thin). 2) Migration using Cinder generic migration with LVM(thin) to vendor driver. 3) Migration using Cinder generic migration with vendor driver to LVM(thin). 4) Migration between two pools from the same vendor driver using driver-specific way.

There are some other scenarios, but for this release we plan to consider the above. For scenarios 1 to 3, we plan to put tests cases into Tempest. For Scenario 4, we plan to put the test into CI. The reference case for Scenario 2 is migration from LVM to Storwize V7000. The reference case for Scenario 3 is migration from Storwize V7000 to LVM.

Documentation Impact

Documentation should be updated to tell the administrator how to use the migrate and retype command. Describe what commands work for what kind of use cases, how to check the migration status, how to configure and check the migration indication, etc.

Reference will be updated to tell the driver maintainers or developers how to change their drivers to adapt this migration improvement via the link http://docs.openstack.org/developer/cinder/devref/index.html.

References

Assisted Snapshot Improvements

Tue, 06 Feb 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/assisted-snapshot-improvements

This spec aims to improve the infrastructure used to coordinate between Cinder and Nova for volume snapshots.

Problem description

The update_snapshot_status API is used to update fields in the Cinder database which the driver performing a snapshot delete uses to determine when Nova’s part is finished and the Cinder driver can take over again.

This currently overloads the ‘progress’ field of the snapshot to carry information from the API layer to the driver. This is hacky and should be reworked as more drivers are interested in supporting assisted snapshots.

This work will also assist in supporting proper transitions between different phases of the snapshot create/delete process.

Use Cases

NA. There is no user/deployer impact as this is a internal only change.

Proposed change

API<->Volume service interaction

Establish a snapshot_admin_metadata table which is similar to the volume_admin_metadata table:

id : Integer, primary key,
key : string,
value : string,
volume_id : id ref volumes.id,
volume : relationship joining to volumes table

Setting a snapshot_status metadata value will allow the API layer to transfer this information to the volume service.

Volume manager<->driver interaction

Split the assisted snapshot processing code out of the driver (currently RemoteFS-based) and into the volume manager. This will allow drivers that support assisted snapshots to have a call for each stage, rather than only a single delete_snapshot call.

The driver will have a property indicating it supports assisted snapshots, which triggers use of the following methods in the manager instead of the manager calling the driver’s create_snapshot:

create_snapshot_assisted_begin(snapshot_ref)
create_snapshot_assisted_complete(snapshot_ref)

delete_snapshot_assisted_begin(snapshot_ref)
delete_snapshot_assisted_complete(snapshot_ref)

Each of these calls can return a dict of fields to be updated for the snapshot and/or snapshot_admin_metadata by the volume manager.

This will move some database accesses currently done in the driver to the volume manager, as well as better allowing support for defined state transitions between the Nova and Cinder phases.

Alternatives

Leave things as they are today, which may lead to a less robust/maintainable infrastructure for assisted snapshots.

Data model impact

Create new snapshot_admin_metadata table as described above.

REST API impact

update_snapshot_status() API will look for a new ‘compute_snapshot_status’ field which will be used to populate the snapshot_admin_metadata entry.

If this field is not present, it will use the ‘progress’ field as before for compatibility.

Security impact

None, interactions in and out of Cinder expose the same information and level of access as today.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Cinder will remain backward compatible with Liberty Nova for the APIs being modified here.

Developer impact

New interfaces for drivers for creating and deleting snapshots assisted by the compute service.

Implementation

Assignee(s)

Primary assignee:: deepakcs
Other contributors:: None

Work Items

Add processing to update_snapshot_status API for new fields
Add new fields to the update_snapshot_status call made by Nova
Add processing in snapshot-tracking code for the new compute_progress field
Create new driver interfaces in the volume manager
Migrate RemoteFS snapshot infrastructure to the new interfaces for the RemoteFSSnapDriver class.

Dependencies

Have Nova send new fields for update_snapshot_status API calls https://review.openstack.org/#/c/134517/

Testing

This will be covered by CI for GlusterFS, the NFS driver (once snapshots are added to it in Mitaka), and CI for other RemoteFS drivers.

Documentation Impact

None

References

Nova change: https://review.openstack.org/#/c/134517/

RBD Volume Encryption

Wed, 31 Jan 2018 00:00:00

https://blueprints.launchpad.net/nova/+spec/libvirt-qemu-native-luks

This feature adds support to the Cinder RBD volume driver to support Cinder’s volume encryption.

This requires a few changes in Cinder and Nova due to the fact that RBD volumes are attached by qemu directly and not as block devices on the host.

This fills a feature gap for the RBD driver in Cinder.

Problem description

The RBD driver does not support volume encryption.

Use Cases

Volume encryption is a common requirement for deployments, particularly where a deployer needs to meet particular security standards.

Proposed change

Enable volume encryption for RBD via qemu’s LUKS block layer.

This means that Nova has to support libvirt operations to manage this qemu feature. This is done here:

https://review.openstack.org/#/c/523958/

We also need Cinder to format volumes upon creation with a LUKS header. This is currently done by os-brick for iSCSI drivers, but can’t be done in the same way for RBD since there is no block device on the compute host, and dm-crypt is not used.

(Note: this will also be true when using qemu’s iSCSI initiator with Nova)

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

This is a security-focused feature, but it uses the already existing infrastructure of Cinder volume encryption.

The way encryption works when using RBD is slightly different from other Cinder drivers. Decryption/encryption is handled inside of qemu rather than at the device-mapper layer on the host via dm-crypt.

This means fewer operations having to be run as root, and less exposure of decrypted data to the rest of the system via block devices.

But, the feature in general has the same security implications as cinder volume encryption does for other drivers.

Notifications impact

None

Other end user impact

None

Performance Impact

Using encryption could result in slightly higher CPU usage on compute nodes. Should be comparable to using encryption with any other Cinder driver.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: eharney
Other contributors:: lyarwood

Work Items

Dependencies

Nova changes here: - https://blueprints.launchpad.net/nova/+spec/libvirt-qemu-native-luks
QEMU 2.6
libvirt 2.2.0

Testing

This feature will be covered by the standard tempest tests used for all volume drivers.

Gate configuration issues are being sorted out here:: https://review.openstack.org/#/c/536350/

Documentation Impact

Document that volume encryption now works for the RBD volume driver
Current limitation: attached volume migration is not supported

References

Provisioning Improvements

Fri, 26 Jan 2018 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/provisioning-improvements

Cinder provisioning is still a source of pain for everyone: end users, admins, and developers. Multiple factors have contributed to our current situation, like the information being dispersed in different specs, developer’s reference, and even the code, but we also have cases of misinterpreted documentation, documentation not keeping up with the project’s evolution, and even misleading or incorrect documentation.

This spec will build on those that preceded it on the same topic [1] [2] and others related to the subject [3] to bring a consolidated and updated view on the matter as well as add a some minor improvements and fix some issues in hopes that we can provide a better experience for all involved.

Problem description

Our current situation is quite chaotic, we have volume creations that fail based on an incorrect capacity calculation that would have succeeded if we had just received an update on the stats of the backend, we have drivers that are reporting incorrect data on the stats, and we have volumes that cannot be created when they should be allowed to.

Before going any further we first need to define the terms we’ll be using to ensure they hold the same meaning for all of us, as this is the source of some of our current issues.

Disagreement on the mapping of these terms and their description is understandable, but for the sake of understanding each other we’ll hold below descriptions as true since they were defined as such in our specs and most of our code.

Improvements on the word used for the terms and field names can be discussed at another time and updated in the specs, documentation, and code accordingly.

For the sake of completeness and to remove any misunderstandings that could lead to different implementations on the drivers, which we currently have, the descriptions will also include some clarifications and examples that may reference current Cinder code.

Terminology

GB:

Even though this is formally known as the symbol representation of a gigabyte -decimal unit of measurement- we will be using it throughout the spec and our code as the symbol for the gibibyte -binary unit of measurement as defined by the International Electrotechnical Commission (IEC), with symbol GiB-, so when we talk about 1GB we are talking about 1024MB, and the same applies to TB and MB.

Total capacity:

It is the total physical capacity that would be available in the storage array’s pool being used by Cinder if no volumes were present.

This is currently being reported by the drivers as total_capacity_gb and, as the name indicates, should be reported in GB and with a precision no greater than 2 decimals.

If the storage array has 5TB of space but the pool used in Cinder is limited to 1TB then the driver should be reporting a total_capacity_gb of 1024GB.

Volume size:

It is the maximum physical size that a volume can take in the storage array.

This is referenced throughout the code as volume_size.

For a thick volume the volume_size will be the same as the free capacity we lost when it was provisioned, whereas for a thin volume it will be greater than the space used for the volume in the storage array until the volume gets completely full.

Free capacity:

It is the current physical capacity available in the storage array’s pool being used by Cinder. The number and volume sizes of the thin and thick volumes that have been provisioned by Cinder or directly in the storage array are irrelevant here.

This is currently being reported by the drivers as free_capacity_gb and, as the name indicates, should be reported in GB and with a precision no greater than 2 decimals.

If the storage array has 5TB of space with a total of 3TB available for all its pools but Cinder is using a pool that has a limit of 1TB of which it has already used 400GB and someone has manually created volumes outside of Cinder that are currently using 124GB of space, then the driver should be reporting a free_capacity_gb of 500GB (1TB = 1024GB = 400GB + 124GB + 500GB).

Provisioned capacity:

The amount of capacity that would be used in the storage array’s pool being used by Cinder if all the volumes present in there were completely full.

This is currently being reported by the drivers as provisioned_capacity_gb and, as the name indicates, should be reported in GB and with a precision no greater than 2 decimals. This is a required field and must always be present.

This includes not only volumes created by Cinder but also all other existing volumes in that backend, but does not include snapshots.

Let’s expand the earlier example from “free capacity” where 524GB of the available 1TB had already been used, and say that the 124GB that were externally created were all used by 1GB thick volumes, and that Cinder was using the 400GB with 400 thick volumes of 1GB and 20 empty thin volumes of 20GB each. In this situation our reported provisioned_capacity_gb value should be 924GB ((124 * 1GB) + (400 * 1GB) + (20 * 20GB)).

If a driver does not report the provisioned_capacity_gb data we’ll use the automatically calculated allocated_capacity_gb as described below.

Allocated capacity:

Contrary to what the name may suggest this is not referring to the “allocated” space on the storage array, but to the provisioned volumes created by this specific Cinder Volume backend process on the storage array’s pool being used by Cinder and that still present.

Important to notice that this refers to a specific service backend, so if you are running a multi-backend Cinder service or multiple Cinder Volume services where you have more than one backend configured to use the same storage array’s pool, then each one of these backends will only be reporting the sum of the volume_size of the volumes they created and not the sum of all the volume_size of the volumes that have been created by a Cinder service.

This is currently being reported by the Volume service as allocated_capacity_gb and, as the name indicates, should be reported in GB.

For two volumes had been created, one thick and one thin, each one of 1GB, then you’ll be reporting 2GB as allocated_capacity_gb, but if you were to unmanage one of those volumes then you would only be reporting 1GB, even if the volume is still there and will still be counted in the provisioned_capacity_gb.

This field is calculated directly by the Cinder core code and drivers should not calculate or report this information on their get_volume_stats method.

Over subscription ratio:

It is the maximum ratio between the “provisioned capacity” and the “total capacity” represented as a real number. A ratio of 1.0 means that the “provisioned capacity” cannot exceed the “total capacity” whereas a value of 5.0 means that the Cinder backend is allowed to create as much as 5 times the “total capacity” of the storage array’s pool in volumes.

This will only have effect when a thin provisioned volume is being created, and will be ignored for thick provisioned.

This is currently being reported by the drivers as max_over_subscription_ratio with a greater or equal value to 1.0, preferably with no more than a 2 decimal precision.

This value is optional, and when missing from the driver’s status report the value defined in the [DEFAULT] section on the Cinder scheduler receiving the request will be used. So vendors should make sure that they are correctly returning this value in their drivers if they support thin provisioning and admins should make sure they have a consistent default value of the max_over_subscription_ratio across all scheduler nodes.

Note that this ratio is per backend or per pool depending on driver implementation.

Reserved percentage:

Represents the percentage of the storage array’s “total capacity” that is reserved and should not be used for calculations. It is represented by an integer value going from 0 up to 100.

This is currently being reported by the drivers as reserved_percentage with a greater or equal value to 1.0, preferably with no more than a 2 decimal precision.

Default value is 0 if the field is missing in the status report from the backend or if the user has not defined it in the backend’s Cinder configuration. This is per backend or per pool depending on driver implementation.

Provisioning support:

Cinder backends may support up to two different types of provisioning, thin and thick and drivers are expected to indicate as capable of one of them at least in their capabilities report.

The way to report support for these is setting to true the boolean fields thin_provisioning_support and/or thick_provisioning_support. And non reported provisioning types will default to false.

A Cinder backend may support both provisioning types at the same time.

Volume provisioning type:

For Cinder backends that only support one of the provisioning types all volumes created on them will be of that type, and we can use the volume type’s extra specs to make the scheduler filter out backends not supporting a specific provisioning type:

‘thin_provisioning_support’: ‘<is> True’ or ‘<is> False’
‘thick_provisioning_support’: ‘<is> True’ or ‘<is> False’

But if our deployment is using a backend that is supporting both provisioning types simultaneously we need to be explicit about the type of provisioning we want for a volume using the volume type’s extra spec provisioning:type and setting it to thin or thick.

If no provisioning:type is defined for a volume it will default to thin if the backend is capable of it, and the driver is expected to honor this assumption.

Incorrect reports

Given above terms which were originally defined in their corresponding specs, even if there may be additional comments in this one, we can determine that there are a good number of Cinder drivers that do not follow these definitions and are reporting what would be incorrect values.

Reporting incorrect values means that on a heterogeneous cloud you’ll have inconsistent scheduling and an admin will not be able to make sense of the stats from the volumes.

To illustrate this here are some of the interpretations we can see across different drivers for the provisioned_capacity_gb:

Sum of all the volumes’ max sizes, which is correct.
Sum of all the volumes’ physical disk usage, which is wrong.
Sum of the Cinder volumes’ physical disk usage, which is wrong.

And something similar happens with the allocated_capacity_gb where drivers go and report the value directly instead of letting the Cinder core code take care of it. Drivers have been known to report here the following information:

Sum of the Cinder volumes’ physical disk usage, which is correct.
Sum of the physical disk usage, which is wrong.
Sum of all the volumes’ max sizes, which is wrong.

Provisioning calculations

Some of the creation failures are based on the provisioned_capacity_gb value being wrong, but there are other cases where Cinder’s calculations for over provisioning do not match industry’s standard definition, which for some admins create confusion and undesired behavior.

Standard provisioning calculation to check if a volume of volume_size fits is:

((provisioned_capacity_gb + volume_size) <=
 (total_capacity_gb
  x (1 - (reserved_percentage / 100.0))
  x max_over_subscription_ratio))

Whereas the Cinder calculations, which were agreed on as the best calculations for being considered safer are:

(volume_size <=
 (free_capacity_gb
  - (total_capacity_gb x reserved_percentage / 100.0))
 x max_over_subscription_ratio)

Calculating max over subscription ratio

Most deployments have very dynamic workloads each with different physical storage requirements, which means that one month we may require many volumes of which we barely use any space and next month we may require fewer volumes but use most of the provisioned capacity.

This makes it almost impossible to accurately model our storage requirements at deployment time, which is precisely when we have to set the max_over_subscription_ratio for our Cinder backends.

As requirements change one option would be to change the configuration and restart our Cinder Volume services, but since Cinder is also in the data path the restart may take a long time to do and will have a considerable impact on our cloud users.

Not being able to determine beforehand the best max_over_subscription_ratio and not being able to easily restart the Cinder service is a common pain that most operators have with backends supporting thin provisioning.

Use Cases

The basic case for fixing the status report is where we would like to have consistent reporting from our backends for the admins to see in the logs and for the scheduler to use.

Any operator using thin provisioning storage that wants to optimize their storage usage and dynamically adjust to the dynamic requirements of its cloud.

As for the alternative calculations it would greatly benefit any backend that is close to their full capacity or one that is creating huge volumes that usually never get filled in.

Proposed change

Incorrect reports

Since we have consolidated all the documentation in one place, this spec, where we clearly state expected driver behavior all driver maintainers will be urged to make their drivers compliant with it. This will mean adapt their drivers to follow this document’s definition for provisioned_capacity_gb and stop reporting the allocated_capacity_gb field.

Automatic over subscription ratio calculation

To allow automatic over subscription ratio calculation we will make existing configuration option max_over_subscription_ratio into a polymorphic option that accepts not only integers and floats, but also a new string value auto that will instruct Cinder to use our default value of 20 as the starting reference when the backend is empty and then, when there is data calculate the current value on each driver stats report with the following formula:

ratio = 1 + (`provisioned_capacity_gb` /
             (`total_capacity_gb` - `free_capacity_gb` + 1))

If the driver is not reporting provisioned_capacity_gb then we’ll proceed to use the allocated_capacity_gb instead:

ratio = 1 + (`allocated_capacity_gb` /
             (`total_capacity_gb` - `free_capacity_gb` + 1))

We are not considering the reserved capacity in this formula because the scheduler’s capacity filter already takes it into consideration when calculating the virtual free capacity.

There are a couple of drivers that are already doing this, Pure and Kaminario’s K2, but with different configuration options, pure_automatic_max_oversubscription_ratio and auto_calc_max_oversubscription_ratio respectively, so those configuration options will take precedence, if configured, over this new feature, but we’ll encourage vendors to deprecate those options.

Drivers that are making use of the max_over_subscription_ratio in a non compatible way will raise an error and fail to start, logging an appropriate message if configured with this new feature.

Provisioning calculations

Instead of keep fighting with admins and developers on which one of the approaches is best -standard calculation or Cinder’s- we will be adding a new configuration option called over_provisioning_calculation which will take values standard and cinder and default to cinder for backward compatibility and that will be used by the CapacityFilter to determine which one of the mechanism to use.

This configuration option will also affect CapacityWeigher as it will need to do the free space calculation according to the standard definition as well.

As one can assume thick provisioning will have no modifications on its behavior.

Alternatives

Don’t support standard over-provisioning calculations.
Instead of modifying CapacityFilter and CapacityWeigher create 2 new classes.
Instead of adding the over_provisioning_calculation configuration option make the filter use the options JSON file provided by scheduler_json_config_location . This data seems to be currently missing on some of the operations like migrate, extend, so that would need to changed.

Data model impact

N/A

REST API impact

The only affected API will be the get_pools API that will be able to return the 2 new fields, total_used_capacity_gb and cinder_used_capacity_gb, when they are being reported by the driver’s get_volume_stats method. Fields will not be present if the drivers are not reporting them.

Security impact

N/A

Notifications impact

N/A

Other end user impact

The user may see new fields when calling cinderclient’s get_pools.

Performance Impact

Depending on the driver and the storage array, performance could increase or decrease, since getting provisioned sizes instead of physical sizes could be faster or slower.

Other deployer impact

With the change of values returned by Cinder backends for allocated_capacity_gb and provisioned_capacity_gb we may experience failures on creating volumes until we correct the values of reserved_percentage and max_over_subscription_ratio in our cloud to the right values, since we may have been using incorrect ones.

A configuration option will be modified:

max_over_subscription_ratio: Changes its type from Float to String and adds a new possible value, auto, that will enable this new feature.

A new configuration options will be added:

over_provisioning_calculation: Will allow to select what kind of calculations the CapacityFilter does to determine if there is space for a volume in a backend. Acceptable values are standard and cinder. Default values will be cinder.

Developer impact

Driver maintainer will need to verify, and fix if necessary, their stat reports for allocated_capacity_gb and provisioned_capacity_gb and if they are using max_over_subscription_ratio in a non compatible way they should fix their driver so that it supports the feature.

Implementation

Assignee(s)

Primary assignee:: None
Other contributors:: None

Work Items

File bugs for drivers that are not in compliance.
Fix drivers stat reporting.
Add support for the 3 new fields, provisioned_capacity_precission, total_used_capacity_gb and cinder_used_capacity_gb in the scheduler, the get_pools API, and the client.
Modify CapacityFilter to support the standard over-provisioning calculation.
Modify CapacityWeigher to support standard over-provisioning calculations.
Add to the volume manager the estimation mechanism for drivers that don’t report provisioned_capacity_gb.
Update all the developers reference docs to ensure that there is no more confusion on what the report stats need to return, and make sure that the wiki page on how to contribute a driver links to that documentation explaining the importance of following it when writing the driver.

Dependencies

N/A

Testing

New unit tests will be added to test the changed code.

Documentation Impact

Since our current documentation is lacking in this aspect this will add and update it to reflect what’s expected of the driver in the stats reports.

End user documentation should also be updated.

References

Cinder API Microversions

Wed, 24 Jan 2018 00:00:00

We need a way to be able to introduce changes to the REST API to both fix bugs and add new features. Some of these changes are backwards incompatible and we currently have no way of doing this. This becomes especially important to introduce backwards incompatible changes to the Nova - Cinder API. Credit and thanks to the authors of the Nova spec upon which this is based, especially Christopeher Yeoh and Sean Dague, and to the Manila team, especially Clinton Knight.

Blueprint: https://blueprints.launchpad.net/cinder/+spec/cinder-api-microversions

Problem description

Many changes to the Cinder REST API require changes to the consumers of the API. For example, If we need to add a required parameter to a method that is called by Nova, we’d need both the Nova calling code and the cinderclient that Nova uses to change. But newer Cinder versions with the change must work with older Nova versions, and there is no mechanism for this at the moment. Adding microversions will solve this problem. With microversions, the highest supported version will be negotiated by a field in the HTTP header that is sent to the Cinder API. In the case where the field ‘versions’ is not sent (i.e. clients and scripts that pre-date this change), then the lowest supported version would be used. This means that our current Cinder API v2 would be the default, and consumers of the API that wished to use a newer version could do so.

Not in Scope: Experimental APIs. This will be done separately. Note that Experimental APIs are only different in how we choose to use the API microversions. For Nova, a microversion does not have to be backwards compatible, and a microversion might remove an API. In this definition of microversions, the Experimental APIs that are used by Manila are no different from microversions that are used by Nova. This spec assumes that Cinder will take the route used by Manila, and use the Experimental API HTTP header to indicate an API that might cause a backwards incompatible change and/or be removed. These policies do not have to be determined now, and can be decided at the time that patches with new microversions are merged.

Use Cases

Allows developers to modify the Cinder API in backwards compatible way and signal to users of the API dynamically that the change is available without having to create a new API extension.
Allows developers to modify the Cinder API in a non backwards compatible way whilst still supporting the old behaviour. Users of the REST API are able to decide if they want the Cinder API to behave in the new or old manner on a per request basis. Deployers are able to make new backwards incompatible features available without removing support for prior behaviour as long as there is support to do this by developers.
Users of the REST API are able to, on a per request basis, decide which version of the API they want to use (assuming the deployer supports the version they want). This means that a deployer who does not upgrade will not break compatibility, and clients that do not upgrade will also remain compatible.

Proposed change

Cinder will use a framework we will call ‘API Microversions’ for allowing changes to the API while preserving backward compatibility. The basic idea is that a user has to explicitly ask for their request to be treated with a particular version of the API. So breaking changes can be added to the API without breaking users who don’t specifically ask for it. This is done with an HTTP header X-OpenStack-Cinder-API-Version which is a monotonically increasing semantic version number starting from 2.1.

If a user makes a request without specifying a version, they will get the DEFAULT_API_VERSION as defined in cinder/api/openstack/wsgi.py. This value is currently 2.0 and is expected to remain so for quite a long time.

For the purposes of this discussion, “the API” is all core and optional extensions in the Cinder tree. Please note that extensions cannot be versioned. It has been discussed that the Cinder team should therefore move the API extensions into core. This move of extensions to core will take some time, and should not block making changes to the API. Since changes to the API extensions cannot be versioned using the API-microversions decorator, we will have to accept this until the work to move extensions to core is completed.

Versioning of the API should be a single monotonic counter. It will be of the form X.Y where it follows the following convention:

X will only be changed if a significant backwards incompatible API change is made which affects the API as whole. That is, something that is only very very rarely incremented.
Y when you make any change to the API. Note that this includes semantic changes which may not affect the input or output formats or even originate in the API code layer. We are not distinguishing between backwards compatible and backwards incompatible changes in the versioning system. It will, however, be made clear in the documentation as to what is a backwards compatible change and what is a backwards incompatible one.

Note that groups of similar changes across the API will not be made under a single version bump. This will minimise the impact on users as they can control changes that they want to be exposed to.

A backwards compatible change is defined as one which would be allowed under the OpenStack API Change Guidelines

A version response would look as follows for GET http://<cinder_URL>:8776

{
   "versions": [
       {
           "id": "v2.0",
           "links": [
               {
                   "href": "http://docs.openstack.org/",
                   "rel": "describedby",
                   "type": "text/html"
               },
               {
                   "href": "http://10.10.10.77:8776/v2/",
                   "rel": "self"
               }
           ],
           "media-types": [
               {
                   "base": "application/json",
                   "type": "application/vnd.openstack.volume+json;version=1"
               },
               {
                   "base": "application/xml",
                   "type": "application/vnd.openstack.volume+xml;version=1"
               }
           ],
           "min_version": "",
           "status": "SUPPORTED",
           "updated": "2014-06-28T12:20:21Z",
           "version": ""
       },
       {
           "id": "v2.1",
           "links": [
               {
                   "href": "http://docs.openstack.org/",
                   "rel": "describedby",
                   "type": "text/html"
               },
               {
                   "href": "http://10.10.10.77:8776/v2/",
                   "rel": "self"
               }
           ],
           "media-types": [
               {
                   "base": "application/json",
                   "type": "application/vnd.openstack.volume+json;version=1"
               },
               {
                   "base": "application/xml",
                   "type": "application/vnd.openstack.volume+xml;version=1"
               }
           ],
           "min_version": "2.0",
           "status": "CURRENT",
           "updated": "2015-09-16T11:33:21Z",
           "version": "2.1"
       }
   ]
}

This specifies the min and max version that the server can understand. min_version will start at 2.0 representing the v2.0 API. Note that this assumes we will drop support for v1.0 in Mitaka. It may eventually be increased if there are support burdens we don’t feel are adequate to support. This response indicates a version of 2.1 as the current version. This number would change with each monotonic increment of the API microversion.

Client Interaction

A client specifies the version of the API they want via the following approach, a new header:

X-OpenStack-Cinder-API-Version: 2.114

This conceptually acts like the accept header. This is a global API version.

Semantically this means:

If X-OpenStack-Cinder-API-Version is not provided, act as if min_version was sent.
If X-OpenStack-Cinder-API-Version is sent, respond with the API at that version. If that’s outside of the range of versions supported, return 406 Not Acceptable.
If X-OpenStack-Cinder-API-Version: latest (special keyword) return max_version of the API.

NOTE about use of “latest” as a microversion: A client should never use “latest” when calling the Cinder API, since it is possible that the client does not have support for the latest server API microversion. The use of “latest” is strictly for testing. An experimental (non-gating) Tempeset test should use the microversion “latest” to detect when the Tempest tests themselves must be updated. This experimental test will fail using “latest” when the Tempest tests are out of date, and we will thus have an automated way to detect when Tempest must be updated.

This means out of the box, with an old client, an OpenStack installation will return vanilla OpenStack responses at v2. The user or SDK will have to ask for something different in order to get new features.

Two extra headers are always returned in the response:

X-OpenStack-Cinder-API-Version: version_number
Vary: X-OpenStack-Cinder-API-Version

The first header specifies the version number of the API which was executed.

The second header is used as a hint to caching proxies that the response is also dependent on the X-OpenStack-Cinder-API-Version and not just the body and query parameters. See RFC 2616 section 14.44 for details.

Implementation design details

On each request the X-OpenStack-Cinder-API-Version header string will be converted to an APIVersionRequest object in the wsgi code. Routing will occur in the usual manner with the version object attached to the request object (which all API methods expect). The API methods can then use this to determine their behaviour to the incoming request.

Types of changes we will need to support:

* Status code changes (success and error codes)
* Allowable body parameters (affects input validation schemas too)
* Allowable url parameters
* General semantic changes
* Data returned in response
* Removal of resources in the API
* Removal of fields in a response object or changing the layout of the response

Note: This list is not meant to be an exhaustive list

Within a controller case, methods can be marked with a decorator to indicate what API versions they implement. For example:

@api_version(min_version='2.0', max_version='2.9')
def show(self, req, id):
   pass

@api_version(min_version='2.17')
def show(self, req, id):
   pass

An incoming request for version 2.2 of the API would end up executing the first method, whilst an incoming request for version 2.17 of the API would result in the second being executed.

For cases where the method implementations are very similar with just minor differences a lot of duplicated code can be avoided by versioning internal methods intead. For example:

@api_version(min_version='2.0')
def _version_specific_func(self, req, arg1):
   pass

@api_version(min_version='2.5')
def _version_specific_func(self, req, arg1):
   pass

def show(self, req, id):
   .... common stuff ....
   self._version_specific_func(req, "foo")
      .... common stuff ....

Reducing the duplicated code minimizes maintenance overhead. So the technique we use would depend on individual circumstances of what code is common/different and where in the method it is.

A version object is passed down to the method attached to the request object so it is also possible to do very specific checks in a method. For example:

def show(self, req, id):
  .... stuff ....

  if req.ver_obj.matches(start_version, end_version):
    .... Do version specific stuff ....

  ....  stuff ....

Note that end_version is optional in which case it will match any version greater than or equal to start_version.

Some prototype code which explains how this work is available here: https://github.com/scottdangelo/TestCinderAPImicroversions

The validation schema decorator would also need to be extended to support versioning

@validation.schema(schema_definition, min_version, max_version)

Note that both min_version and max_version would be optional parameters.

A method, extension, or a field in a request or response can be removed from the API by specifying a max_version:

@api_version(min_version='2.0', max_version='2.9')
def show(self, req, id):
  ....  stuff ....

If a request for version 2.11 is made by a client, the client will receive a 404 as if the method does not exist at all. If the minimum version of the API as whole was brought up to 2.10 then the extension itself could then be removed.

The minimum version of the API as a whole would only be increased by a consensus decision between Cinder developers who have the overhead of maintaining backwards compatibility and deployers and users who want backwards compatibility forever.

Because we have a monotonically increasing version number across the whole of the API rather than versioning individual plugins we will have potential merge conflicts like we currently have with DB migration changesets. Sorry, I don’t believe there is any way around this, but welcome any suggestions!

Client Expectations

As with systems which supports version negotiation, a robust client consuming this API will need to also support some range of versions otherwise that client will not be able to be used in software that talks to multiple clouds.

The concrete example is nodepool in OpenStack Infra. Assume there is a world where it is regularly connecting to 4 public clouds. They are at the following states:

- Cloud A:
  - min_ver: 2.100
  - max_ver: 2.300
- Cloud B:
  - min_ver: 2.200
  - max_ver: 2.450
- Cloud C:
  - min_ver: 2.300
  - max_ver: 2.600
- Cloud D:
  - min_ver: 2.400
  - max_ver: 2.800

No single version of the API is available in all those clouds based on the age of some of them. However within the client SDK certain basic functions like boot will exist, though one might get different additional data based on the version of the API. The client should smooth over these differences when possible.

Realistically this is a problem that exists today, except there is no infrastructure to support creating a solution to solve it.

Alternatives

One alternative is to make all the backwards incompatible changes at once and do a major API release. For example, change the url prefix to /v3 instead of /v2. And then support both implementations for a long period of time. This approach has been difficult in the past and has cause long periods of time before adoption by various users.

Data model impact

None

REST API impact

As described above there would be additional version information added to the GET /. These should be backwards compatible changes.

Otherwise there are no changes unless a client header as described is supplied as part of the request.

Security impact

None

Notifications impact

None

Other end user impact

SDK authors will need to start using the X-OpenStack-Cinder-API-Version header to get access to new features. The fact that new features will only be added in new versions will encourage them to do so.

python-cinderclient is in an identical situation and will need to be updated to support the new header in order to support new API features.

Performance Impact

None

Other deployer impact

None

Developer impact

This will affect how Cinder developers modify the REST API code and add new extensions. This will affect the Volume Manager and the ability to remove locks and instead return VolumeIsBusy for volumes in various -ing states.

Implementation

Assignee(s)

Primary assignee:: Scott DAngelo

Work Items

Port Manila code for api-microversions Status: Done (see https://review.openstack.org/#/c/224910/)
Add examples of increments and decorators (scottda) Status: Done (see https://github.com/scottdangelo/TestCinderAPImicroversions)
Implement changes for python-cinderclient (scottda)
test with python-cinderclient changes (scottda)
Cinder Doc changes (scottda)

Dependencies

Any Cinder spec which makes backwards incompatible changes to the API is dependent on this spec

Testing

It is not feasible for tempest to test all possible combinations of the API supported by microversions. We will have to pick specific versions which are representative of what is implemented. The existing Cinder tempest tests will be used as the baseline for future API version testing.

Documentation Impact

Documents concerning the API will need to reflect these changes. These are begun in the WIP cinder code changes, and will live in cinder/api/openstack/rest_api_version_history.rst

References

http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/kilo/implemented/api-microversions.rst
Manila code for api-microversions: https://review.openstack.org/#/c/207228/8
WIP implementation code: https://review.openstack.org/#/c/224910/
Test Cases and code: https://github.com/scottdangelo/TestCinderAPImicroversions

Cinder Volume Active/Active support - Job Distribution

Wed, 24 Jan 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support

Right now cinder-volume service can run only in Active/Passive HA fashion.

One of the reasons for this is that we have no concept of a cluster of nodes that handle the same storage back-end.

This spec introduces the concept of cluster to Cinder and aims to provide a way for Cinder’s API and Scheduler nodes to distribute jobs to Volume nodes on a High Availability deployment with Active/Active configuration.

Problem description

Right now cinder-volume service only accepts Active/Passive High Availability configurations, and current job distribution mechanism does not allow having multiple services grouped in a cluster where jobs can be queued to be processed by any of those nodes.

Jobs are currently being distributed using a topic based message queue that is identified by the volume_topic, scheduler_topic, or backup_topic prefix joined with the host name and possibly the backend name if it’s a multibackend node like in cinder-volume.localhost@lvm, and that’s the mechanism used to send jobs to the Volume nodes regardless of the physical address of the node that is going to be handling the job, allowing an easier transition on failover.

Chosen solution must be backward compatible as well as allow the new Active/Active configuration to effectively send jobs.

In the Active/Active configuration there can be multiple Volume services - this is not mandatory at all times, as failures may leave us with only 1 active service - with different host configuration values that can interchangeably accept jobs that are handling the same storage backend.

Use Cases

Proposed change

Basic mechanism

To provide a mechanism that will allow us to distribute jobs to a group of nodes we’ll add a new cluster configuration option that will uniquely identify a group of Volume nodes that share the same storage backends and therefore can accept jobs for the same volumes interchangeably.

This new configuration option, unlike the host option, will be allowed to have the same value on multiple volume nodes, with the only requisite that all nodes that share the same value must also share the same storage backends and they must also share the same configurations.

By default cluster configuration option will be undefined, but when a string value is given a new topic queue will be created on the message broker to distribute jobs meant for that cluster in the form of cinder-volume.cluster@backend similar to already existing host topic queue cinder-volume.host@backend.

It is important to notice that cluster configuration option is not a replacement of the host option as both will coexist within the service and must exist for Active-Active configurations.

To be able to determine the topic queue where an RPC caller has to send operations we’ll add cluster_name field to any resource DB table that currently has the host field we are using for non Active/Active configurations. This way we don’t need to check the DB, or keep a cache in memory, to figure out in which cluster is this service included, if it is in a cluster at all.

Once the basic mechanism of receiving RPC calls on the cluster topic queue is in place, operations will be incrementally moved to support Active-Active if the resource is in a cluster, as indicated by the presence of a value in the cluster_name resource field.

The reason behind this progressive approach instead of an all or nothing approach is to reduce the possibility of adding new bugs and facilitating quick fixes by just reverting a specific patch.

This solution makes a clear distinction between independent services and those that belong to a cluster, and the same can be said about resources belonging to a cluster.

To facilitate the inclusion of a service in a cluster, the volume manager will detect when the cluster value has changed from being undefined to having a value and proceed to include all existing resources in the cluster by filling the cluster_name fields.

Having both message queues, one for the cluster and one for the service, could prove useful in the future if we want to add operations that can target specific services within a cluster.

Heartbeats

With Active/Passive configurations a storage backend service is down whenever we don’t have a valid heartbeat from the service and is up if we do. These heartbeats are reported in the DB in services table.

On Active/Active configurations a service is down if there is no valid heartbeat from any of the services that constitute the cluster, and it is up if there is at least one valid heartbeat.

Services will keep reporting their heartbeats in the same way that they are doing it now, and it will be Scheduler’s job to separate between individual and clustered services and aggregate the latter by cluster name.

As explained in REST API impact the API will be able to show cluster information with the status -up or down- of each cluster, based on the services that belong to it.

Disabling

This new mechanism will change the “disabling working unit” from service to cluster for services that are in a cluster. Which means that once all operations that go through the scheduler have been moved to support Active-Active configurations, we won’t be able to disable an individual service belonging to a cluster and we’ll have to disable the cluster itself. For non clustered services, disabling will work as usual.

Disabling a cluster will prevent schedulers from taking that cluster, and therefore all its services, into consideration during filtering and weighting and the service will still be reachable to all operations that don’t go through the scheduler.

It stands to reason that sometimes we’ll need to drain nodes to remove them from a cluster, but this spec and its implementation will not be adding any new mechanism for that. So existing mechanism, using SIGTERM, should be used to perform graceful shutdown of cinder volume services.

Current graceful shutdown mechanism will make sure that no new operations are received from the messaging queue while it waits for ongoing operations to complete before stopping.

It is important to remember that graceful shutdown has a timeout that will forcefully stop operations if they take longer than the configured value. Configuration option is called graceful_shutdown_timeout, goes in [DEFAULT] section and takes a default value of 60 seconds; so this should be configured in our deployments if we think this is not long enough for our use cases.

Capabilities

All Volume services periodically report their capabilities to the schedulers to keep them updated with their stats, that way they can make informed decisions on where to perform operations.

In a similar way to the Service state reporting we need to prevent concurrent access to the data structure when updating this information. Fortunately for us we are storing this information in a Python dictionary on the schedulers, and since we are using an eventlet executor for the RPC server we don’t have to worry about using locks, the inherent behavior of the executor will prevent concurrent access to the dictionary. So no changes are needed there to have exclusive access to the data structure.

Although rare, we could have a consistency problem among volume services where different schedulers would not have the same information for a given backend.

When we had only 1 volume service reporting for each given backend this was not a situation that could happen, since received capabilities report was always the latest and all scheduler services were in sync. But now that we have multiple volume services reporting on the same backend we could receive two reports from different volume services on the same backend and they could be processed in different order on different schedulers, thus making us have different data on each scheduler.

The reason why we can’t assure that all schedulers will have the same capabilities stored in their internal structures is because capabilities reports can be processed in different order on different services. Order is preserved in almost all stages, volume services report in a specific order and message broker preserves this order and they are even delivered in the same order, but when each service processes them we can have greenthreads execution in different order on different scheduler services thus ending up with different data on each service.

This case could probably be ignored since it’s very rare and differences would be small, but in the interest of consistent of the backend capabilities on Scheduler services, we will timestamp the capabilities on the volume services before they are sent to the scheduler, instead of doing it on the scheduler as we are doing now. And then we’ll have schedulers drop any capabilities that are older than the one in the data structure.

By making this change we facilitate new features related to capability reporting, like capability caching. Since capability gathering is usually an expensive operation and in Active-Active configurations we’ll have multiple volume services requesting the same capabilities with the same frequency for the same back-end, we could consider capability caching as solution to decrease the cost of the gathering on the backend.

Alternatives

One alternative to proposed job distribution would be to leave the topic queues as they are and move the job distribution logic to the scheduler.

The scheduler would receive a job and then send it to one of the volume services that belong to the same cluster and is not down.

This method has one problem, and that is that we could be sending a job to a node that is down but whose heartbeat hasn’t expired yet, or one that has gone down before getting the job from the queue. In these cases we would end up with a job that is not being processed by anyone and we would need to either wait for the node to go back up or the scheduler would need to retrieve that message from the queue and send it to another active node.

An alternative to proposed heartbeats is that all services report using cluster@backend instead of host@backend like they are doing now and as long as we have a valid heartbeat we know that the service is up.

There are 2 reasons why I believe that sending independent heartbeats is a superior solution, even if we need to modify the DB tables:

Higher information granularity: We can report not only which services are up/down but also which nodes are up/down.
It will help us on job cleanup of failed nodes that do not come back up. Although cleanup is not part of this spec, it is good to keep it in mind and facilitate it as much as possible.

Another alternative for the job distribution, which was the proposed solution in previous versions of this specification, was to use host configuration option as the equivalent to cluster grouping a new added node configuration option that would serve to identify individual nodes.

Using such solution may lead to misunderstandings with the concept of hosts as clusters, whereas using the cluster concept directly there is no such problem, wouldn’t allow a progressive solution as it was a one shot change, and we couldn’t send messages to individual volume services since we only had the host message topic queue.

There is a series of patches showing the implementation of the node alternative mechanism that can serve as a more detailed explanation.

Another possibility would be to allow disabling individual services within a cluster instead of having to disable the whole cluster, and this is something we can take up after everything else is done. To do this we would use the normal host message queue on the cinder-volume service to receive the enable/disable of the cluster on the manager and that would trigger a start/stop of the cluster topic queue. But this is not trivial, as it requires us to be able to stop and start the client for the cluster topic from the cinder volume manager (it is managed at the service level) and be able to wait for a full stop before we can accept a new enable request to start the message client again.

Data model impact

Final result:

A new clusters table will be added with the following fields:

id: Unique identifier for the cluster
name: Name of the cluster, it is used to build the topic queue in the
same way the host configuration option is used. This comes from the cluster configuration option.
binary: For now it will always be “cinder-volume” but when we add backups
it’ll also accept “cinder-backup”.
disabled: To support disabling clusters.
disabled_reason: Same as in service table.
race_preventer: This field will be used to prevent potential races that
could happen if 2 new services are brought up at the same time and both try to create the cluster entry at the same time.

A cluster_name field will be added to existing services, volumes, and consistencygroups tables.

REST API impact

Service listing will return cluster_name field when requested with the appropriate microversion.

A new clusters endpoint will be added to list -detailed and summarized-, show, and update operations with their respective policies.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Negligible if we implement the aggregation of the heartbeats on a SQL query using exist instead of retrieving all heartbeats and doing the aggregation on the scheduler.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Gorka Eguileor (geguileo)
Other contributors:: Michal Dulko (dulek) Scott DAngelo (scottda) Anyone is welcome to help

Work Items

Add the new clusters table and related operations.
Add Cluster Versioned Object.
Modify job distribution to use new cluster configuration option.
Update service API and add new clusters endpoint.
Update cinder-client to support new endpoint and new field in services.
Move operations to Active-Active.

Dependencies

None

Testing

Unittests for new API behavior.

Documentation Impact

This spec has changes to the API as well as a new configuration option that will need to be documented.

References

None

Report backend state in service list

Thu, 18 Jan 2018 00:00:00

https://blueprints.launchpad.net/cinder/+spec/report-backend-state-in-service-list

Storage driver reports state of backend storage device and let admin operator know it via service list for maintenance purpose.

Problem description

Currently, Cinder couldn’t report backend state to service, operators only know that cinder-volume process is up, but isn’t aware of whether the backend storage device is ok. Users still can create volume and go to fail over and over again. To make maintenance easier, operator could query storage device state via service list and fix the problem more quickly. If device state is down, that means volume creation will fail.

Use Cases

In large scale cloud system, there could be many backends existing in the system. If volume, snapshot or other resources creation goes to failure, operators or cloud management system could query the service first and get the backend device state in every service. If device state is down, specify that storage device has got some problems. Give operators/management system more information to locate bug more quickly.

Proposed change

Each driver reports the backend state in “get_volume_stats” by adding key/value: “backend_state: up/down”[1].
When calling ‘service list’, get this information from scheduler for every backend.
Add ‘backend_state: up/down’ in response body of service list API if context is admin.
Before all drivers support this feature, if the result of get_volume_stats doesn’t include the backend state, Cinder will set backend_state to ‘up’ by default.

Alternatives

Add cinder manage command to query backend device state from driver directly.

Data model impact

None

REST API impact

Add backend_state: up/down into response body of service list and also need a microversions for this feature:

GET /v3/{project_id}/os-services

RESP BODY:

{"services": [{"host": "host@backend1",
               ...,
               "backend_status": "up",
              },
              {"host": "host@backend2",
               ...,
               "backend_status": "down",
              }]
}

Security impact

None

Notifications impact

None.

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Driver maintainer needs to add backend state when reporting volume stats.

Implementation

Assignee(s)

Primary assignee:: wanghao<wanghao749@huawei.com>

Work Items

Implement code in Cinder API and scheduler.
Update cinderclient to support this function.
Add change to API doc.

Dependencies

None

Testing

Both unit and Tempest tests need to be created to cover the code change that mentioned in “Proposed change”.

Documentation Impact

The cinder API documentation will need to be updated to reflect the REST API changes.

References

[1]https://docs.openstack.org/cinder/latest/contributor/drivers.html

Improvement to query consistency group detail

Thu, 21 Dec 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/improvement-to-query-consistency-group-detail

Return the volume id list if they exist in CG when querying CG detail.

Problem description

Currently, querying consistency group detail will not return list of volumes ids that exist in this CG, end users don’t know how many and which volumes are in it. They must query the volume detail one by one to get the CG’s id and form the list of volume ids. It’s very inconvenient to user experience.

Use Cases

After adding volumes to a consistency group, end user wants to know how many and which volumes are in a CG before they decide to make CG snapshot. According to this feature, they can get this information easily by querying consistency group detail, and make further adjustment, like adding more volumes, or remove some volumes from this consistency group.

Proposed change

1.Add volume id list into response body of querying a consistency group detail if user wants those volume’s information.

2.Add group_id query filter to /volumes to get a list of volumes in a CG. For example “cinder list –group-id <uuid of CG>”.

NOTE: Since we’re working on Generic Volume Group[1], to avoid unnecessary code migration later, we will implement the ‘group_id’ query filter first, and after Generic Volume Group is merged, we will implement the #1 way dependent on the Generic Group spec.

Alternatives

None

Data model impact

For performance impact, if we want to add index to consistencygroup_id column, there is data model impact.

REST API impact

Add volume id list into response body of querying a CG detail if specifying the argument ‘list_volume=True’, this will be dependent on the Generic Group spec now, so just leave a example here:

GET /v3/{project_id}/consistencygroups/{consistency_group_id}?list_volume=True

RESP BODY:

{"consistencygroup": {"status": "XXX",
                      "description": "XXX",
                      ...,
                      "volume_list":['volume_id1',
                                     ...,
                                     'volume_idn']
                      }
}

Add a filter “group_id=xxx” in URL of querying volume list/detail:

GET /v3/{project_id}/volumes?group_id=XXX

Security impact

None

Notifications impact

None.

Other end user impact

None

Performance Impact

There is no performance impact if user is NOT using ‘list_volume=True’. We just need to consider to add a additional DB query to get all volume ids which have same CG id. So the performance impact should be small with ‘list_volume=True’. If considering large scale volumes, we can add index to consistencygroup_id column of volume table to reduce performance impact.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: wanghao<wanghao749@huawei.com>

Work Items

Implement code in db query and add list to response body.
Update cinderclient to support this function.
Add change to API doc.

Dependencies

None

Testing

Both unit and Tempest tests need to be created to cover the code change that mentioned in “Proposed change”.

Documentation Impact

The cinder API documentation will need to be updated to reflect the REST API changes.

References

[1]https://review.openstack.org/#/c/303893/

Extending IBMNAS driver to support NAS based GPFS storage deployments

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-gpfs-nas-to-ibmnas

Currently, the ibmnas driver works for an nfs export from Storwize V7000 Unified and SONAS products. It does not have the capability to work with nfs exports provided from a gpfs server.

Problem description

Currently, the ibmnas driver does not have the capability to work with nfs exports provided from a gpfs server.

Lacking this capability will limit the end users from using remote gpfs NAS servers as a backend in OpenStack environment.

Use Cases

Proposed change

Add/Reuse functions in ibmnas.py to support all minimum features listed (github.com/openstack/cinder/blob/master/doc/source/devref/drivers.rst) for NAS based GPFS server backends.

Alternatives

The existing gpfs driver can be extended to support NAS based gpfs storage deployments. But this implementation requires many other new funtions to be introduced, which are already existing and can be reused in ibmnas driver. Apart from this in future, we have planned to support all NFS/GPFS related IBM products via ibmnas driver. Hence extending ibmnas driver is more advantageous than extending gpfs driver.

Data model impact

None

REST API impact

None

Security impact

No specific security issues needs to be considered. Insecure file permissions (OSSN-0014) is fixed in the driver and is addressed by https://review.openstack.org/#/c/101919/

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

This requires an additional option to be configured while deploying OpenStack with IBMNAS products (sonas, v7ku, gpfs-nas).

New configuration option needs to be filled in cinder.conf ibmnas_platform_type = <sonas> | <v7ku> | <gpfs-nas>
This change needs to be explicitly enabled on IBMNAS driver CI certification

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: sasikanth <sasikanth.eda@in.ibm.com>
Other contributors:: nilesh-bhosale <nilesh.bhosale@in.ibm.com>

Work Items

Add/Reuse functions in ibmnas.py to support NAS based GPFS storage deployments.

Dependencies

None

Testing

Unit tests - Existing test_ibmnas.py will be improved to handle the new code changes/functions.
Tempest tests - No additional testcases needs to be written, this feature can be tested with the existing tempest.
Cinder driver certification tests - Driver certification tests will be executed and results will be submitted to the community (as the changes will altogether enable a new storage platform).
CI tests - We are working towards 3rd party CI environment and will continuously run tests across the respective hardware platform.

Documentation Impact

ibmnas driver documentation needs to updated with this new configuration option.

ibmnas_platform_type = <sonas> | <v7ku> | <gpfs-nas>

This option is used for selecting the appropriate backend storage. Valid values are v7ku for using IBM Storwize V7000 Unified sonas for using IBM Scale Out NAS and gpfs-nas for using NAS based GPFS server deployment

References

None

Volume Replication

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/volume-replication

Volume replication is a key storage feature and a requirement for features such as high-availability and disaster recovery of applications running on top of OpenStack clouds. This blueprint is an attempt to add initial support for volume replication in Cinder, and is considered a first take which will include support for: * Replicate volumes (primary to secondary approach) * Promote a secondary to primary (and stop replication) * Re-enable replication * Test that replication is running properly

It is important to note that this is a first pass at volume replication. The process of implementing replication for drivers has uncovered a number of challenges that will be addressed in a future revision of replication that will address the ability to have different replication types and the ability to replicate across multiple backends.

While this blueprint focuses on volume replication, a related blueprint focuses on consistency groups, and replication will be extended to support it.

Use Cases

Problem description

The main use of volume replication is resiliency in presence of failures. Examples of possible failures are:

Storage system failure
Rack(s) level failure
Datacenter level failure

Here we specifically exclude failures like media failures, disk failures, etc. Such failures are typically addressed by local resiliency schemes.

Replication can be implemented in the following ways:

Host-based - Requires Nova integration
Storage-based
- Typical block based approach - replication is specified between two existing volumes (or groups of volumes) on the controllers.
- Typically file system based approach - a file (in Cinder context, the file representing a block device) placed in a directory (or group or fileset, etc) that is automatically copied to a specified remote location.

Assumptions:

Replication should be transparent to the end-user, failover, failback and test will be executed by the cloud admin. However, to test that the application is working, the end-user may be involved, as they will be required to verify that his application is working with the volume replica.
The storage admin will provide the setup and configuration to enable the actual replication between the storage systems. This could be performed at the storage back-end or storage driver level depending on the storage back-end. Specifically, storage drivers are expected to report with whom they can replicate and report this to the scheduler.
The cloud admin will enable the replication feature through the use of volume types.
The end-user will not be directly exposed to the replication feature. Selecting a volume-type will determine if the volume will be replicated, based on the actual extra-spec definition of the volume type (defined by the cloud admin).
Quota management: quota are consumed as 2x as two volumes are created and the consumed space is doubled. We can re-examine this mechanism after we get comments from deployers.

Proposed change

Introduction:

The proposed design provides just a framework in Cinder for backend volume drivers to implement volume replication using the facilities in the storage backend. As such, this spec provides guidance as to how volume replication should be implemented but the actual implementation will vary depending upon the backend in question.

The key to enabling replication starts with adding an extra spec to the volume type to indicate that replication is desired. That extra spec is then used in the volume driver to enable the set-up and control of replication on the storage backend in each of the different functions documented below.

Since Cinder is just providing the framework for backend volume drivers to implement replication, details of replication implementation are left to the backend to implement. The backend driver developer will need to decide for their storage backend the best way to enable replication. For instance one storage provider may feel that implementing synchronous replication is the best choice while another storage provider may choose asynchronous. A provider could also choose to make it a configurable option. Implementing volume replication in Cinder in this manner allows the greatest flexibility to the backend developer to implement replication.

It is also important to note that the developer documentation must provide examples of how this is implemented in the Storwize driver. This is an important item to note as it is not currently possible to demonstrate volume replication in Cinder’s reference implementation, LVM. Therefore developer documentation will have to serve as the reference.

Add extra-specs in the volume type to indicate replication:

capabilities:replication <is> True - if True, the volume is to be replicated, if supported, by the backend driver. If the option is not specified or False, then replication is not enabled. This option is required to enable replication.

Create volume with replication enabled:

Backend drivers that wish to enable replication will need to update their create_volume() function to check for the ‘capabilities:replication <is> True’ extra spec. It is up to the backend driver developers to implement replication in a manner that is compatible with their storage backend.

When a replicated volume is created it is expected that the volume dictionary will be populated as follows:
- volume[‘replication_status’] = ‘copying’
- volume[‘replication_extended_status’] = <driver specific value>
- volume[‘driver_data’] = <driver specific value>
The replica volume is hidden from the end user as the end user will never need to directly interact with the replica volume. Any interaction with the replica happens through the primary volume.

Further details around the dictionary fields above may be seen in the data “Data Model Impact” section below.

Create Volume from Snapshot:

If the volume type extra specs include ‘capabilities:replication <is> True’ for the new volume, the driver needs to create a volume replica at volume creation time and set up replication between the newly created volume and its associated replica. The volume dictionary should be populated in the same manner as create volume.

Create Cloned Volume:

If the volume type extra specs include ‘capabilities:replication <is> True’ for the new volume, the driver needs to create a volume replica at clone creation time and set up replication between the newly created volume and its associated replica. The volume dictionary should be populated in the same manner as create volume.

Create Replica Test Volume:

Create a clone of the replica (secondary) volume. This clone can then be used for testing replication to ensure that fail-over can be executed when necessary. It is important to note that this doesn’t actually execute the the promote path as the intention is not to promote the replica but it gives a method to ensure that the replica contains data and would be useful if it had to be promoted.

The administrator is able to access this functionality using the –source-replica option when creating a volume.

Delete volume:

For volumes with replication enabled the replica needs to be deleted along with the primary copy. So, if a volume type has ‘capabilities:replication <is> True’ set, the driver will need to do the additional deletion.

Get Volume Stats:

If the storage backend driver supports replication the following state should be reported: * replication = True (None or False disables replication)

Re-type volume:

Changing volume-type is the mechanism an admin can use to make an existing volume replicated, or to disable replication for a volume. Change the volume-type of a volume to a volume-type that includes ‘capabilities:replication: <is> True’ (and didn’t have it before) should result in adding a secondary copy to a volume. Change the volume-type of a volume to a volume-type that no longer includes ‘capabilities:replication: <is> True’ should result in removing the secondary copy while preserving the primary copy.

Returns either:
A boolean indicating whether the retype occurred, or A tuple (retyped, model_update) where retyped is a boolean indicating if the retype occurred, and the model_update includes changes for the volume db.

The steps to implement this would look as follows: * Do a diff[‘extra_specs’] and see if ‘replication’ is included.

If replication was enabled for the original volume_type but is not not enabled for the new volume_type, then replication should be disabled.

The replica should be deleted.

The volume dictionary should be updated as follows:

volume[‘replication_status’] = ‘disabled’

volume[‘replication_extended_status’] = None

volume[‘driver_data’] = None

If replication was not enabled for the original volume_type but is enabled for the new volume_type, then replication should be enabled.

A volume replica should be created and the replication should be set up between the volume and the newly created replica.

The volume dictionary should be updated as follows: - volume[‘replication_status’] = ‘copying’ - volume[‘replication_extended_status’] = <driver specific value> - volume[‘driver_data’] = <driver specific value>

Get Replication Status:

This will be used to update the status of replication between the primary and secondary volume.

This function is called by the “_update_replication_relationship_status” function in ‘manager.py’ and is the mechanism to update the status replication between the primary and secondary copies.

The actual state of the replication, as the storage backed is aware of, should be returned and the Cinder database should be updated to reflect the status reported from the storage backend.

It is expected that the following model update for the volume will happen:

volume[‘replication_status’] = <error | copying | active | active-stopped | inactive>

‘error’ if an error occurred with replication.

‘copying’ replication copying data to secondary (inconsistent)

‘active’ replication copying data to secondary (consistent)

‘active-stopped’ replication data copy on hold (consistent)

‘inactive’ if replication data copy is stopped (inconsistent)

volume[‘replication_extended_status’] = <driver specific value>

volume[‘driver_data’] = <driver specific value>

Note for get replication status, that the replication_extended_status and driver_data may not need to be updated.

Promote replica:

Promotion of a replica means that the secondary volume will take over for the primary volume. This can be thought of as a ‘fail over’ operation. Once promotion has happened replication between the two volumes, at the storage level, should be stopped, the replica should be available to be attached and the replication status should be changed to ‘inactive’ if the change is successful, otherwise it should be ‘error’.

A model update for the volume is returned.

As with the functions above, the volume driver is expected to update the volume dictionary as follows:

volume[‘replication_status’] = <error | inactive>

‘error’ if an error occurred with replication.

‘inactive’ if replication data copy on hold (inconsistent)

volume[‘replication_extended_status’] = <driver specific value>

volume[‘driver_data’] = <driver specific value>

Re-enable replication:

Re-enabling replication would be used to fix the replication between the primary and secondary volumes. Replication would need to be re-enabled as part of the fail-back process to make the promoted volume and the old primary volume consistent again.

The volume driver returns a model update to reflect the actions taken.

The backend driver is expected to update the following volume dictionary entries:

volume[‘replication_status’] = <error | copying | active | active-stopped | inactive>

‘error’ if an error occurred with replication.

‘copying’ replication copying data to secondary (inconsistent)

‘active’ replication copying data to secondary (consistent)

‘active-stopped’ replication data copy on hold (consistent)

‘inactive’ if replication data copy is stopped (inconsistent)

volume[‘replication_extended_status’] = <driver specific value>

volume[‘driver_data’] = <driver specific value>

Notes:

The replication_extended_status should be used to store information that the backend driver will need to track replication status. For instance, the Storwize driver, will use the replication_extended_status to track the primary copy status and synchronization status for the primary volume and the copy status, synchronization status and synchronization progress for the replica (secondary) volume.

The driver_data field may be, optionally, used to contain any additional data that the backend driver may require. Some backend drivers may not need to use the driver_data field.

Driver API:

promote: Promotes a replica that is in active or active-stopped state to
be the primary.
reenable: Reenables replication on a volume that is in inactive,
active-stopped or error status.

Alternatives

Replication can be performed outside of Cinder, and OpenStack can be unaware of it. However, this requires vendor specific scripts, and is not visible to the admin user, as only the storage system admin will see the replica and the state of the replication. Also all recovery actions (failover, failback) will require both the the storage and cloud admins to work together. While replication in Cinder reduces the role of the storage admin to only the setup phase, and the cloud admin is responsible for failover and failback with (typically) no need for intervention from the cloud admin.

Data model impact

The volumes table will be updated:

Add replication_status column (string) for indicating the status of replication for a give volume. Possible values are:
- ‘copying’ - Data is being copied between volumes, the secondary is inconsistent.
- ‘disabled’ - Volume replication is disabled.
- ‘error’ - Replication is in error state.
- ‘active’ - Data is being copied to the secondary and the secondary is consistent.
- ‘active-stopped’ - Data is not being copied to the secondary (on hold), the secondary volume is consistent.
- ‘inactive’ - Data is not being copied to the secondary, the secondary copy is inconsistent.
- Add replication_extended_status column to contain details with regards to replication status of the primary and secondary volumes.
- Add replication_driver_data column to contain additional details that may be needed by a vendor’s driver to implement replication on a backend.

State diagram for replication (status)

<start>
                                         any error
                                         condition    +-------+
Create volume   +-----+                +------------> | error |
                      |                               +---+---+
                      |                                   | Storage admin to
                      |                                   | fix, and status
                      |                                   | check will update
                +-----+-----+                             |
+-------------> |  copying  |           any state <-------+
|               +-----+-----+
|                    |
|             status |
|             check  |       status check
|               +----+-----+ +------> +----------------+
|               | active   |          | active-stopped |
|               +----+-----+ <------+ +----------------+
|                    |       status check
|                    |
|                    | promote to primary
|                    |
| re-enable     +----+-----+
+------------+  | inactive |
                +----------+

<end>

REST API impact

Create volume API will have “source-replica” added:

{
    "volume":
    {
        "source-replica": "Volume uuid of primary to clone",
    }
}

Promote volume to be the primary volume
- Promote the secondary copy to be primary. the primary will become secondary and Replication should become inactive.
- Method type: POST
- Normal Response Code: 202
- Expected error http response code(s)
  - 500: Replication is not enabled for volume
  - 500: Replication status for volume must be active or active-stopped, but current status is: <status>
  - 500: Volume status for volume must be available, but current status is: <status>
- V2/<tenant id>/volumes/os-promote-replica/<volume uuid>
- This API has no body
Re-enable replication between the primary and secondary volume.
- Re-enable the replication between the primary and secondary volume. Typically follows a promote operation on the replication.
- Method type: POST
- Normal Response Code: 202
- Expected error http response code(s)
  - 500: Replication is not enabled
  - 500: Replication status for volume must be inactive, active-stopped, or error, but current status is: <status>
- /v2/<tenant id>/volumes/os-reenable-replica/<volume uuid>
- This API has no body

Security impact

Does this change touch sensitive data such as tokens, keys, or user data? No.
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login? No.
Does this change involve cryptography or hashing? No.
Does this change require the use of sudo or any elevated privileges? No.
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer. No.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML. Yes, enabling replication consume cloud and storage resources.

Notifications impact

Will add notification for promoting and re-enabling replication for volumes.

Other end user impact

End-user to use volume types to enable replication.
Cloud admin to use the replication-promote, replication-reenable and create –source-replica commands in the python-cinderclient to execute failover, failback and test.

Performance Impact

Extra db calls identifying if replication exists are added to retype, snapshot operations, etc will add a small latency to these functions.

Other deployer impact

Added options for volume types (see above)
Add new driver capabilities, needs to be supported by the volume drivers, which may imply changes to the driver configuration options.
This change will require explicit enablement (to be used by users) from the cloud administrator.

Developer impact

Change to the driver API is noted above. Third party backends that wish to enable replication will need to add replication support to their driver.
The API will expand to include consistency groups following the merge of consistency group support to Cinder.

Implementation

Assignee(s)

Primary assignee:: ronenkat
Other contributors:: Jay Bryant - E-Mail: jsbryant@us.ibm.com IRC: jungleboyj

Work Items

Cinder public (admin) APIs for replication
DB schema updates for replication
Cinder driver API additions for replication
Cinder manager update for replication
Testing

Dependencies

Related blueprints: Consistency groups https://blueprints.launchpad.net/cinder/+spec/consistency-groups
LVM to support replication using DRBD, in a separate contribution.

Testing

Testing in gate is not supported due to the following considerations:
- LVM has no replication support, to be addressed using DRBD in a separate contribution.
- requires setting up at least two nodes using DRBD
Should be discussed/addressed as support for LVM is added.
3rd party driver CI will be expected to test replication.

Documentation Impact

Public (admin) API changes.
Details how replication is used by leveraging volume types.
Driver docs explaining how replication is setup for each driver.
Provide examples of volume replication implementation for the Storwize backend.

References

Etherpad on improvements needed in documentation:: https://etherpad.openstack.org/p/cinder-replication-redoc

Introduce abstract interface model in volume drivers

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/abc-volume-drivers

Instead of using a loose interface definition of volumes drivers use the ABC python library to build abstract classes that enforces the driver to implement the needed functionality. Goal is to fail fast and not using exceptions during runtime if the driver is not using the correct interface.

Problem description

The defined volume interface in cinder.volume.driver is quite lose. A driver can decide whether to implement a certain functionality or not. From the outside (manager layer) it is not visible which functionality a driver implements. So the only way to discover that is to try to call a function of a feature set and to see if it raises a NotImplementedError.

Use Cases

Proposed change

Build a base VolumeDriver class and subclasses that describe feature sets, like:

                         +-------------------------+
        +----------------+     BaseVolumeDriver    +---------------+
        |                |       {abstract}        |               |
        |                +-----------^-------------+               |
        |                            |                             |
        |                            |                             |
+-------+-------------+  +-----------+-------------+  +------------+---------+
| VolumeDriverBackup  |  |   VolumeDriverSnapshot  |  |  VolumeDriverImport  |
|     {abstract}      |  |      {abstract}         |  |      {abstract}      |
+---------------------+  +-------------------------+  +----------------------+

If a driver implements the backup functionality and supports volume import it should inherit from the interface classes like:

class FooDriver(VolumeDriverBackup, VolumeDriverImport):
    ...

The management layer can observe the feature set’s of a given driver by using isinstanceof():

volume_driver = FooDriver(..)
if isinstanceof(volume_driver, VolumeDriverBackup):
    pass

Usage of python ABC is preferable since it gives a variety of advantages (see [1], [2]). It will fail on instantiation level with an TypeExcetion. Other OpenStack project already using this library (see [3]).

Driver Migration

Instead of changing all drivers in one big step it better to migrate stepwise. In order to do so the VolumeDriver class can exist with the same interface and use NotImplementedError exceptions as before. With that all existing drivers can be migrated to the new concept one after another.

Alternatives

Only implement subclasses and don’t use ABC

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

There is no heavy performance implication expected due to the reason ABCMeta and it’s functionality is only very limited used inside of the relevant data path. The following general object function are potential less performant than before (see [5]):

- __new__()
- __subclasscheck__(), issubclass()
- __instancecheck__(), isinstance()

The performance is depending on the depths of used class hierarchy. The proposed concept is quite plain in that aspect (maximum 2 levels). In general it’s a comparison between the used cycles for raising/catching an exception and iteration in a for loop over all abstract classes.

Other deployer impact

None.

Developer impact

This change will change all implemented drivers slightly. The functionality itself shouldn’t be changed at all but all driver need to be adopted to the new class model.

Implementation

Assignee(s)

Primary assignee:: Marc Koderer (m-koderer)
Other contributors:: Danny Al-Gaaf (danny-al-gaaf)

Work Items

Will be tracked in etherpad.

Dependencies

None.

Testing

Unit tests need to be adapted massively since there are catching NotImplementedError exceptions all over the place.

Documentation Impact

None.

References

[1]: http://legacy.python.org/dev/peps/pep-3119/ [2]: http://dbader.org/blog/abstract-base-classes-in-python [3]: http://lists.openstack.org/pipermail/openstack-dev/2013-August/014089.html [4]: https://bugs.launchpad.net/tempest/+bug/1346797 [5]: https://hg.python.org/cpython/file/2.7/Lib/abc.py

Add support for chiscsi iscsi helper

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/chiscsi-iscsi-helper

The Chelsio iSCSI target(chiscsi) serves as a drop in replacement for the IET target, aiming to provide the same functionality as IET. This spec aims at adding support for said target implementation as a pure iscsi_helper.

chiscsi supports offloading of iSCSI PDU’s when required hardware (supported Chelsio Network cards) is available but will work on any regular NIC as well. Offloading or lack thereof requires no user intervention once target drivers are installed. Implementation is initiator agnostic as well, no changes needed on initiator side.

Problem description

chiscsi target is not currently supported by openstack

For a Deployer trying to use offloaded iSCSI support on target side, no option is currently available.
Manual intervention is currently required to export volumes, as cinder does not understand chiscsi target implementation.

Use Cases

Proposed change

Add one more iscsi_helper option to cover chiscsi, the driver for this will interact with the chiscsi target implementation to provide same functionality as iet.

Use of offloading is dependent on required hardware being present but is completely optional. No intervention is required to enable offload and offloading will happen in a manner completely transparent to initiator side.

No initiator side changes are required to make use of chiscsi, with or without offload support. No extra configuration options are required.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

If iSCSI offload is available, there is a significant performance boost to be gained. If offloading is not used, performance and resource usage would be roughly on par with IET or better.

Other deployer impact

No new config options are required besides an extra allowed value for ‘iscsi_helper’ that would need to be explicitly set to ‘chiscsi’.
chiscsi target needs to be installed before it can be used.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: anish7
Other contributors:: kxie

Work Items

Using iet helper as base, create iscsi_helper for chiscsi, with equivalent commands for all required apis

Dependencies

Ability to use chiscsi target obviously depends on target driver being installed, and command utility available on path. No other dependencies

Testing

Current test for IET target should work just fine for chiscsi

Documentation Impact

None except listing chiscsi as an available iscsi_helper

References

http://www.chelsio.com/iscsi-target-software/

Consistency Groups Kilo Update

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/consistency-groups-kilo-update

Consistency Groups support was introduced in Juno. This proposal is to enhance it by adding a few new features.

Problem description

Create CG from CG snapshot

Currently a user can create a Consistency Group and create a snapshot of a Consistency Group. To restore from a Cgsnapshot, however, the following steps need to be performed:
1. Create a new Consistency Group.
2. Do the following for every volume in the Consistency group:
  1. Call “create volume from snapshot” for snapshot associated with every volume in the original Consistency Group.
There’s no single API that allows a user to create a Consistency Group from a Cgsnapshot.
Modify CG After a CG is created and volumes are created and added to CG, you can delete the entire CG with all volumes in it, but there’s no API to add an existing volume to the CG or remove a volume from it.
DB Schema Changes Volume types are required when creating a Consistency Group. Currently volume types are stored in a field in the consistencygroups table in the Cinder database. There’s a limitation to this approach, however, because the size of the column is fixed.

Use Cases

Proposed change

Create CG from CG snapshot
- Add an API that allows a user to create a Consistency Group from a Cgsnapshot.
- Add a Volume Driver API accordingly.
Modify Consistency Group
- Add an API that adds existing volumes to CG and removing volumes from CG after it is created.
- Add a Volume Driver API accordingly.
DB Schema Changes

The following changes are proposed:
- A new cg_volumetypes table will be created.
- This new table will contain 3 columns:
  - uuid of a cg_volumetype entry
  - uuid of a consistencygroup
  - uuid of a volume type
- Upgrade and downgrade functions will be provided for db migrations.

Alternatives

Without these propsed changes, we have to deal with the current limitations.

Data model impact

DB Schema Changes The following changes are proposed: * A new cg_volumetypes table will be created. * This new table will contain 3 columns:
- uuid of a cg_volumetype entry
- uuid of a consistencygroup
- uuid of a volume type

REST API impact

New Consistency Group APIs changes

Create Consistency Group from Cgsnapshot * V2/<tenant id>/consistencygroups * Method: POST * JSON schema definition for V2:
```
{
    "consistencygroup":
    {
        "name": "my_cg",
        "description": "My consistency group",
        "cgsnapshot": my_cgsnapshot,
    }
}
```
- In the Create Consistency Group API, if cgsnapshot is not specified, the code path stays the same as before and the request goes to the scheduler; If cgsnapshot is specified, the request will be sent to the backend where the original Consistency Group resides.
- Cinder API will be responsible for creating the consistencygroup entry and volume entries in the database. Cinder driver will be responsible for creating it in the backend.
Update Consistency Group * V2/<tenant id>/consistencygroups/<cg uuid> * Method: PUT * JSON schema definition for V2:
```
{
    "consistencygroup":
    {
        "name": "my_cg",
        "description": "My consistency group",
        "addvolumes": [volume uuid 1, volume uuid 2,...]
        "removevolumes": [volume uuid 8, volume uuid 9,...]
    }
}
```
- This method can update name, description, as well as volumes in the consistency group. The list after “addvolumes” will contain UUIDs of volumes to be added to the group and the list after “removevolumes” will contain UUIDs of volumes to be removed from the group. The API will validate the input name, description, UUIDs in addvolumes and removevolumes fields against the information in Cinder db and send the request to the volume manager. Manager will call driver to do the update on the backend. The API will update Cinder db.

Cinder Volume Driver API

The following new volume driver APIs will be added:

def create_consistencygroup_from_cgsnapshot(self, context,
    consistencygroup, volumes, cgsnapshot, snapshots)

def modify_consistencygroup(self, context, consistencygroup,
    old_volumes, new_volumes)

Security impact

Notifications impact

Other end user impact

python-cinderclient needs to be changed to support the new APIs.

Create CG from CG snapshot

cinder consisgroup-create --name <name> --description <description>
--cgsnapshot <cgsnapshot uuid or name>

Modify CG

cinder consisgroup-modify <cg uuid or name> --name <new name>
--description <new description> --addvolumes
<volume uuid> [<volume uuid> ...] --removevolumes
<volume uuid> [<volume uuid> ...]

Performance Impact

Other deployer impact

None. The db schema changes are internal and should be transparent to end users.

Developer impact

Driver developers can implement the new driver APIs.

Implementation

Assignee(s)

Primary assignee:: xing-yang

Other contributors:

Work Items

API changes:
- Create CG from CG snapshot API
- Modify CG API
Volume Driver API changes:
- Create CG from CG snapshot
- Modify CG
DB schema changes

Dependencies

Testing

New unit tests will be added to test the changed code.

Documentation Impact

Documentation changes are needed.

References

Failover to alternative iSCSI portals on login failure

Thu, 16 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/iscsi-alternative-portal

This spec proposes to add fail-over feature to alternative iSCSI portals on attaching iSCSI volumes when the main iSCSI portal is unaccessible.

Problem description

When the main iSCSI portal is unreachable by network failure etc., volume attach/detach will fail even though the other portal addresses is reachable. To enable nova-compute to fall-back to alternative portal addresses, cinder should tell the alternative portal addresses to nova.

Use Cases

Proposed change

Add optional parameters ‘target_alternative_portals’, ‘target_alternative_iqns’, and ‘target_alternative_luns’ into ‘connection_info’ returned by os-initialize_connection volume action API that represents a list of alternative addresses of the iSCSI portals and the corresponding iqn and lun to each portal. (iqns and luns can be the same when the addresses are pointing the same portal.)

When nova-compute recognizes these parameters, it will fall-back to alternative portal address/target/lun specified by the parameters on failure during discovery. Old nova-compute just ignores the new parameters, so it works without changes if the primary portal address is alive.

Note that this proposal is for single iSCSI data path use-case (fail-over). This means, if the primary portal and target is failed, the other LUNs already attached on the path will be unaccessible until the path is recovered manually. Multiple data path (multipath) use-case is covered with another spec.

This change must be applied also to cinder/brick/initiator.

Alternatives

None

Data model impact

Backend drivers, if they need, may put multiple portal addresses and iqns into ‘provider_location’ field of volume tables.

For example, the LVM iSCSI driver will put the addresses in the following format in order to pass them as a volume dict to target admin helpers:

provider_location = ‘<portal ip>:<port>;<portal ip 2>:<port>,<portal> <iqn>’

e.g.

provider_location = ‘10.0.1.2:3260;10.0.2.2:3260,0 iqn.2010-10.org.openstack:
volume-12345678-abcd-1234-5678-12345678abcd’

(Note that ‘volume-12345678…’ is a name of target, not a volume; they are the same in LVM iSCSI driver.)

Other backend drivers can add multiple portal addresses and iqns into cinder.conf, or can retrieve them dynamically from the array.

REST API impact

‘os-initialize_connection’ volume action API response may have additional parameters ‘target_alternative_portals’ and ‘target_alternative_iqns’, which contain a list of portal IP address:port pairs, a list of alternative iqn’s and lun’s corresponding to each portal address. For example:

{"connection_info": {"driver_volume_type": "iscsi", ...
                     "data": {"target_portal": "10.0.1.2:3260",
                              "target_alternative_portals": [
                                               "10.0.2.2:3260",
                                               "10.0.3.2:3260"],
                              "target_iqn": "iqn.2014-2.org.example:vol1-1",
                              "target_alternative_iqns": [
                                            "iqn.2014-2.org.example:vol1-2",
                                            "iqn.2014-2.org.example:vol1-3"],
                              "target_lun": 1,
                              "target_alternative_luns": [2, 3],
                              ...}}}

In this case, “iqn.2014-2.org.example:vol1-1”, lun=1 is for portal “10.0.1.2:3260”, “iqn.2014-2.org.example:vol1-2”, lun=2 is for portal “10.0.2.2:3260”, and “iqn.2014-2.org.example:vol1-3”, lun=3 is for portal “10.0.3.2:3260”.

The portals may have the same targets and/or luns in some backends.

Backend drivers are responsible to specify the order of alternative portals. Nova and Cinder brick library should first try connecting to the main portal and target. If failed, it should try connecting to the alternative portals and targets in the order of the list provided. For example, if a backend driver wants to round-robin portals for each LUN, the driver may return different portal addresses as main addresses for LUNs in the same target.

Security impact

None

Note that if any CHAP credentials are provided in connection_info, they must be applied on the all target_portals.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Backend driver may have additional settings to enable alternative iSCSI portals. For example, to utilize this feature in iSCSI LVM driver, we needs to specify a list of alternative IP addresses of the cinder-volume node where iSCSI targets run on.

Developer impact

To enable multiple iSCSI portals functionality, backend drivers must change the implementation of initialize_connection method to return the additional parameters ‘target_alternative_portals’, ‘target_alternative_iqns’ and ‘target_alternative_luns’.

Implementation

Assignee(s)

Primary assignee:: tsekiyama

Work Items

Implement this feature in LVM iSCSI driver as a sample
Modify Nova and Cinder brick library to fail-over to alternative portals

Dependencies

None

Testing

Unit tests should be added for drivers which support this feature, so that initialize_connection will return correct connection_info.
To test this feature in tempest, multiple addresses must be asigned to the test environment in order to access alternative portal addresses. Implementation in LVM iSCSI driver would be useful for testing.

Documentation Impact

A section to describe this feature should be added.

If the driver needs additional settings for this feature, the documentation for them should be added.

References

nova-specs: Failover to alternative iSCSI portals on login failure https://review.openstack.org/#/c/137468/
cinder-specs: Enhance iSCSI multipath support (multipath use-case) https://review.openstack.org/#/c/136500/

Enhance iSCSI multipath support

Thu, 16 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/enhance-iscsi-multipath

This spec proposes to enhance iSCSI multipath support by defining the way to tell multiple iSCSI portals/iqns/luns to access a volume (LU) to Nova.

Problem description

Currently, nova-compute supports multipath for iSCSI volume data path. It depends on response to targets discovery of from the main iSCSI portal, expecting multiple portal addresses are contained.

However, some arrays only respond to discovery with a single portal address, even if secondary portals are available. In this case, nova-compute cannot know secondary portals and corresponding iSCSI target IQN, so nova-compute cannot establish multiple sessions for the target(s). To enable nova-compute to login to secondary portals, cinder should tell the secondary portal addresses and corresponding target iqns/luns.

Telling secondary portal addresses and iqns/luns is also useful for arrays which can respond to discovery with multiple portals addresses and IQNs, since compute can access to the volume via secondary portals even when the main portal is unaccessible due to network trouble. (Note that compute should attach the volume via multipath device “dm-X” so that the session to the main portal can be re-established when the network is recovered.)

Use Cases

Proposed change

If nova-compute can support multipath for iSCSI (that is, if multipathd is installed and CONF.libvirt.iscsi_use_multipath is set True), nova-compute should call Cinder initialize_connection API with a new ‘multipath=True’ optional key in ‘connector’ dictionary. This parameter is passed to the backend driver so that the driver can export a volume to multiple ports.

When the ‘multipath=True’ is specified, ‘connection_info’ of the API response will contain multiple portal addresses and corresponding target iqns/luns. Current ‘target_portal’/’target_iqn’/’target_lun’ parameters will be replaced with ‘target_portals’/’target_iqns’/’target_luns’ which have a list of the addresses of the iSCSI portals and the corresponding iqn/lun to each portal. When nova-compute recognizes these parameters, it will log into every portal address/target/lun. When the backend doesn’t support (or not configured to support) multipath, the driver may return current ‘target_portal’/ ‘target_iqn’/’target_lun’ set.

When the ‘multipath=False’ is specified or the key is omitted, the API must return ‘target_portal’/’target_iqn’/’target_lun’ set.

This change must be applied also to cinder/brick/initiator.

Alternatives

None

Data model impact

None

REST API impact

‘os-initialize_connection’ volume action API will take new optional parameter ‘multipath’ in the connector argument, that means the connector can handle multipath to iSCSI volumes.

When the ‘multipath=True’ is specified, ‘os-initialize_connection’ volume action API respond with ‘target_portals’/’target_iqns’/’target_luns’ in ‘connection_info’ which contain a list of portal IP address:port pairs, a list of secondary iqn’s and lun’s corresponding to each portal address.

For example:

{"connection_info": {"driver_volume_type": "iscsi", ...
                     "data": {"target_portals": ["10.0.1.2:3260",
                                                 "10.0.2.2:3260"],
                              "target_iqns": [
                                            "iqn.2014-2.org.example:vol1-1",
                                            "iqn.2014-2.org.example:vol1-2"],
                              "target_luns": [1, 2],
                              ...}}}

In this case, “iqn.2014-2.org.example:vol1-1”, lun=1 is for portal “10.0.1.2:3260”, and “iqn.2014-2.org.example:vol1-2”, lun=2 is for portal “10.0.2.2:3260”.

Security impact

None

Note that if any CHAP credentials are provided in connection_info, they must be applied on the all target_portals.

Notifications impact

None

Other end user impact

None

Performance Impact

Since nova-compute can omit discovery query to find multiple portals and targets, this change may make volume-attach/detach operations faster.

Other deployer impact

Backend driver may have additional settings to enable iSCSI portal multipath. For example, to utilize this feature in iSCSI LVM driver, we needs to specify a list of secondary IP addresses of the cinder-volume node where iSCSI targets run on.

Developer impact

If drivers want to use iSCSI multipath, and the connector has the ‘multipath=True’ value, then drivers should change to return ‘target_portals’, ‘target_iqns’ and ‘target_luns’.

Existing drivers that support target_portal discovery via the existing process won’t need to change unless they want to honor the ‘multipath=True’ connector entry when exporting a volume.

Existing drivers which support multipath with the existing design must work even if they don’t support this change. Such drivers will return single ‘target_portal’/’target_iqn’/’target_lun’ even if ‘multipath=True’ is specified. Then the connector must send discovery query to the returned portal in order to find the multipath portals and targets, like the existing design.

Implementation

Assignee(s)

Primary assignee:: tsekiyama

Work Items

Implement this feature in LVM iSCSI driver as a sample
Enable nova and brick library to handle multiple portals/iqns/luns.

Dependencies

None

Testing

Unit tests should be added for drivers which support this feature, so that initialize_connection will return correct connection_info.
To test this feature in tempest, multiple addresses must be asigned to the test environment in order to establish multiple sessions to volumes. Implementation in LVM iSCSI driver would be useful for testing.

Documentation Impact

A section to describe this feature should be added.

If the driver needs additional settings for this feature, the documentation for them should be added.

References

Enable multipath for libvirt iSCSI Volume Driver (merged) https://review.openstack.org/#/c/17946/
Failover to alternative iSCSI portals on login failure (for single path) https://review.openstack.org/#/c/131502/
Nova-specs: Support iSCSI portal multipath https://review.openstack.org/#/c/134299/

Limit Volume Copy Bandwidth Per Backend

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/limit-volume-copy-bps-per-backend

Currently, cinder has a global configuration to limit a volume copy bandwidth to mitigate slow down of instances volume access during large image copies. ( https://blueprints.launchpad.net/cinder/+spec/limit-volume-copy-bandwidth )

This spec proposes to make this config variable per backend.

Problem description

The current global config has some issues.

When multiple backends exist, we cannot assign different bandwidth limit. For instance, it is desirable to allow large bps for SSD than for HDD.
If volume copy runs concurrently, bandwidth larger than the bps limit is consumed. From the viewpoint of QoS (to keep bandwidth for access from instances), we should limit total volume copy bandwidth per backend.

Use Cases

Proposed change

Make the cinder.conf parameter ‘volume_copy_bps_limit’ writable in each backend section. (If volume_copy_bps_limit is specified in DEFAULT section as existing design, that value is used as a global default. This keeps backwards compatibility.)

The bps limit is divided if multiple volume copy operations run concurrently. For example, if volume_copy_bps_limit is set to 100MB/s, and 2 copies are running at the same time in a backend, each copy can use up to 50MB/s. If one of the copies finished, the other copy’s limit is updated to 100MB/s.

On initialization of VolumeDriver class, a ‘Throttle’ class object is created and stored in ‘_throttle’ member for each backend driver object, that manages the total bandwidth allowed and volume copy operations in flight.

It should be passed to copy_volume() as following, so that it can set/update the bandwidth limit.

volume_utils.copy_volume(…, throttle=self._throttle)

In copy_volume method, Throttle class method is called to setup cgroups (or some other QoS feature, when the Throttle class is overridden).

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

When volume copy I/O bandwidth is limited, it takes more time to complete volume copy. Users are required to balance between volume copy performance and interference to instances performance.

Other deployer impact

This feature is disabled by default. Users who want to use this feature need to set ‘volume_copy_bps_limit’ in cinder.conf.

Developer impact

A ‘_throttle’ member is added to VolumeDriver class, which should be passed to volume_utils.copy_volume() and related function. Otherwise, bandwidth limit will be ignored.
In Linux environments, ‘_throttle’ object will be implemented using cgroups. Drivers for non-Linux environments (e.g. Windows) can override the initialization of ‘_throttle’ to use another method for throttling.

Implementation

Assignee(s)

Primary assignee:: tsekiyama (tomoki.sekiyama@hds.com)

Work Items

Use ‘backend_driver.configuration.volume_copy_bps_limit’ etc. if ‘CONF.volume_copy_bps_limit’ is not specified.
Replace current setup_blkio_cgroup() with new ‘Throttle’ class
Pass the Throttle object from the backend drivers to copy_volume()

Dependencies

None

Testing

With volume_copy_bps_limit = 100MB/s for a fake backend driver:

start a volume copy, then the bps limit is set to 100MB/s

start a second volume copy, then the limit is updated to 50MB/s for both

finish one of the copies, then the limit is resumed to 100MB/s

Documentation Impact

The cinder client documentation about volume_copy_bps_limit in cinder.conf should address that copy bps limit can be specified for each backend driver by writing this config in the backend section.

References

None

Adopt Guru Meditation report

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/guru-meditation-report

This spec proposes adoption for Oslo Guru Meditation reports in Cinder. The new feature will enhance debugging capabilities of all official Cinder services, by providing an easy way to gather runtime data about current threads and configuration, among other things, to developers, operators, and tech support.

Problem description

Currently Cinder does not have a way to gather runtime data from active service processes. The only information that is available to deployers, developers, and tech support to analyze is what was actually logged by the service in question. Additional data could be usefully used to debug and solve problems that occur during Cinder operation. Among other things, we could be interested in stack traces of all (both green and real) threads, pid/ppid info, package version, configuration as seen by the service, etc.

Oslo Guru reports provide an easy way to add support for gathering this kind of information to any service. Report generation is triggered by sending a special (USR1) signal to a service. Reports are generated on stderr, that can be piped into system log, if needed.

Use Cases

Guru reports are extensible, meaning that we will be able to add more information to those reports in case we see it needed.

Guru reports has been supported by Nova.

Proposed change

First, a new oslo-incubator module (reports.*) should be synchronized into cinder tree. Then, each service entry point should be extended to register reporting functionality before proceeding to its real main().

Data model impact

None.

REST API impact

None.

Security impact

In theory, the change could expose service internals to someone who is able to send the needed signal to a service. That said, we can probably assume that the user is already authorized to achieve a lot more than just having an access to stack traces and configuration used. Also, if deployers are afraid of the information leak for some reason, they could also make sure their stderr output is channeled into safe place.

Because this report is triggered by user, there is no need to add config option to turn on/off this feature.

Notifications impact

None.

Other end user impact

None.

Performance Impact

The feature does not require any additional resources until it’s triggered by the user. Default report generation is not expected to take too long. Report extensions will need to be assessed on case by case basis. In any case, reports are not expected to be generated too often and are assumed to be debugging capability, not something to trigger once per minute just in case.

IPv6 Impact

None.

Other deployer impact

Deployers may be interested in making sure those reports are collected somewhere (e.g. stderr should be captured by syslog).

Developer impact

None.

Community Impact

None.

Alternatives

We could reimplement the wheel, but we hopefully won’t.

Implementation

Assignee(s)

wanghao

Work Items

sync reports.* module from oslo-incubator
adopt it in all cinder services using a wrapper located under cinder/cmd/…

Dependencies

reports.* module is currently going thru graduation consideration. In case it’s graduated into oslo.reports library before cinder switches to it, we won’t actually need to sync any code from oslo-incubator but instead add a new external oslo dependency. If Cinder switches to the module before graduation is complete, then we’ll need to adopt oslo.reports later as part of usual oslo liaison effort.

Testing

Automated tests are needed to ensure the guru report is working correctly.

Documentation Impact

User documentation will need to be updated to introduce this changes.

Developer documentation should be updated to include information on how to add support for the reporting feature.

References

oslo-incubator module: http://git.openstack.org/cgit/openstack/oslo-incubator/tree/openstack/common/report
blog about nova guru reports: https://www.berrange.com/posts/2015/02/19/nova-and-its-use-of-olso-incubator-guru-meditation-reports/
oslo.reports repo: https://github.com/directxman12/oslo.reports

Cinder internal tenant

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-internal-tenant

This spec proposes the addition of config options and internal plumbing required for a cinder-internal tenant. This tenant can then be used by Cinder as the owner of internal objects that we don’t really want owned by any normal users.

Problem description

There are several use cases for ‘internal’ cinder objects where we would like to prevent a normal user from seeing them. The approach that seems to have the most backing is to a special tenant that cinder can use for owning these internal volumes/snapshots/etc.

Use Cases

There are a few different use cases for the internal tenant. The generic use case is for any volume or snapshot object in Cinder that we do not want exposed to normal users, but would like to keep track of and have treated as normal volumes and snapshots.

There are a few features which would be able to take advantage of this:

Image Caching - Volumes (and potentially snapshots) that are used as part of the image cache could be owned by the tenant.
Volume Migration - Temporary volumes while migrations are taking place could be owned by the cinder-internal tenant.
Non-Disruptive Backups - Temporary snapshots and volumes could be owned by the cinder-internal tenant.

Proposed change

I think initially the easiest way to make this work is to add new config options that take in the credentials for the tenant. This would require an admin to create and configure the tenant manually and then set the values in cinder.conf. These config options will be:

cinder_internal_tenant_username
cinder_internal_tenant_password

These config options will default to ‘None’. For some features like the image cache this isn’t such a big deal. If the credentials are none, and the tenant is unavailable the cache just won’t actually create cache objects and things fall back to the original functionality. It will however be more difficult for features like migration or backups where they may not be able to fall back nicely to a behavior that does not need the credentials.

Things like quota and other permissions would be managed by an admin only, cinder will not modify things automatically. In the future it would be possible to have default values for the config options, and allow cinder to mange the tenant. That functionality will be out of scope for this initial implementation.

For the initial change I don’t think we should keep any of this info in the cinder db. Everything comes from the config file and is re-checked at startup.

Any usage of the tenants objects should assume they could be deleted or cleaned up at any time, I don’t think this is any different than the case today where a user could delete a second volume halfway through migration though. So this is not a regression. The idea behind this being that as an admin you could periodically clean these objects up if they start to accumulate, but you wouldn’t really need to worry about which ones are ‘safe’.

Alternatives

Another way to approach this problem is to add ‘hidden’ flags to volumes and snapshots. The biggest downside of this is that all API’s need to then know about this and start filtering, and potentially clients and commands need new ‘–hidden’ or ‘–show-all’ kind of flags. In addition these hidden volumes may either take quota from a real user or not be accounted for at all.

Data model impact

None

REST API impact

None

Security impact

Care will need to be taken to ensure the tenant credentials are not exposed to anyone other than cloud administrators and that users cannot create volumes or other objects as the internal tenant.

Notifications impact

None

Other end user impact

This change by itself shouldn’t have any end user impact. Things built on top of it may change what a user sees while using commands like migrate.

Performance Impact

None

Other deployer impact

New config options and setup steps when configuring cinder.

Additional documentation to read.

Developer impact

New internal API’s to use to create objects as the internal tenant.

Implementation

Assignee(s)

Primary assignee:: patrick-east
Other contributors:: None

Work Items

Add in support for the new config options.
Add in utility functions to get the tenant information and anything else required to use the internal tenant for operations.
Possibly add support in to DevStack to configure this by default for future development on top of the feature.
Documentation on how to create and configure the tenant.

Dependencies

None

Testing

Unit tests for the utility functions.

This change by itself won’t require tempest test changes or additions, but features utilizing it will.

Documentation Impact

New instructions for how to create and configure the tenant in the installation instructions.
http://docs.openstack.org/trunk/config-reference/content/
http://docs.openstack.org/user-guide-admin/

References

http://eavesdrop.openstack.org/meetings/cinder/2015

/cinder.2015-05-27-16.00.log.html
https://blueprints.launchpad.net/cinder/+spec/image-volume-cache

Cinder Nested Quota Driver

Thu, 16 Nov 2017 00:00:00

bp cinder-nested-quota-driver

Nested quota driver will enable OpenStack to enforce quota in nested projects.

bp hierarchical-multitenancy

Problem description

OpenStack is moving towards support for hierarchical ownership of projects. In this regard, Keystone will change the organizational structure of OpenStack, creating nested projects.

The existing Quota Driver in Cinder called DbQuotaDriver is useful to enforce quotas at both the project and the project-user level provided that all the projects are at the same level (i.e. hierarchy level cannot be greater than 1).

The proposal is to develop a new Quota Driver called NestedQuotaDriver, by extending the existing DbQuotaDriver which will allow enforcing quotas in nested projects in OpenStack. The nested projects are having a hierarchical structure, where each project may contain users and projects (can be called sub-projects).

Users can have different roles inside each project: A normal user can make use of resources of a project. A project-admin, for example can be a user who in addition is allowed to create sub-projects, assign quota on resources to these sub-projects and assign the sub-project admin role to individual users of the sub-projects. Resource quotas of the root project can only be set by the admin of the root project. The user roles can be set as inherited, and if set, then an admin of a project is automatically an admin of all the projects in the tree below.

Use Cases

Actors

Mike - Cloud Admin (i.e. role:cloud-admin) of ProductionIT
Jay - Manager (i.e. role: project-admin) of Project CMS
John - Manager (i.e. role: project-admin) of Project ATLAS
Eric - Manager (i.e. role: project-admin) of Project Operations
Xing - Manager (i.e. role: project-admin) of Project Services
Walter - Manager (i.e. role: project-admin) of Project Computing
Duncan - Manager (i.e. role: project-admin) of Project Visualisation

The nested structure of the projects is as follows.

{
   ProductionIT: {
                  CMS  : {
                            Computing,
                            Visualisation
                        },
                  ATLAS: {
                            Operations,
                            Services
                      }
               }
 }

Mike is an infrastructure provider and offers cloud services to Jay for Project CMS, and John for Project ATLAS. CMS has two sub projects below it named, Visualisation and Computing, managed by Duncan and Walter respectively. ATLAS has two sub projects called Services and Operations, managed by Xing and Eric respectively.

Mike needs to be able to set the quotas for both CMS and ATLAS, and also manage quotas across the entire projects including the root project, ProductionIT.
Jay should be able to set and update the quota of Visualisation and Computing.
Jay should be able to able to view the quota of CMS, Visualisation and Computing.
Jay should not be able to update the quota of CMS, although he is the Manager of it. Only Mike can do that.
Jay should not be able to view the quota of ATLAS. Only John and Mike can do that.
Duncan, the Manager of Visualisation should not be able to see the quota of CMS. Duncan should be able to see the quota of Visualisation only, and also the quota of any sub projects that will be created under Visualisation.
The total resources consumed by a project is divided into
1. hard_limit - Maximum capacity that can be consumed.
2. used - Resources used by the resource “volumes” in a project.
  (excluding child-projects)
3. reserved - Resources reserved for future use by the project does
  not include descendants.
The quota information regarding number of volumes in different projects are as follows,

Name

hard_limit

used

reserved

ProductionIT

1000

100

100

CMS

300

25

15

Computing

100

50

50

Visualisation

150

25

25

ATLAS

400

25

25

Services

100

25

25

Computing

200

50

50

Consider following use-cases :-

Suppose, Mike(admin of root project or cloud admin) increases the hard_limit of volumes in CMS to 400

Suppose, Mike increases the hard_limit of volumes in CMS to 500

Suppose, Mike delete the quota of CMS

Suppose, Mike reduces the hard_limit of volumes in CMS to 350

Suppose, Mike reduces the hard_limit of volumes in CMS to 200

Suppose, Jay(Manager of CMS)increases the hard_limit of volumes in CMS to 400

Suppose, Jay tries to view the quota of ATLAS

Suppose, Duncan tries to reduce the hard_limit of volumes in CMS to 400.

Suppose, Mike tries to increase the hard_limit of volumes in ProductionIT to 2000.

Suppose, Mike deletes the quota of Visualisation.

Suppose, Mike deletes the project Visualisation.

Name	`hard_limit`	`used`	`reserved`
ProductionIT	1000	100	100
CMS	300	25	15
Computing	100	50	50
Visualisation	150	25	25
ATLAS	400	25	25
Services	100	25	25
Computing	200	50	50

Suppose the company doesn’t want a nested structure and want to restructure in such a way that there are only four projects namely, Visualisation, Computing, Services and Operations.

Project Priority

The code in the existing DBQuotaDriver is deprecated and hence we need an update. Also as the entire OpenStack community is moving toward hierarchical projects this can be an useful addition to Cinder.

Proposed change

The default quota (hard limit) for any newly created sub-project is set to 0. The neutral value of zero ensures consistency of data in the case of race conditions when several projects are created by admins at the same time. Suppose the default value of number of volumes allowed per project is 100, and A is the root project. And an admin is creating B, a child project of A, and another admin is creating C, again a child project of A. Now, the sum of default values for number of volumes of B and C are crossing the default value of A. To avoid this type of situations, default quota is set as Zero.
A project is allowed to create a volume, only after setting the quota to a non-zero value (as default value is 0). After the creation of a new project, quota values must be set explicitly by a Cinder API call to a value which ensures availability of free quota, before resources can be claimed in the project.
A user with role “cloud-admin” in the root project and with inherited roles is called Cloud-Admin and is permitted to do quota operations across the entire hierarchy, including the top level project. Cloud-Admins are the only users who are allowed to set the quota of the root project in a tree.
A person with role “project-admin” in a project is permitted to do quota operations on its sub-projects and users in the hierarchy. If the role “project-admin” in a project is set as inheritable in Keystone, then the user with this role is permitted to do quota operations starting from its immediate child projects to the last level project/user under the project hierarchy. Note: The roles like “cloud-admin” and “project-admin” are not hard coded. It is used in this Blueprint, just for demonstration purpose.
The total resources consumed by a project is divided into
1. Used Quota - Resources used by the volumes in a project.
  (excluding child-projects)
2. Reserved Quota - Resources reserved for future use by the project does
  not include descendants.
3. Allocated Quota - Sum of the quota hard_limit values of immediate
  child projects
The free quota available within a project is calculated as
free quota = hard_limit - (used + reserved + allocated)

Free quota is not stored in the database; it is calculated for each project on the fly.
An increase in the quota value of a project is allowed only if its parent has sufficient free quota available. If there is free quota available with the parent, then the quota update operation will result in the update of the hard_limit value of the project and allocated value update of its parent project. That’s why, it should be noted that updating the quota of a project requires the token to be scoped at the parent level.
- Hierarchy of Projects is as A->B->C (A is the root project)
  
  Name
  
  hard_limit
  
  used
  
  reserved
  
  allocated
  
  A
  
  100
  
  0
  
  0
  
  50
  
  B
  
  50
  
  20
  
  0
  
  10
  
  C
  
  10
  
  10
  
  0
  
  0
  
  Free quota for projects would be:
  
  A:Free Quota = 100 {A:hard_limit} - ( 0 {A:used} + 0 {A:reserved} +
  50 {A:Allocated to B}) = 50
  
  B:Free Quota = 50 {B:hard_limit} - ( 20 {B:used} + 0 {B:reserved} +
  10 {B:Allocated to C}) = 20
  
  C:Free Quota = 10 {C:hard_limit} - ( 10 {C:used} + 0 {C:reserved} +
  0 {C:Allocated}) = 0
  
  If Project C hard_limit is increased by 10, then this change results in:
  
  Name
  
  hard_limit
  
  used
  
  reserved
  
  allocated
  
  A
  
  100
  
  0
  
  0
  
  50
  
  B
  
  50
  
  20
  
  0
  
  20
  
  C
  
  20
  
  10
  
  0
  
  0
  
  If Project C hard_limit needs to be increased further by 20, then this operation will be aborted, because the free quota available with its parent i.e. Project B is only 10. So, first project-admin of A should increase the hard_limit of Project B (using scoped token to Project A, because of action at level A) and then increase the hard_limit of Project C (again scoped token to Project B)
  
  Please consider the use cases mentioned above. The quota information of various projects, including the allocated quota is as follows,
  
  ProductionIT : hard_limit=1000, used=100, reserved=100, allocated=700
  
  CMS : hard_limit=300, used=25, reserved=15, allocated=250
  
  Computing : hard_limit=100, used=50, reserved=50, allocated=0
  
  Visualisation : hard_limit=150, used=25, reserved=25, allocated=0
  
  ATLAS : hard_limit=400, used=25, reserved=25, allocated=300
  
  Services : hard_limit=100, used=25, reserved=25, allocated=0
  
  Computing : hard_limit=200, used=50, reserved=50, allocated=0
  - Suppose Mike tries to increase the volumes quota in CMS to 400. Since Mike is having the role of admin in ProductionIT which is the parent of CMS, he can increase the quota of CMS provided that the token is scoped to ProductionIT. This is required because the increase of quota limit in CMS results in the corresponding reduction of free quota in ProductionIT.
    
    Using the above formula, free quota of ProductionIT is given by, free quota = hard_limit - used - reserved - allocated free quota = 1000 - 100 - 100 - (400 + 400) free quota = 0
    
    So maximum permissible quota for CMS is 300 + 100 = 400
    
    Note:ProductionIT:allocated = CMS:hard_limit + ATLAS:hard_limit
    
    Minimum quota of CMS is given by, CMS:used + CMS:reserved + CMS:allocated = 25 + 15 + 250 = 290
    
    Note: CMS:allocated = Visualisation:hard_limit + Computing:hard_limit
    
    Since 290(minimum quota) <= 400(new quota) <=400(maximum quota), quota operation will be successful. After update, the quota of ProductionIT and CMS will be as follows,
    
    ProductionIT : hard_limit=1000, used=100, reserved=100, allocated=800
    
    CMS : hard_limit=400, used=25, reserved=15, allocated=250
  - Suppose Mike tries to increase the intances quota in CMS to 500. Then it will not be successful, since the maximum quota available for CMS is 400.
  - Suppose Jay who is the Manager of CMS increases the volumes quota in CMS to 400, then it will not be successful, since Jay is not having admin or project-admin role in ProductionIT which is the parent of CMS.
  - Suppose Mike tries to increase the quota of ProductionIT to 2000, then it will be successful. Since ProductionIT is the root project, there is no limit for the maximum quota of ProductionIT. And also, Mike is having admin role in ProductionIT. For a private cloud the hard_limit depends on the internal inventory that is maintained internally by the cloud provider. Mike the Cloud Admin will have an access to these details and will update the hard_limit depending on the available inventory. So hard_limit is bounded by the available inventory for the Private Cloud and it will vary for each Private Cloud.
A decrease in the quota value of a project is allowed only if it has free quota available, free quota > 0 (zero), hence the maximum decrease in quota value is limited to free quota value.

Name	`hard_limit`	`used`	`allocated`
A	100	0	50
B	50	20	10
C	10	10	0

Name	`hard_limit`	`used`	`allocated`
A	100	0	50
B	50	20	20
C	20	10	0

Hierarchy of Projects is A->B->C, where A is the root project
Project A (hard_limit = 100, used = 0, reserved = 0, allocated = 50) Project B (hard_limit = 50, used = 20, reserved = 0, allocated = 10) Project C (hard_limit = 10, used = 10, reserved = 0, allocated = 0)

If Project B hard_limit is reduced by 10, then this change results in Project A (hard_limit = 100, used = 0, reserved = 0, allocated = 40) Project B (hard_limit = 40, used = 20, reserved = 0, allocated = 10) Project C (hard_limit = 10, used = 10, reserved = 0, allocated = 0)

If Project B’s hard_limit needs to be reduced further by 20, then this operation will be aborted, because the free quota of Project B should be greater than or equal to (20+0+10).

Suppose Mike tries to reduce the volumes quota in CMS to 350, it will be successful since the minimum quota required for CMS is 290.
Suppose Mike tries to reduce the volumes quota of CMS to 200, then it will not be successful, since it violates the minimum quota criteria.
Delete quota is equivalent to updating the quota with zero values. It will be successful if the allocated quota is zero. Authentication logic is same as that of update logic.
- Suppose Mike tries to delete the quota of CMS then it will not be successful, since allocated quota of CMS is non-zero.
- Suppose Mike deletes the quota of Visualisation, then it will be successful since the allocated quota of Visualisation is zero. The deleted quota of Visualisation will add to the free_quota of CMS. The quota of CMS will be CMS :hard_limit=300, used=25, reserved=15, allocated=100.
- Suppose, Mike deletes the project Visualisation, the quota of Visualisation should be released to its parent, CMS. But in the current setup, Cinder will not come to know, when a project is deleted from keystone. This is because, Keystone service is not synchronized with other services, including Cinder. So even if the project is deleted from keystone, the quota information remains there in cinder database. This problem is there in the current running model of OpenStack. Once the keystone service is synchronized, this will be automatically taken care of. For the time being, Mike has to delete the quota of Visualisation, before he is deleting that project. Synchronization of keystone with other OpenStack services is beyond the scope of this blueprint.
Suppose if Jay, who is the Manager of CMS tries to view the quota of ATLAS, it will not be successful, since Jay is not having any role in ATLAS or in the parent of ATLAS.
Suppose Duncan who is the Manager of Visualisation tries to update the quota of CMS, it will not be successful, because he is not having admin or project-admin role in the parent of CMS.
Suppose if the organization doesn’t want a nested structure and wants only four projects namely, Visualisation, Computing, Services and Operations, then the setup will work like the current setup where there is only one level of projects. All the four projects will be treated as root projects.
In case of parallel quota consumption i.e. quota consumption by various sub-projects at the same time, if the request is not satisfied, then the request should be re-tried after sometime after increasing the parent quota. This needs admin intervention for this spec. As discussed in notifications impact section going ahead with notification/event based support we can overcome admin intervention. In upcoming work I will add details on how do we serialize request. This is important because increase in hard_limit for a subproject (depending on the free_quota in the parent) and updating the allocated quota at the parent level needs to be an atomic operation. Once the atomic operation is performed new incoming request can be served.

Alternatives

For quota update and delete operations of a project, the token can be scoped to the project itself, instead to its parent. But, we are avoiding that, because the quota change in the child project lead to change in the free quota of the parent. Because of that, according to this bp, for quota update and delete operations, the token is scoped to the parent.

Data model impact

Create a new column allocated in table quotas with default value 0. This can be done by adding a migration script to do the same.

REST API impact

None

Security impact

The parameter hard_limit used in the spec needs to be closely tied to the actual inventory present in each individual private cloud. As most of the calculations are based of hard_limit the transparent view of available inventory is needed for quota calculations. This is beyond the scope of this spec but bringing it up for clarity. Also care should be taken to ensure that this does not allow any quota escape vulnerabilities. Given it is a more complex model, it is well worth reviewers spending time on actively trying to subvert the model (time of check v/s time of use, etc).

Notifications impact

Any change of quota values (not usage, but limits) will generate an event. This can be useful for general debugging, auditing to figure out who did what. For this spec will restrict the notification scope to just tracing and auditing but going ahead event or notification mechanism can be used to notify the starving sub-project, to retry later, when the parent has enough free quota available. Also going ahead when Keystone deletes a project a notification can go to a queue and then cinder can consume this event to proactively free up the quota from cinder db.

Other end user impact

Only Cloud-Admin or immediate parent can set quota on sub-project. Cloud-Admin can even set and update his/her own quota.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

vilobhmm

Other contributors:

Work Items

For the first implementation patch of this spec we will only fix up the quota calculations to be sub-project aware not add any new editing capabilities. Also we will skip updating of quotas by anybody except root project admin.
For second implementation patch of this spec we will add quota editing by project admin of sub-projects as well.
A role called “cloud-admin” will be created and assigning that role to a user in root project and making it inheritable.
Role called “project-admin” will be created. The user with “project-admin” role in a project will be able to do quota operations in projects starting from its immediate child projects to the last level project/user under the project hierarchy, provided it is inheritable.

Note:The roles like “cloud-admin” and “project-admin” are not hard coded. It is used in this Blueprint, just for demonstration purpose.
A new Quota Driver called NestedQuotaDriver will be implemented by extending the existing DbQuotaDriver, to enforce quotas in hierarchical multitenancy in OpenStack.
A migration script will be added to create the new column allocated in table quotas, with default value 0.

Dependencies

Depends on bp hierarchical-multitenancy

Testing

Unit tests will be added for all the REST APIs calls.
Add unit tests for integration with other services.

Documentation Impact

Documentation will be updated to give details about the quota calculation and how the quota will be assigned and managed by the projects while using nested quota driver(in projects which have hierarchical support). These deployment need to be on or later Kilo since the hierarchical support was added since Kilo release.

References

Clone Consistency Group

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/clone-cg

This proposal is to add a feature to create a cloned consistency group from an existing one.

Problem description

Currently a user can create a Consistency Group first and then clone one volume from the existing CG at a time and add it to the new CG until all volumes are cloned and added to the CG.

The proposal wants to enhance the existing create CG from source API and make this a one step rather than multi-step process.

Note: The cloned volumes in the new CG are not guanranteed to be consistent. From the consistency point of view, it’s no different from creating a empty CG first and then creating volume one after another and adding to the CG.

The existing Create CG from source API allows user to create a CG from the group snapshot (CG snapshot), which should be consistent.

Use Cases

Suppose a user already has a CG with volumes in it. He/she now wants to create another CG with all volumes cloned from the other CG. The proposed change will make this much easier for the user.

Proposed change

The existing Create CG from Source API takes an existing CG snapshot as the source.
This blueprint proposes to modify the existing API to accept an existing CG as the source.

Alternatives

Without the proposed changes, we can create a CG from an existing CG with these steps:

Create an empty CG.
Create a cloned volume from an existing volume in an existing CG and add to the new CG.
Repeat the above step for all volumes in the CG.

Data model impact

DB Schema Change: A new column source_cg_id will be added to the consistencygroups table.

REST API impact

Consistency Group API change

Create Consistency Group from Source

V2/<tenant id>/consistencygroups/create_from_src

Method: POST

JSON schema definition for V2:

{
    "consistencygroup-from-src":
    {
        "name": "my_cg",  # existing
        "description": "My consistency group",  # existing
        "cgsnapshot_id": "xxxxxxxx",  # existing
        "consistencygroup_id": "xxxxxxxx",  # new
    }
}

In the existing Create Consistency Group from Source API, add a new parameter “consistencygroup_id” to allow an existing CG to be the source of the new CG.

Cinder Volume Driver API

Two new optional parameters will be added to the existing volume driver API:
- def create_consistencygroup_from_src(self, context, group, volumes, cgsnapshot=None, snapshots=None, src_group=None, src_volumes=None)
- Note only “src_group” and “src_volumes” are new parameters.

Security impact

Notifications impact

Other end user impact

python-cinderclient needs to be changed to support the modified API.

Create Cloned CG cinder consisgroup-create-from-src –name <name> –description <description> –consistencygroup <cg uuid or name>

Performance Impact

Other deployer impact

None.

Developer impact

Driver developers can add support to this feature in the modified driver API.

Implementation

Assignee(s)

Primary assignee:: xing-yang

Other contributors:

Work Items

API change: * Modify Create CG from Source API
Volume Driver API change: * Modify corresponding driver API
DB schema change

Dependencies

Testing

New unit tests will be added to test the changed code.

Documentation Impact

Documentation changes are needed.

References

Cinder DB Archiving

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/db-archiving

We actually don’t delete rows from db, we mark it only as deleted (we have special column). So the DB is growing and growing, so this causes problem with performance.

Problem description

A lot of unused data in the DB causes different kind of problems with performance and maintaining:

Need to filter ‘marked as deleted’ rows in every query
DB contains a lot of unused data in each table and operators need to filter it if they use database directly in an emergency case
Storage usage utilization (low priority)

Use Cases

Proposed change

Create shadow table for each table and copy “deleted” rows from main to shadow table.

We need to provide several method for archiving data:

archive not more than N “deleted” rows in each table
archive “deleted” rows older than specified data
archive all “deleted” data

Shadowing could be started as a periodic task or admin management util.

Alternatives

Create event-based solution to make opportunity operators to subscribe on “deleting” events and store deleted data somewhere.

Data model impact

Create shadow tables
Implement migrations to store current “deleted” rows in shadow table
Shadow tables could have blob field to store some “deleted” data and to not impose restrictions on database schema changes.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

No performance hit from filtering out a potentially massive number of deleted records on every query.

Other deployer impact

Operator or deployer could use periodic task or Cinder management tool to archive “deleted” data.

Developer impact

Developers should care about migrations for shadow tables as well, as for original tables:

table creation or deletion requires creating or deleting corresponding shadow tables
when a table is modified, the shadow tables have to get modified
when one or more columns are moved to a new table, columns from shadow table should also moved to a new shadow table with data migration
downgrades should be implemented for shadow tables too: new tables have to get removed and the migrated columns will have to get reverted

Implementation

Assignee(s)

Primary assignee:: Ivan Kolodyazhny (e0ne)
Other contributors:: Boris Pavlovich (boris-42)

Work Items

None

Dependencies

None

Testing

Unit tests for both API and Tempest will be implemented,

Documentation Impact

Cloud Administration Guide will be updated to introduce new cinder-manage command:

http://docs.openstack.org/admin-guide/blockstorage-manage-volumes.html

References

Nova’s spec for db archiving: https://review.openstack.org/#/c/18493/
Discussion in openstack-dev mailing list:

http://lists.openstack.org/pipermail/openstack-dev/2014-March/029952.html

Enhance List Operations with Pagination Keys

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/extend-limit-implementations

Currently, the pagination keys, like limit, marker, sort key and sort direction and the next link generation, have been implemented for the volume list operations, but not available in other list operations. This blueprint implements the pagination keys and the next link generation for other items, like snapshots, volume transfers, consistency group, consistency group snapshots and backups.

Problem description

The pagination keys mentioned in the above section have been implemented in the list operations for volumes. This feature can be extended to other cinder concepts. The reason why snapshots, volume transfers, consistency groups, consistency group snapshots, and backups are being targeted is because these entities have relatively high frequencies to query and it is more probable for them to reach a large amount comparing to other entities, like type, availability zone, etc.

The next link generation is already available in volume list, so it is straightforward to implement it for other cinder concepts. Please refer to the _get_collection_links function in the cinder.api.common.ViewBuilder class.

Use Cases

Suppose the maximum page size 1000 and there are more than 1000 volume snapshots. If there is no pagination key marker implemented for snapshot, the maximum snapshots the user can query from the API is 1000. The only way to query more than 1000 snapshots is to increase the default maximum page size, which is a limitation to the cinder list operations. If no next link is generated, it is not possible to retrieve more items than the maximum page size. Even if the next link is generated in this case, it will still return the same results as the previous request without the key marker honored. In conclusion, both the pagination keys and the next link generation are necessary to add for the cinder entities.

Proposed change

The change depends on some work items, which have been put in the “Dependencies” section.

The list APIs for snapshots, volume transfers, consistency group, consistency group snapshots and backups will support the following parameters:

limit: Key used to determine the number of returned items in the response
marker: Key used to determine the item, where to start the query
sort: This key is a comma-separated list of sort keys. Sort directions can optionally be appended to each sort key, separated by the ‘:’ character

Multiple sort keys and sort directions will be implemented as they are done for volumes in the patch “GET volumes API sorting REST/volume/DB updates”, https://review.openstack.org/#/c/141915/.

The change is compliant with the syntax for client sorting arguments defined in the patch “CLI Sorting Argument Guidelines”, https://review.openstack.org/#/c/145544/.

The caller can specify these parameters in the request to query the list of the items, e.g. /snapshots?limit=5&marker=snapshot_id&sort=key1:asc,key2:desc,key3:asc

If there are more items available to query in further requests, the next link should be correctly generated for snapshots, volume transfers, consistency group, consistency group snapshots and backups.

Alternatives

It is possible that we define different pagination keys, other than limit, marker, sort information to implement the pagination for other items in Cinder.

However, to keep the consistency in code change, it is better to follow the pattern that volume list does.

Another alternative is cinder sticks with no pagination for other entities except volumes. However, the default maximum page size 1000 can be exceeded, by some entities, like snapshots, in the production environment. It will cause issues in other entities in future.

Data model impact

None

REST API impact

The following existing v2 GET APIs will support the new sorting parameters:

/v2/{tenant_id}/{items}
/v2/{tenant_id}/{items}/detail

{items} will be replaced by the appropriate entities as follows:

For snapshots:
- /v2/{tenant_id}/snapshots
- /v2/{tenant_id}/snapshots/detail
For volume transfers:
- /v2/{tenant_id}/os-volume-transfer
- /v2/{tenant_id}/os-volume-transfer/detail
For consistency group:
- /v2/{tenant_id}/consistencygroups
- /v2/{tenant_id}/consistencygroups/detail
For consistency group snapshots:
- /v2/{tenant_id}/cgsnapshots
- /v2/{tenant_id}/cgsnapshots/detail
For backups:
- /v2/{tenant_id}/backups
- /v2/{tenant_id}/backups/detail

The existing API needs to support the following new Request Parameters for the above cinder concepts:

Parameter	Style	Type	Description
limit	query	integer	Key used to determine the number of returned items in the response
marker	query	string	Key used to determine the item, where to start the query
sort	query	list	A comma-separated list of sort keys, with the sort directions optionally appended to each sort key

In the event that an invalid pagination key is specified then a “badRequest” error response (code 400) will be returned.

For the limit key, it returns a message like “Invalid input received: Invalid limit key”.
For the marker key, it returns a message like “Invalid input received: Invalid marker key”.
For the sort information, it returns a message like “Invalid input received: Invalid sort key” or “Invalid input received: Invalid sort direction” depending on different circumstances.

The next link will be put in the response returned from cinder if it is necessary.

For snapshots, it replies:

{
    "snapshots": [<List of snapshots>],
    "snapshots_links": [{'href': '<next_link>', 'rel': 'next'}]
}

For volume transfers, it replies:

{
    "transfers": [<List of transfers>],
    "transfers_links": [{'href': '<next_link>', 'rel': 'next'}]
}

For consistency group, it replies:

{
    "consistencygroups": [<List of consistencygroups>],
    "consistencygroups_links": [{'href': '<next_link>', 'rel': 'next'}]
}

For consistency group snapshots, it replies:

{
    "cgsnapshots": [<List of cgsnapshots>],
    "cgsnapshots_links": [{'href': '<next_link>', 'rel': 'next'}]
}

For backups, it replies:

{
  "backups": [<List of backups>],
  "backups_links": [{'href': '<next_link>', 'rel': 'next'}]
}

Security impact

None

Notifications impact

None

Other end user impact

The cinderclient should be updated to accept limit, marker, sort information, in snapshots, volume transfers, consistency group, consistency group snapshots and backups.

The user will be able to specify pagination keys, like limit, marker, sort information to list snapshots, volume transfers, consistency group, consistency group snapshots and backups.

Performance Impact

None

Other deployer impact

The deployer should be aware that the flag osapi_max_limit can set the maximum number of items that a collection resource returns in ONE single response, but it will not limit the number of items returned for the cinderclient request any longer.

After the pagination keys and the next link generation are implemented for the cinder entities, the cinderclient request can retrieve more items than the flag osapi_max_limit sets, because it can fetch the items multiple times via the next link with the marker key until all the items are returned if no limit key is set or the number of items equals to limit if the limit key is set.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Vincent Hou (sbhou@cn.ibm.com)
Other contributors:: None

Work Items

Since the logic code of next link generation is finished in a common class. We do not need to repeat the work any more.

Therefore, I see the following work items:

Add the code to call the common functions to get the limit, marker and sort parameters processed for snapshots, volume transfers, consistency group, consistency group snapshots and backups.
Add the code to do the database query with the paginations keys for snapshots, volume transfers, consistency group, consistency group snapshots and backups.
Add the support of the paginations keys for snapshots, volume transfers, consistency group, consistency group snapshots and backups in cinderclient.
Modify the existing APIs to support passing the limit, marker, and sort information from the API layer to the database layer.
Unit tests to ensure that these pagination keys and the next link generation is supported for snapshots, volume transfers, consistency group, consistency group snapshots and backups.

Dependencies

Cinder pagination: https://blueprints.launchpad.net/cinder/+spec/cinder-pagination, accepted
GET volumes API sorting REST/volume/DB updates: https://review.openstack.org/#/c/141915/, WIP
CLI Sorting Argument Guidelines: https://review.openstack.org/#/c/145544/, accepted
Server sorting guidelines: https://review.openstack.org/#/c/145579, merged

Testing

Both unit and Tempest tests need to be created to ensure that snapshots, volume transfers, consistency group, consistency group snapshots and backups support the pagination keys of limit, marker, and sort information, and the next link generation is available if necessary.

Documentation Impact

The API documentation will need to be updated to accept limit, marker, sort key and sort direction, in snapshots, volume transfers, consistency group, consistency group snapshots and backups as it does for volumes.

References

None

Retrieve Supported Volume Type Extra Specs

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/get-volume-type-extra-specs

Provide an interface to obtain a volume driver’s capabilities.

Problem description

Definitions

Volume Type: A group of volume policies.
Extra Specs: The definition of a volume type. This is a group of policies e.g. provision type, QOS that will be used to define a volume at creation time.
Capabilities: What the current deployed backend in Cinder is able to do. These correspond to extra specs.

The current implementation of volume type extra specs management process in Horizon and the cinder client are error-prone. Operators manage extra specs without having guidance on what capabilities are possible with a backend. Without knowing the capabilities, the operator doesn’t know what the possible extra spec key/values are.

Today operators have to make sure they’re reading the right documentation that corresponds to the version of their storage backend. Documentation can become out of date with their volume driver.

It would be more convenient, and a better user experience if the operator was able to ask Cinder for the capabilities of the current deployed storage backends, instead of having to guess.

Use Cases

As an operator, I want to get a list of the capabilities for my deployed storage backends in Cinder.

With these capabilities, I have keys and possible values that can be used within a volume type’s extra specs.

Potentially with the API endpoint, Horizon could use it to improve the GUI for managing extra specs.

Proposed change

Cinder volume drivers will implement a new method get_capabilities, which will return a dictionary of supported capabilities from the storage backend.

The dictionary will include some brief information about the backend, and capabilities that correspond to extra spec keys and values.

Features like create volume, create snapshots, etc are considered minimum features [1]. Unlike well defined keys [2], minimum features are required to implement. With well defined keys, drivers just need to report if the capability is not supported. If the backend has some vendor unique capabilities, the backend driver can also define vendor unique keys for supported capabilities.

Alternatives

Operators could continue to guess what extra specs they should be setting in volume types. Operators could attempt to find documentation for volume type extra specs. As a last resort, operators could dig through different Cinder volume driver code to figure out the possible extra types. These aren’t reliable solutions.

Data model impact

None

REST API impact

New endpoint GET /v2/tenant/get_capabilities/ubuntu@lvm1_pool:

{
  "namespace": "OS::Storage::Capabilities::ubuntu@lvm1_pool",
  "volume_backend_name": "lvm",
  "pool_name": "pool",
  "driver_version": "2.0.0",
  "storage_protocol": "iSCSI",
  "display_name": "Capabilities of Cinder LVM driver",
  "description":
   "These are volume type options provided by Cinder LVM driver, blah, blah.",
  "visibility": "public",
  "properties": {
   "thin_provisioning": {
      "title": "Thin Provisioning",
      "description": "Sets thin provisioning.",
      "type": "boolean"
    },
    "compression": {
      "title": "Compression",
      "description": "Enables compression.",
      "type": "boolean"
    },
    "vendor:compression_type": {
      "title": "Compression type",
      "description": "Specifies compression type.",
      "type": "string",
      "enum": [
        "lossy", "lossless", "special"
      ]
    },
    "replication": {
      "title": "Replication",
      "description": "Enables replication.",
      "type": "boolean"
    },
    "qos": {
      "title": "QoS",
      "description": "Enables QoS.",
      "type": "boolean"
    },
    "vendor:minIOPS": {
      "title": "Minimum IOPS QoS",
      "description": "Sets minimum IOPS if QoS is enabled.",
      "type": "integer"
    },
    "vendor:maxIOPS": {
      "title": "Maximum IOPS QoS",
      "description": "Sets maximum IOPS if QoS is enabled.",
      "type": "integer"
    },
    "vendor:minIOPS": {
      "title": "Burst IOPS QoS",
      "description": "Sets burst IOPS if QoS is enabled.",
      "type": "integer"
    },
    "vendor:persona": {
      "title": "Persona",
      "description": "I am something..." ,
      "default": "Generic",
      "enum": [
        "Generic",
        "ONTAP-legacy",
        "VMware",
        "OpenVMS",
        "HPUX",
        "WindowsServer",
        "Generic-ALUA",
        "Generic-legacy",
        "HPUX-legacy",
        "AIX-legacy",
        "EGENERA"]
    }
  }
}

The well defined keys are indicated without a prefix like the “qos”. These are fairly standard base keys for Cinder backends. We expect most devices to report at least a boolean True/False for these keys.

The vendor unique keys are optional and are indicated with a prefix of vendor name + colon(:). (ex. abcd:minIOPS) Vendor driver can use anything for the vendor unique keys, but the vendor name prefix shouldn’t contain colon because of the separator and it will be automatically replaced by underscore(_). (abc:d -> abc_d)

Let’s look at compression here:

This is a well defined key, we expect devices to report True or False regarding whether they support it or not. In the case where not only does a device support it, but it can be configured, the option keys are listed under the “options” portion. This is simply the <key-name> of the option, and a list of valid values that can be specified for it. NOTE, if the options key is empty ({}) that means there are NO options that can be set on that capability key.

The vendor:fireproof capability:

This is a vendor unique key, and is indicated by being prefixed with “vendor name” + “:”. Also, note that the default is True and that there are NO options. The example indicates that this device is ALWAYS fireproof, you can’t change that, it just is, what it is.

The thin_provisioning capability:

This is a well defined key which is not supported by this particular vendor. As a result, it defaults to False, and provides no options.

The qos capability describes some corner cases for us:

This key is a well defined key, that’s very customizable via options. Well defined portion is whether the capability is supported or not (again True/False), again, some devices may allow deploying volumes with or without QoS on the same device, so you can specify that with <capability-key-name>=true|false.

If a device doesn’t support this (ie always true), then this entry is omitted.

The other key piece is vendor unique keys. For those that allow additional special keys to set QoS those key names are provided in list format as valid keys that can be specified and set as related to Quality of Service.

The vendor:persona key is another good example of a vendor unique capability:

This is very much like QoS, and again, note that we’re just providing what the valid values are.

You’ll notice that the data-structure follows the settings you would put in your extra-specs. This particular case doesn’t have any options other than the base key itself, so the structure is rather simple.

Security impact

The endpoint mentioned in the API Impact will only be available through the admin_api policy. Operators or other OpenStack services will have the ability to query this information.

Notifications impact

None

Other end user impact

Cinder Client Example:

The operator wants to define new volume types for their OpenStack cloud. The operator would fetch a list of capabilities for a particular backend’s pool:

First get list of services:

$ cinder service-list

With one of the listed pools, pass that to capabilities-list:

$ cinder capabilities-list block1@lvm#pool

host_name: block1
volume_backend_name: lvm
pool_name: pool
driver_version: 2.0.0
storage_protocol: iSCSI

capabilities:

  compression:
    default: true
    options:
      compression: true, false
      compression_type: lossy, lossless, special

  thin_provisioning:
    default: false

  replication:
    default: false

  qos:
    default: true,
    options:
      qos: true, false
      vendor_keys:
        vendor:minIOPS,
        vendor:maxIOPS,
        vendor:burstIOPS

  vendor:fireproof:
    default: true
    options: {}

  vendor:persona:
    default: Generic
    options:
      Generic
      ONTAP-legacy
      VMware
      OpenVMS
      HPUX
      WindowsServer
      Generic-ALUA
      Generic-legacy
      HPUX-legacy
      AIX-legacy
      EGENERA

Horizon:

Horizon will be updated to include the displaying of the supported capabilities so operators can select and set the values while creating or editing the volume type extra specs.

If the volume type does not have any volume backend name associated with it, Horizon will not have any extra specs keys to display. Administrators can still enter in key/value pairs of their own. This is the same behavior as the current process.

If a driver does not publish the extra specs dictionary, which will be the case for any drivers that do not get updated, then no client-side filtering will be performed, and the behavior will basically revert to the current situation where the administrator in horizon will need to know and enter the key/value pairs without any additional guidance.

Performance Impact

None

Other deployer impact

None

Developer impact

Driver maintainers would need to implement a method get_capabilities. This should fetch from the storage backend a list of capabilities and translate it to the dictionary structure:

{
  'driver_version:' '2.0.0',
  'storage_protocol:' iSCSI,
  'capabilities:' {
    'compression': {
      'default': True,
      'options': {
        'compression_type': ['lossy', 'lossless', 'special'],
        'compression': [True, False]
      }
    },
    'thin_provisioning': {
      'default': True,
      options: {
        'thin_provisioning': [True, False]
      }
    },
    'qos': {
      'default': True,
      options: {
        'qos': [True, False],
    }
   }
   'replication': {
     'default': True,
     options: {
       replication: [True, False]
     }
   }
 }

There’s nothing keeping a vendor reporting fewer or more keys, but the following are strictly enforced:

The data structure
The information in the capabilities
The well defined capabilities.
Driver version
Storage protocol

Implementation

Assignee(s)

Primary assignee:

jgravel (julie.gravel@hp.com)

Other contributors:

Work Items

Add new endpoint to Cinder API.
Add RPC call.
Add new volume manager method for get_capabilities.
Update LVM reference implementation with get_capabilities method.

Dependencies

The decision on what the well defined capabilities will be [2].

Testing

Unit tests
Eventually, tempest tests once all drivers are supporting it.

Documentation Impact

Update the Cinder developer documentation for driver maintainers to reference how to push capabilities from their volume driver.

References

[1] - http://docs.openstack.org/developer/cinder/devref/drivers.html#minimum-features
[2] - https://review.openstack.org/#/c/150511
Related horizon spec: https://blueprints.launchpad.net/horizon/+spec/vol-type-extra-specs-describe

Generic image cache functionality

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/image-volume-cache

Currently some volume drivers implement the clone_image method and use an internal cache of volumes on the backend that hold recently used images. For storage backends that can do very efficient volume clones it is a potentially large performance improvement over having to attach and copy the image contents to a each volume. To make this functionality easier for other volume drivers to use, and prevent any duplication in the code base, we should implement a more unified way to do this caching.

Problem description

If we don’t create a unified approach to this image caching we will end up with multiple vendors implementing it in volume drivers. This means duplicated code, redundant vendor prefixed config options, and a non-uniform way for admins to set up and configure image caching.

Use Cases

The primary (and I think only) use case for this is when creating a volume from an image more than once. As an end user I will see (potentially) faster volume creation from an image after the first time, and as an admin once I set this up once in the backend configuration it should require no more interaction.

Proposed change

When creating a volume from an image using the cache there are a few high level steps to take:

Check if an image has a corresponding entry in the cache.
If it is in the cache but has been updated, delete it.
If it wasn’t in the cache (or isn’t anymore)
- Evict an old image entry if room is needed.
- Create a volume from an image (the ‘normal’ way of copying data over), this volume will henceforth be known as the ‘image-volume’. This volume would be owned by a special tenant that is controlled by Cinder.
- Update the cache to know about this new image-volume and its image contents.
Clone the image-volume.

This new behavior would be enabled via a new volume driver config option called ‘image_volume_cache_enabled’. The size of this cache will then be defined by a few new config options:

image_volume_cache_max_size_gb
image_volume_cache_max_size_percent
image_volume_cache_max_count

These options are scoped to each backend.

In the _create_from_image of CreateVolumeFromSpecTask we can add in the logic to check first if the image cache is enabled for the target backend. If it is then we can use the cache, if not we use the slow path only. This would be done after driver.clone_image is called to give a backend a chance to do an even more optimal clone if possible.

For the actual image-volume I think it would be best if we can re-use the normal Cinder Volume model and database table. Ideally we can leverage the cinder-internal tenant feature so that the special tenant will own these cached images.

We then would need a new table to track our image cache. This table will have Volume, Image, and Host id’s, as well as image meta-data to make a decision as to whether or not it is still ‘valid’. More details in the Data Model Impact section.

This information gives us enough to look up whether or not a image is in the cache for a target backend, and make the decision to either create a new one or not and then call _create_from_source_volume with the image-volume.

As far as the volume driver is concerned it would be creating volumes from an image and cloning volumes, it would not need to be aware of the cache at all for the creation methods.

For any backends which need to use snapshots they will have to transition from image-volume -> snapshot -> volume. In the future this image cache table could always be expanded to have a snapshot_id in addition to the volume_id, but for the first iteration of this it will only deal with volumes as the cache backing.

There is a possibility of a race within the cache where an image-volume would be getting evicted and is selected to be used for a clone operation. If this happens we should catch the error and attempt to create the volume by downloading the image. It is also possible that multiple of the same images are cached at the same time, this is considered ‘OK’, eventually extra ones will be evicted if space is needed.

In future iterations of this we could expose the objects in the cache as public snapshots, and store the backing volume for other public snapshots in the same way we do the cached objects. This should allow for quite a bit of re-use in the code, and give users another way to get quick copies of volumes for backends with optimized snapshot->volume and volume->volume operations. In the first iteration this functionality will not be provided. It is worth noting so that we can ensure the implementation is extensible to allow for it in the future.

Alternatives

A very simple alternative to this is to simply push all this logic and behavior into each volume driver and live with the duplicated code and config options. This however prohibits backends that cannot store meta-data for their volumes anywhere.

Another alternative is to not use the actual Volume objects and to expand the image cache table to have enough information for a backend to create volumes and clones from it (maybe metadata type values?). Then we could add new API’s to the volume driver such as ‘create_cached_image_volume’, ‘delete_cached_image_volume’, ‘create_volume_from_cached_image’, and so on. This would put more of the logic into the volume drivers, but requires them to implement these new methods. The benefit here is that we don’t start having special ‘internal’ volumes. The downside being that each driver then has to implement all of these methods which would do almost the same thing as create_volume, delete_volume, etc, but with different parameters.

Data model impact

A new database table for the image cache.

+------------+-----------+--------------------------------+
|    NAME    |    TYPE   |           DESCRIPTION          |
+------------+-----------+--------------------------------+
|            |           |                                |
| id         |  String   |  Auto generated UUID           |
|            |           |                                |
| updated_at |  DateTime |  The updated_at time from the  |
|            |           | image metadata                 |
|            |           |                                |
| host_id    |  String   |  ForeignKey for Host.id that   |
|            |           | has the backing volume         |
|            |           |                                |
| volume_id  |  String   |  ForeignKey for Volume.id of   |
|            |           | the backing volume             |
|            |           |                                |
| image_id   |  String   |  The image id from the image   |
|            |           | metadata                       |
+------------+-----------+--------------------------------+

REST API impact

None

Security impact

The special Cinder owned tenant could potentially be a risk if someone was able to get a hold of its credentials or access the image-volumes. Worst case someone could alter the cached image volumes if they had permission to attach and write to them directly.

Care will have to be taken to ensure it isn’t accessible by normal users.

Notifications impact

New info log messages and event notifications for whether the cache hit or missed. That way there is enough info to determine how effective it is and if settings need to be adjusted.

Other end user impact

None

Performance Impact

This should improve performance on average for systems that can do efficient volume clones when doing create volume from image. There will be many factors involved as to how much it changes, but it is unlikely to be much slower.

It is possible this will add some delay on occasional requests which hit a ‘worst case’ scenario of having to do the database lookups, trying to create from a cached image, failing because it got evicted, and then doing the image download. In situations where that occurs frequently the cache size could be modified or the feature disabled.

Other deployer impact

New configuration options for a cinder backend that would potentially need to be set:

image_volume_cache_enabled (Boolean)
image_volume_cache_max_size_gb (Integer)
image_volume_cache_max_size_percent (Integer)
image_volume_cache_max_count (Integer)

Developer impact

Just new DB API’s and tables to be aware of.

Implementation

Assignee(s)

Primary assignee:: patrick-east
Other contributors:: None

Work Items

DB changes
create_volume flow changes

Dependencies

None

Testing

DB migration tests
Unit tests for DB API’s
Unit tests for flow changes

Documentation Impact

New configuration options.

References

https://blueprints.launchpad.net/cinder/+spec/cinder-internal-tenant

Implement force_detach for safe cleanup

Thu, 16 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/implement-force-detach-for-safe-cleanup

Currently, there is no API to synchronize the state in the Cinder database and the backend storage when a volume is in a stuck ‘attaching’ or ‘detaching’ state. For example, if a volume is stuck in detaching, but is attached/connected by the storage driver, the admin is allowed to use ‘cinder reset-state <vol_id>’ to set the state of the volume to ‘detached’. This will be mismatched with the backend storage state, which has the volume connected/exported to the compute host and nova instance. If the volume state is reset to ‘available’ and ‘detached’, it can be attached to a second instance (note that this is not multi-attach) and lead to data corruption. This spec proposes to plumb cinder APIs that allow an admin to set the state to ‘available’ and ‘detached’ in a safe way, meaning that the backend storage and cinder db are synchronized. An attempt was made to merge a Nova spec to add a ‘–force’ option, but it has stalled and will likely be replaced with some Nova code changes. See: https://review.openstack.org/#/c/84048/44 Nova devs have proposed an idempotent ‘nova volume-detach…’ that will 1) detach the volume from the VM (libvirt.detach) 2) call cinder to detach and Ignore Cinder Errors 3) remove the BlockDeviceMapping from Nova DB. This will have the effect of cleaning up Nova side but possibly leaving Cinder in an error/out_of_sync state, and will require a manual cleanup for Cinder, i.e. cinderclient ‘force-detach’. (cinderclient patch: https://review.openstack.org/#/c/187305/)

Problem description

It is not uncommon to have a volume stuck in a state that prevents the user from using the volume or changing state, i.e. ‘attaching’, ‘deleting’, ‘creating’, ‘detaching’. The python-cinderclient has the command ‘reset-state’ to allow an admin to change the volume state in the cinder database to any state desired. In order to fix the problem in a safe and robust way, however, the admin must check the state of the volume with regards to the backend storage and Nova. Is the volume attached according to the metadata of the backend? Is there an iSCSI connection (or FC) to the compute host? Does Nova BlockDeviceMapping have an entry for the volume? Cleanup to the Cinder DB is possible, cleanup on the storage backend may be possible, and changing the Nova DB requires direct changes (SQL commands) on the Nova database. Synchronizing state requires detailed knowledge of all of the components, and often invasive and dangerous changes to production systems. Changing python-cinderclient ‘reset-state’ has been proposed[1], but that idea was rejected in favor of implementing separate APIs for achieving the desired state[2]. This spec is to implement ‘force_detach’. There is already a ‘force_delete’ API for help with volumes stuck in ‘creating’ or ‘deleting’. In fact, there is already a force_detach API that calls terminate_connection and detach, but this appears to be unused elsewhere in the code base and should be properly implemented to test whether the driver can comply, and in the case of success from the driver the function should cleanup the Cinder DB. It could be argued that there should be an analogous ‘force_attach’ or perhaps more aptly named ‘force_rollforward_attach’, but it seems that simply forcing a detach of a volume will put the volume back in a state where the attach can be attempted again.

Use Cases

UseCase1: Cinder DB ‘attaching’, storage back end ‘available’, Nova DB does not show block device for this volume. An attempt was made to attach a volume using ‘nova volume-attach <instance> <volume_id>’ The Cinder DB is set to ‘attaching’, but the volume was never attached. Current Fix: Using python-cinderclient ‘reset-state’ to set the Cinder DB to ‘available’ will fix this. Proposed Fix: An attempt to detach from Nova will fail, since Nova does not know about this volume. Implementing Cinder force_detach and exposing to the cinderclient will allow cinder to call the backend to cleanup (default implementation is terminate_connection and detach, but can be overridden) and then set Cinder DB state to ‘available’ and ‘detached’.

UseCase2: Cinder DB ‘attaching’, storage back end ‘attached’, Nova DB shows block device for this volume. The Cinder DB is set to ‘attaching’, but the volume was attached. Current Fix: Using python-cinderclient ‘reset-state’ to set the Cinder DB to ‘available’ will result in an out-of-sync state, since the volume is, in fact, attached. Nova will not allow this volume to be re-attached. Proposed Fix: An attempt to detach from Nova will cleanup on the Nova side (after proposed Nova changes) but fail in Cinder since the state is ‘attaching’. Implementing Cinder force_detach and exposing to the cinderclient will allow cinder to call the backend to cleanup (default implementation is terminate_connection and detach, but can be overridden) and then set Cinder DB state to ‘available’ and ‘detached’.

UseCase3: Cinder DB ‘detaching’, storage back end ‘available’, Nova DB does not show block device for this volume. An attempt was made to detach a volume using ‘nova volume-detach <instance> <volume_id>’ The Cinder DB is set to ‘detaching’ and the volume is actually detached. Current Fix: Using python-cinderclient ‘reset-state’ to set the Cinder DB to ‘available’ will fix this. Proposed Fix: An attempt to detach from Nova will fail, since Nova does not know about this volume. Implementing Cinder force_detach and exposing to the cinderclient will allow cinder to call the backend to cleanup (default implementation is terminate_connection and detach, but can be overridden) and then set Cinder DB state to ‘available’ and ‘detached’.

UseCase4: Cinder DB ‘detaching’, storage back end ‘attached’, Nova DB has a block device for this volume. An attempt was made to detach a volume using ‘nova volume-detach <instance> <volume_id>’ The Cinder DB is set to ‘detaching’, but the volume is actually attached. Current Fix: Using python-cinderclient ‘reset-state’ to set the Cinder DB to ‘available’ will result in an out-of-sync state, since the volume is, in fact, attached. Nova will not allow this volume to be re-attached. Proposed Fix: An attempt to detach from Nova will cleanup on the Nova side (after proposed Nova changes) but fail in Cinder since the state is ‘attaching’. Implementing Cinder force_detach and exposing to the cinderclient will allow cinder to call the backend to cleanup (default implementation is terminate_connection and detach, but can be overridden) and then set Cinder DB state to ‘available’ and ‘detached’.

UseCase5: During an attach, initialize_connection() times out. Cinder DB is ‘available’, volume is attached, Nova DB does not show the block device. Current Fix: None via reset-state. Manual intervention on the back end storge is required Proposed Fix: Code in nova/volume/cinder.py L#366 calls initialize_connection, which can time out. A patch is up for review to put this in a try block and cleanup if there is an exception: https://review.openstack.org/#/c/138664/6/nova/volume/cinder.py This patch could be modified to call force_detach() instead of terminate_connection to insure the DB status entry is set to available and allow for driver detach and any driver specific code to be called for cleanup.

Proposed change

Nova must be changed, since it currently checks the state in the Cinder DB and will fail a call to ‘volume-detach’ if Cinder does not show the volume to be ‘attached’ and ‘in-use’. It is being proposed that Nova ingnore Cinder state and do the libvirt.detach and removal of volume entry in BDM. Nova will call Cinder and ignore any errors, leaving cleanup on the Cinder side to be accomplished via manual intervention (i.e. ‘cinder force-detach….’ (Links to proposed Nova changes will be provided ASAP)

Cinder force-detach API currently calls:

volume_api.terminate_connection(...)
self.volume_api.detach(...)

This will be modified to call into the VolumeManager with a new force_detach(…)

api/contrib/volume_actions.py: force_detach(…)

try:
    volume_rpcapi.force_detach(...)
except: #catch and add debug message
    raise #something

self._reset_status(..) #fix DB if backend cleanup is successful

volume/manager.py: force_detach(…)

self.driver.force_detach(..)

Individual drivers will implement force_detach as needed by the driver, most likely calling terminate_connection(..) and possibly other cleanup. The force_detach(..) api should be idempotent: It should succeed if the volume is not attached, and succeed if the volume starts off connected and can be successfully detached.

Alternatives

Leave things as they are, requiring the admin to make manual changes using APIs or commands on the back end storage to keep the state in sync. Nova has no API to cleanup the BlockDeviceMapping table. Using ‘reset-state’ can work, as in UseCase1 and UseCase3, or it can break things and render a volume incapable of being attached, as in UseCase2 and UseCase4.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

Detach notification will indicate force_detach

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

BaseVD class will implement force_detach as it is today, calling terminate_connection and detach. Driver developers can override this function if there is more they wish to do in their driver.

Implementation

Assignee(s)

Primary assignee: scott-dangelo

Work Items

Changes to core Cinder code Drivers implementation of force_detach, if it is desired to override cinderclient changes (https://review.openstack.org/#/c/187305/)

Dependencies

None Nova changes are independent of this spec

Testing

Unit tests for test_force_detach* will be modified as appropriate. Tempest tests will be added to verify a volume in an attaching or detaching state can be force_detached and then successfuly re-attached.

Documentation Impact

Docs will need to be updated for the python-novaclient changes.

References

[1] https://blueprints.launchpad.net/cinder/+spec/reset-state-with-driver: https://review.openstack.org/#/c/134366/2

[2] https://etherpad.openstack.org/p/cinder-meetup-winter-2015 L#405

Non Disruptive Backup

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/non-disruptive-backup

Currently a backup operation can only be performed when a volume is detached. This blueprint proposes to use a temporary snapshot or volume to do non-disruptive backup.

Problem description

If a volume is attached, it has to be detached first before it can be backed up. This is not efficient and it disrupts normal operations by cloud users just to do a backup.

Use Cases

The use case is that an end user would like to do a backup without detaching the volume.

Proposed change

For an attached volume, taking a temporary snapshot is usually less expensive than creating a whole temporary volume. If we can attach the snapshot and read it directly, we do not have to create another temporary volume. It is quicker to complete the backup this way.

If a driver has not implemented attach snapshot and does not have a way to read from a snapshot, we can create a temporary volume from the attached source volume, and backup the temporary volume.

For the LVM driver, because the backup service is on the same node as the volume driver, we can just read the local path of the temporary snapshot after it is created from the attached source volume. We will create a temporary snapshot, but there is no need to attach snapshot for LVM.

In the future, if the backup service can be on a different node than the volume driver, then we can look into whether to do attach snapshot for LVM as well, however, there are concerns that introducing the attach snapshot logic will complicate the LVM code. This is for the future, so it can be decided later.

This proposal will be implemented in multiple steps.

Step 1:

For the LVM driver, take a temporary snapshot of the attached volume and back it up.

Take a temporary snapshot
Do backup from the local path of the temporary snapshot
Cleanup temporary snapshot

For other drivers, provide a default implementation by creating a temporary volume and backing it up.

Create a temporary volume from the source volume
Attach the temporary volume
Do backup from the temporary volume
Detach the temporary volume
Cleanup temporary volume

Note: If the volume is not attached, it will be backed up the same way as before.

Step 2:

Provide a more optimized way for other drivers to use a temporary snapshot to backup an attached volume. For drivers that cannot read from a local path of a snapshot, we need to define an interface for attaching a snapshot, similar to attaching a volume.

Take a temporary snapshot
Attach the temporary snapshot
Do backup from the temporary snapshot
Detach the temporary snapshot
Cleanup temporary snapshot

Step 3 (future):

If the backup service can use the volume driver on a remote node in the future, we can provide a way for the call to happen through RPC. This is similar to step 2, except that it will call the volume manager on a remote node using RPC API to attach snapshot.

Alternatives

There are a couple of alternatives:

Detach the volume and back it up.
Take a snapshot of the attached volume, create a volume from the snapshot and then back it up.

Data model impact

Step 1:

Add the following new columns to the backups table for the temporary volume and snapshot id:

temp_volume_id = Column(String(36)) temp_snapshot_id = Column(String(36))

Add the following new column to the volumes table to persist the status of a volume before it is changed to ‘backing-up’ status. This will be used to restore the volume status back to either ‘available’ or ‘in-use’ after the backup is complete. This is also used for cleaning up failed backups when the backup service is restarted. This is because the cleanup procedure is different for ‘available’ and ‘in-use’ volumes.

previous_status = Column(String(255))

Step 2:

Add the following new column to the snapshots table. The snapshots table already has provider_id and provider_location like the volumes table. This provider_auth column is needed to save the authentication data when a target is created for the snapshot. This is needed if we want to attach the snapshot.

provider_auth = Column(String(255))

REST API impact

Step 1:

Change the existing create backup API to take a force flag. If the volume is ‘in-use’, the force flag has to be True. By default it is False. The force flag is not needed for ‘available’ volumes.

Create backup * V2/<tenant id>/backups * Method: POST * JSON schema definition for V2:

{
    "backup":
    {
        "display_name": "nightly001",  # existing
        "display_description": "Nightly backup",  # existing
        "volume_id": "xxxxxxxx",  # existing
        "container": "nightlybackups",
        "force": True,  # new
    }
}

Step 2:

The following driver APIs will be added to support attach snapshot and detach snapshot.

attach snapshot:

def _attach_snapshot(self, context, snapshot, properties,
                     remote=False)
def create_export_snapshot(self, conext, snapshot)
def initialize_connection_snapshot(self, snapshot, properties,
                                   ** kwargs)

detach snapshot:

def _detach_snapshot(self, context, attach_info, snapshot,
                     properties, force=False, remote=False)
def terminate_connection_snapshot(self, snapshot, properties,
                                  ** kwargs)
def remove_export_snapshot(self, context, snapshot)

Alternatively we can use an is_snapshot flag for volume and snapshot to share common code without adding new functions, but it will make the code confusing and hard to read. So there is a trade off between reducing code duplication and increasing code readibilty here.

Security impact

None

Notifications impact

None

Other end user impact

End user will be able to create a backup without detaching the volume.

Performance Impact

No obvious performance impact.

If we can attach the snapshot and back it up with the proposed change, it will be cleaner and easier than manually taking a snapshot, creating a volume from the snapshot, and then backing it up and deleting it.

Other deployer impact

The deployer will be able to backup an attached volume.

Developer impact

Driver developers can implement the proposed new driver APIs for more efficient backup. This is not required though. A default implementation will be provided to create a temporary volume from the source volume as discussed earlier.

Implementation

Assignee(s)

Primary assignee:: <xing-yang>
Other contributors:: <None>

Work Items

Step 1: Provide a default implementation. Patch in review here: https://review.openstack.org/#/c/193937/
Step 2: Provide a more optimal implementation by adding new driver APIs to support attach snapshot. WIP patch proposed here: https://review.openstack.org/#/c/201249/
Step 3 (future): This will happen after the backup service is decoupled from the volume driver.

Dependencies

None

Testing

Unit tests will be provided.

Documentation Impact

Documentation will be modified to describe this feature.

References

Link to summit discussion: https://etherpad.openstack.org/p/cinder-liberty-backup-using-snapshot

Offload rbd’s copy_volume_to_image function from host to ceph cluster

Thu, 16 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/optimze-rbd-copy-volume-to-image

Offload rbd’s copy_volume_to_image function from cinder-volume host to ceph cluster if cinder volume back end and glance storage back end are in the same ceph storage cluster.

Problem description

Suppose cinder volume back end and glance storage back end are in the same ceph storage cluster. Suppose the volume’s capacity and data size in the following description are 1G. Suppose cinder volume back end use “volumes” pool, and glance storage back end use “images” pool.

Currently upload-to-image routine works as follows: 1. cinder use command “rbd export” to export the rbd image as a local file in cinder-volume host. Read 1G, and Write 1G. 2. cinder call glance upload/update API to upload the exported local file to glance storage back end. Read 1G, and write 1G.

So when we upload a 1G volume to image, we would read 2G data and write 2G data. If we offload this function from cinder-volume host to ceph storage cluster, we may only need to read 1G data and write 1G data, which could reduce the half amount of data transmission compared to the current routine.

Benefits:

Reduce the read/write cost of rbd’s copy_volume_to_image function, especially when the volume’s capacity and data size are big.
Reduce the IO pressure on cinder-volume host when doing the copy_volume_to_image operation.

Use Cases

Proposed change

By offload rbd’s copy_volume_to_image function from host to ceph cluster , we need to do the following changes.

Check whether cinder volume back end and glance storage back end are in the same ceph storage cluster or not.
If not, keep the current work routine not change.
If yes, use rbd.Image.copy(dest_ioctx, dest_name) function to copy volume from one pool to another pool because cinder volume back end and glance storage back end always use different ceph pools.

Alternatives

Solution A: use rbd.Image.copy(dest_ioctx, dest_name) function to offload rbd’s copy_volume_to_image function from host to ceph cluster.

Solution B: use ceph’s functions of clone image and flatten clone image to offload rbd’s copy_volume_to_image function from host to ceph cluster. It contains the following steps. * Create a volume snapshot snap_a and protect the snapshot snap_a. * Clone a child image image_a of snap_a. * Flatten the child image image_a, and snap_a has not been depended on. * Unprotect the snapshot snap_a and delete it.

Using a volume created based on “cirros-0.3.0-x86_64-disk” to show the time-consuming between solution A and solution B. * [Common]Create a volume based on image “cirros-0.3.0-x86_64-disk”.

root@devaio:/home# cinder list
+--------------------------------------+-----------+------+------+
|                  ID                  |   Status  | Name | Size |
+--------------------------------------+-----------+------+------+
| 3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8 | available | None |  1   |
+--------------------------------------+-----------+------+------+

[Solution-A-step-1]Copy volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8 from “volumes” pool to “images” pool.

root@devaio:/home# time rbd cp
volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8 images/test

Image copy: 100% complete...done.

real    0m3.687s
user    0m1.136s
sys     0m0.557s

[Solution-B-step-1]Create a snapshot of volume 3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8 and protect it.

root@devaio:/home# time rbd snap create --pool volumes --image
volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8 --snap image_test

real    0m3.152s
user    0m0.018s
sys     0m0.013s

root@devaio:/home# time rbd snap protect
volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8@image_test

real    0m3.043s
user    0m0.016s
sys     0m0.012s

[Solution-B-step-2]Do clone operation on volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8@image_test.

root@devaio:/home# time rbd clone
volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8@image_test
images/snapshot_clone_image_test

real    0m0.102s
user    0m0.020s
sys     0m0.016s

[Solution-B-step-3]Flatten the clone image images/snapshot_clone_image_test.

root@devaio:/home# time rbd flatten images/snapshot_clone_image_test

Image flatten: 100% complete...done.

real    0m10.228s
user    0m1.375s
sys     0m0.443s

[Solution-B-step-4]Unprotect the snap volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8@image_test.

root@devaio:/home# time rbd snap unprotect
volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8@image_test

real    0m0.064s
user    0m0.019s
sys     0m0.015s

[Solution-B-step-5]Delete the no dependency snap volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8@image_test.

root@devaio:/home# time rbd snap rm
volumes/volume-3d6e5781-e3ac-4106-bfed-0aa0dd3af1f8@image_test

real    0m0.235s
user    0m0.017s
sys     0m0.013s

By the above test result of solution A and solution B, solution A needs (real:0m3.687s, user:0m1.136s, sys:0m0.557s) to finish the volume copy operation and solution B needs (real:0m16.824s, user:0m1.465s, sys:0m0.512s) to do that. So using solution A to offload rbd’s copy_volume_to_image function from host to ceph cluster.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Offload rbd’s copy_volume_to_image function from host to ceph cluster could make full use of ceph’s inherent data copy feature and the hardware capacity of ceph storage cluster to expedite the volume data copy speed, reduce the amount of data transmission and reduce the IO load on cinder-volume host.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ling-yun<zengyunling@huawei.com>

Work Items

Implement code that mentioned in “Proposed change”.

Dependencies

Cinder volume back end and glance storage back end are in the same ceph storage cluster.

Testing

Both unit and Tempest tests need to be created to cover the code change that mentioned in “Proposed change” and ensure that Cinder copy volume to image feature works well while introducing the feature of offload rbd’s copy_volume_to_image function from host to ceph cluster.

Documentation Impact

None

References

None

VolumeReplication_V2

Thu, 16 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/replication-v2

This spec proposes Version 2 of replication. The goal is to take some of the lessons we’ve learned from the first version of replication that was added in the Juno release and see if we can improve it a bit and make it something that is more widely usable by other backend devices.

This spec proposes a use of our entire tool-box to implement replication.

Capabilities - determine if we can even do anything related to replication
Types/Extra-Specs - provide mechanism for vendor-unique custom info and help level out some of the unique aspects among the different back-ends.
API Calls - provide some general API calls for things like enable, disable etc

It would also be preferable to simplify the state management a bit if we can.

Problem description

The current implementation is fairly complex and has proven to be difficult to implement for backend devices as well as difficult to maintain. There is also some concern around how states are managed and stored in the data base.

The existing design is great for some backends, but is challenging for many devices to fit in to.

Use Cases

TBD.

Proposed change

This spec proposes that we make some fairly significant changes to the replication feature in Cinder. We’d still rely on using capabilities and types to identify whether a backend supports replication and ensure we can place a volume correctly if we want to use the feature. The big difference however is around the creation of the replica and the life-cycle that goes with it. For the first iteration we would only support a single remote device, but this is something that’s considered in this spec and could easily be extended to be included in future work. Backend devices (drivers) would be listed in the cinder.conf file and we add entries to indicate pairing. This could look something like this in the conf file:

[driver-foo]
volume_driver=xxxx
valid_replication_devices='backend=backend_name-a',
  'backend=backend_name-b'....

Alternatively the replication target can potentially be a device unknown to Cinder

[driver-foo]
volume_driver=xxxx
valid_replication_devices='remote_device={'some unique access meta}',...

Or a combination of the two even

[driver-foo]
volume_driver=xxxx
valid_replication_devices='remote_device={'some unique access meta}',
  'backend=backend_name-b'....

NOTE That the remote_device access would have to be handled via the configured driver.

This proposal suggests that we would decouple the replication information from the create call. The flow would be something like this:

Create a volume of type “replication_capable=True” and any custom info needed by a specific backend.
Cinder uses the existing functionality of the capabilities filter to pick a backend that supports the requested replication type. This is consistent with the current create workflow, we just add another capability to the scheduler.
Add the following API calls - enable_replication(volume) - disable_replication(volume) - failover_replicated_volume(volume) - update_replication_targets()
- [mechanism to add tgts external to the conf file * optional]
- get_replication_targets() + [mechanism for an admin to query what a backend has configured]

Special considerations

volume-types There should not be a requirement of an exact match of volume-types between the primary and secondary volumes in the replication set. If a backend “can” match these exactly, then that’s fine, if they can’t, that’s ok as well.

Ideally, if the volume fails over the type specifications would match, but if this isn’t possible it’s probably acceptable, and if it needs to be handled by the driver via a retype/modification after the failover, that’s fine as well.
async vs sync This spec assumes async replication only for now. It can easily be extended later for the synchronous case, but for now it’s specific to async. If/When sync is added it can be specified as an additional backend capability. It’s also possible for this to be specified via extra-specs if desired.
transport Implementation details and the how the backend performs replication is completely up to the backend. The requirements are that the interfaces and end results are consistent.
Cinder does not need to be aware of both backend devices but CAN be This spec is intended to provide flexibility, that means that if an admin wishes to configure a backend device that is unknown to Cinder that absolutely fine. The opposite is true as well of course, that detail is outlined in this spec.
Tenant visibility The visibility by tenants is LIMITED!!! In other words the tenant should know very little about what’s going on (if anything at all).

For example, a service provider may sell replication simply as a volume-type defined as “highly available” and have that equate to replication. The point is there’s absolutely no reason an end user should have to know anything at all about replication (unless it costs them more money).
What about devices that can’t do individual volume-rep It’s up to them to figure out what they want to do. If for example they replicate by pool, then maybe they can be sophisticated enough to put all the volumes of replication type in the same pool and replicate the entire pool.

There are lots of options here I think, the point of this spec is that it does not exclude any implementation.

Workflow diagram

Create call on the left:

No change to workflow

Replication calls on the right:

Direct to manager then driver via host entry

     +-----------+
+--< +Volume API + >---------+        Enable routing directly to
|    +-----------+           |        Manager then driver, via host
|                            |
|                            |
|    +-----------+           |
+--> + TaskFlow  |           |
+--< +-----------+           |
|                            |
|                            |
|    +-----------+           |
+--> + Scheduler |           |
+--< +-----------+           |
|                            |
|                            |
|    +-----------+           |
+--> +  Manager  | <---------+
+--< +-----------+ >---------+
|                            |
|                            |
|    +-----+-----+           |
+--> +  Driver   + <---------+
     +-----+-----+

In the case of calls like attach, extend, clone, delete etc; if either the backend host is not reachable, or if the primary_host_status column is set, we’ll redirect to the host in the secondary_hosts column. If that’s unavailable then we fail, just like we do today.

See DB section below

Alternatives

There are all sorts of alternatives, the most obvious of which is to leave the implementation we have and iron it out. Maybe that’s good, maybe that’s not. In my opinion this approach is simpler, easier to maintain and more flexible; otherwise I wouldn’t propose it. The fact that there’s only one vendor that’s implemented replication in the existing setup and they have a number of open issues currently we’re not causing a terrible amount of churn or disturbance if we move forward with this now.

The result will be something that should be easier to implement and as an option will have less impact on the core code.

Data model impact

What new data objects and/or database schema changes is this going to require?

None, for the first pass we should be able to effectively use the existing replication related columns.

REST API impact

We would need to add the API calls mentioned above:

enable_replication(volume)
disable_replication(volume)
failover_replicated_volume(volume)
udpate_replication_targets() [mechanism to add tgts external to the conf file * optional]
get_replication_targets() [mechanism for an admin to query what a backend has configured]

I think augmenting the existing calls is better than reusing them, but we can look at that more closely in the submission stage.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?

Nope
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?

Nope, not that I know of
Does this change involve cryptography or hashing?

Nope, not that I know of
Does this change require the use of sudo or any elevated privileges?

Nope, not that I know of
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.

Nope, not that I know of
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Nope, not that I know of

For more detailed guidance, please see the OpenStack Security Guidelines as a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These guidelines are a work in progress and are designed to help you identify security best practices. For further information, feel free to reach out to the OpenStack Security Group at openstack-security@lists.openstack.org.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cinderclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but when considering large scale deployments the proposed call may in fact be performed on hundreds of nodes.
Scheduler filters get called once per host for every volume being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries can have a profound impact on performance, especially in critical sections of code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other volume drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that targets (LVM) are stored in, how do we handle any used directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the volumes in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other volume drivers would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: john-griffith
Other contributors:: <launchpad-id or None>

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

Dependencies

Include specific references to specs and/or blueprints in cinder, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cinder (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Please discuss how the change will be tested. We especially want to know what tempest tests will be added. It is assumed that unit test coverage will be added so that doesn’t need to be mentioned explicitly, but discussion of why you think unit tests are sufficient and we don’t need to add more tempest tests would need to be included.

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

What is the impact on the docs team of this change? Some changes might require donating resources to the docs team to have the documentation updated. Don’t repeat details discussed above, but please reference them here.

Obviously this is going to need docs

References

Please add any useful references here. You are not required to have any reference. Moreover, this specification should still make sense when your references are unavailable. Examples of what you could include are:

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. link to any vendor documentation)
Anything else you feel it is worthwhile to refer to

The specs process is a bit much, we should revisit it. It’s rather bloated, and while the first few sections are fantastic for requiring thought and planning, towards the end it just gets silly.

RPC and VersionedObject Compatibility

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/rpc-object-compatibility

The following proposal is to enable rolling upgrade in cinder. More specifically, it allows cinder services (cinder-api, cinder-scheduler, cinder-backup, and cinder-volume) to be updated one at a time and yet still be operational (only one service is taken down for upgrade instead of all). It follows the rolling upgrade feature in nova, using oslo_versionedobjects, but is different in that there is no indirection API. Instead, RPC versions and objects are going to be pinned to a backwards compatible version and later switched to the latest version when all services are upgraded.

Problem description

Today, when a new milestone is released, it can take quite a while to upgrade all the service within an OpenStack cloud to the latest release. Upgrade involves: synching the database to the latest schema and updating each distributed cinder service. This process can take down part of a cloud for a few hours or possibly longer. Operators ideally like to reduce the down time for their cloud when upgrading to the next release.

Use Cases

As an operator, I want to individually upgrade each service to the latest release, but not take down my entire cloud in the process.

Proposed change

There are a couple of problems with upgrading a distributed service such as cinder: 1. The content sent over RPC can change. 2. The RPC interface itself can change, i.e. new keyword arguments.

To address these problems, the content can be made backwards compatible using oslo_versionedobject, and the RPC interface can be pinned to a specific version until all components within cinder are upgraded.

The implementation is as follows: 1. When a service starts, it registers the RPC and object versions it knows about, i.e. minimum and latest versions.

2. On upgrade, the admin upgrades each individual cinder service. When the service starts, it knows that there is a newer version, but continue to run at the current/pinned version indicated in the database. The database is the source of truth for which version a service should run.

There is no specific order for which cinder service should be upgraded first. The database keeps track of the RPC and object versions that the service should be using. The service will continue to use the specific versions until an all clear is given by the admin (see below).

3. While other cinder services are being upgraded, the service has to constantly check the database for the current and available RPC and object versions. Since constantly accessing the database is a major performance hit, it can be minimized by caching a copy of the table tracking the versions. The copy will be valid for a few seconds (e.g. 5 seconds), and it is refreshed afterwards. This at least brings down the number of database access for each RPC call.

A flag is used internally within the service to track if it should run in backwards compatibility mode (e.g. BACKWARD_COMPAT_MODE), i.e. RPC and object versions to be made backwards compatible. If the current version is lower than the available version, then BACKWARD_COMPAT_MODE is set to true. Otherwise, it is set to false. There is logic in each service’s rpcapi.py to set the correct keyword arguments for a given RPC version. There is also logic in the objects being sent over RPC to set the correct attributes for a given target version. More specifically, each object should be made backwards compatible by a call to it’s own obj_make_compatible(), which transforms the object to a given target version.

4. Once all cinder services are upgraded, the admin executes a “cinder-manage version upgrade” to switches the current/pinned versions to the latest available versions. This command basically updates the service_versions table in the cinder database. Since the current version is equal to the available version, BACKWARD_COMPAT_MODE is set to false and each RPC call no longer needs to check the database.

By using the BACKWARD_COMPAT_MODE flag, each cinder service only needs to be restarted once (i.e. once for upgrade). Once the flag is turned off, each cinder service can dynamically switch over to using the new RPC and object versions.

Once most of the cinder internals are switched to use oslo_versionedobject and the RPC compatibility layer (this feature) is added, rolling upgrade should be supported (hopefully in the Liberty release and onwards). Rolling upgrade should be limited to two releases older. For example, release N would be backwards compatible with release L (Liberty) and M, and release O would be backwards compatible with release M and N. For a host/service that is too old, it should raise an exception on startup and not be allowed to start.

Alternatives

The upgrade process can be left as-is, where all the components must be taken down and upgraded at the same time. Some cloud operators, e.g. Rackspace, have automated their upgrade procedure to a few minutes. However, other cloud operators have a need for this feature.

Data model impact

A new table, called service_versions, would be created to track the RPC and object versions. The schema would be as follows:

service_versions = Table(
    'service_versions', meta,
    Column('created_at', DateTime(timezone=False)),
    Column('updated_at', DateTime(timezone=False)),
    Column('deleted_at', DateTime(timezone=False)),
    Column('deleted', Boolean(create_constraint=True, name=None)),
    Column('id', Integer, primary_key=True, nullable=False),
    Column('service_id', String(length=255)),
    Column('rpc_current_version', String(length=36)),
    Column('rpc_available_version', String(length=36)),
    Column('object_current_version', String(length=36)),
    Column('object_available_version', String(length=36)),
    mysql_engine='InnoDB'
)

The service_id is service name + host. The *_current_version is the version a service is currently running and pinned at. The *_available_version is the version a service knows about and (if newer) could upgrade to.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

A new command, i.e. “cinder-manage version upgrade”, would be introduced to switch the cinder services to use the latest RPC and object versions.

Performance Impact

During upgrade, before a call is sent over RPC, it would have to check if it needs to be backwards compatible. If so, it would need to massage the RPC interface and object to be backwards compatible. It would incur a cost on performance because there would be extra database calls to find the current and available RPC and object versions. Since accessing the database before each RPC call is a major performance hit, it can be minimized by caching a copy of the table tracking the versions. The copy will be valid for a few seconds (e.g. 5 seconds), and it is refreshed afterwards. This at least brings down the number of database access for each RPC call. Once upgrade is done and all services are upgrade to the latest versions, there would no longer be a need to check the database.

Other deployer impact

None

Developer impact

Any new changes to the RPC interface within cinder-api, cinder-scheduler, cinder-volume, or cinder-backup would have to add to the backwards compatible layer from when this feature merges and onwards. Also, any new changes to an object, e.g. volume, snapshot, etc., must be made backwards compatible in the object’s obj_make_compatible().

Implementation

Assignee(s)

Primary assignee:: thang-pham
Other contributors:: DuncanT (who thought of the feature)

Work Items

Create service_versions table to track RPC and object versions.
Register each cinder service’s RPC and object versions on startup.
Create RPC compatibility layer in each cinder component’s rpcapi.py to massage the object and RPC interface before it is sent over RPC.
Create a “cinder-manage version upgrade” CLI to switch each cinder service to use the latest versions.

Dependencies

oslo_versionobjects for volumes, backups, service, consistency_group, quota need to be merged so that objects can be made backwards compatible.

Testing

Ideally, there should be a rolling upgrade test within tempest (e.g. grenade) to test basic RPC and object pinning between different releases. However, such testing would only apply to release M and onwards because most of the oslo_versionedobject and RPC compatibility layers are not in previous releases.

Documentation Impact

It should be documented that operators can upgrade cinder components individually, without taking down the entire cloud. At the end of the process, a “cinder-manage version upgrade” must be executed to switch the services to use the latest RPC and object versions.

References

Etherpad: https://etherpad.openstack.org/p/cinder-rolling-upgrade
Versioning prototype: https://review.openstack.org/#/c/184404/

Backup Force Delete API

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-force-delete-backup

Provide an API to force delete a backup being stucked in creating or restoring, etc..

Problem description

Currently there are volume force-delete and snapshot force-delete functions, but there is not a force-delete function for backups. Force-delete for backups would be beneficial when backup-create fails and the backup status is stuck in ‘creating’. This situation occurs when database just went down after backup volume and metadata and update the volume’s status to ‘available’, leaving the backup’s status to be ‘creating’ without having methods to deal with through API, because backup-delete api could only delete backup item in status of ‘available’ and ‘error’.

Use Cases

If backup create successfully in object storage, but become stuck in update backup’s status because database just went down. Then use force-delete API, we could directly delete the backup item(include all the stuff in storage backend and db entry info) without manually change the backup’s status in db to error or restart cinder-backup and call backup-delete function, which is very useful for administrators.

Proposed change

A new API function and corresponding cinder command will be added to force delete backups.

The proposal is to provide a method for administrator to quickly delete the backup item that is not in the status of ‘available’ or ‘error’.

It’s an admin-only operation.

Alternatives

First, login in the cinder database, use the following update sql to change the backup item status to ‘available’ or ‘error’.

update backups set status=’available’(or ‘error’) where id=’xxx-xxx-xxx-xxx’;

Second, call backup delete api to delete the backup item.

Data model impact

None

REST API impact

Add a new REST API to delete backup in v2:

POST /v2/{tenant_id}/backups/{id}/action

{
    "os-force_delete": {}
}

Normal http response code:: 202
Expected error http response code:: 404

Security impact

None

Notifications impact

Delete notification should include whether force was used or not

Other end user impact

A new command, backup-force-delete, will be added to python-cinderclient. This command mirrors the underlying API function.

Force delete a backup item can be performed by: $ cinder backup-force-delete <backup>

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ling-yun

Work Items

Implement REST API
Implement cinder client functions
Implement cinder command

Dependencies

None

Testing

Need to test the force delete with an in-progress backup and ensure that it deletes successfully and cleans up correctly.

Documentation Impact

The cinder client documentation will need to be updated to reflect the new command.

http://docs.openstack.org/admin-guide/blockstorage-manage-volumes.html

The cinder API documentation will need to be updated to reflect the REST API changes.

References

None

Support import/export snapshots in cinder

Thu, 16 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/support-import-export-snapshots

Provide a mean to import/export snapshots. Import/export snapshots function could be an complement for the function of import/export volume.

Problem description

Import/export snapshots function:

It could provide the ability to import volumes’ snapshot from one cinder to another cinder, and and import “non” OpenStack snapshots already on a back-end device into OpenStack/cinder.
Export snapshots works the same way as export volumes.

Benefits:

We could use the import snapshots as volume templates to create volumes from snapshots.
Import snapshots function could provide an effective means to manage the import volumes.

::
For those volume drivers which could not delete volume with snapshots, we could not delete the import volume that has snapshots. By using import snapshots function to import snapshots, we could first delete the import snapshots, and then delete the import volumes.

Use Cases

Proposed change

By supporting import/export snapshots, we need to do the following changes.

Add import/export snapshots API.
Add import/export snapshots work flow.

::
Import snapshots would check whether the snapshot’s parent volume exists in cinder. If not, it would raise an exception.NotFound. If the snapshot doesn’t exist in volume back end or the volume doesn’t have related snapshot, it would raise exception.ManageExistingInvalidReference. Currently import snapshots function doesn’t support import those snapshots(generally created by storage’s snap clone operation) without parent volumes. We should use import volumes function to import snapshots without parent volume.

Export snapshots work almost the same as delete snapshot, but it doesn’t delete the snapshot data. For those volume drivers which could directly force delete volumes with snapshots, export snapshots function has no side effect. For those volume drivers which could not delete volumes with snapshots, export snapshots function has a side effect that would cause delete volumes failed. For lvm driver and rbd driver, if unmange volume’s snapshots and delete the volume, the delete operation fail and the volume’s status is available because these two drivers would not continue to delete volume if the volume has snapshots. Drivers raise an VolumeIsBusy exception to volume manager, and volume manager reset the volume’s status to available.
Add manage_existing_snapshot, manage_existing_snapshot_get_size and unmanage_snapshot interface in volume driver and implement these two interface in LVM driver.

Alternatives

None

Data model impact

None

REST API impact

Add one API “manage_snapshot”

The rest API look like this in v2:

  {
      'snapshot':{
      'host': 'cinder-volume',
      'volume_id': 'volume-058660ab-6771-4f82-9627-687b179175d6'
      'ref': '_snapshot-058660ab-6771-4f82-9627-687b179175d6'
      }
  }

* The <string> 'host' means cinder volume host name.
  No default value.

* The <string> 'volume_id' means the import snapshot's volume id.
  No default value.

* The <string> 'ref' means the import snapshot name
  exist in storage back-end.
  No default value.

The response body of it is like:

  {
      "snapshot": {
          "status": "creating",
          "description": null,
          "created_at": "2014-08-25T11:14:31.469591",
          "metadata": {},
          "volume_id": "fbd83a45-cce7-4333-b991-dafd2251edd4",
          "size": 1,
          "id": "71543ced-a8af-45b6-a5c4-a46282108a90",
          "name": null
      }
  }

* The <string> 'status' will be creating.
* The <string> 'description' means the import snapshot display name.
* The <date> 'created_at' means the import time.
* The <map> 'metadata' means the snapshot's metadata.
* The <string> 'volume_id' means the snapshot's volume id.
* The <string> 'id' means the snapshot's id.
* The <string> 'name' means the alias of snapshot id.

Add one API “unmanage_snapshot”.

The rest API look like this in v2:

POST /v2/{project_id}/os-snapshot-manage/{id}/action

{
    'os-unmanage':{}
}

The status code will be HTTP 202 when the request has succeeded.

Security impact

None

Notifications impact

None.

Other end user impact

Users can import snapshots that are already exist in volume back-end.
Export snapshots function has an side effect for those volume drivers which could not delete volumes with snapshots. If using export snapshots function, it would cause the subsequent delete volume operation fail.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ling-yun<zengyunling@huawei.com>

Work Items

Implement code that mentioned in “Proposed change”.
Implement code in python-cinderclient.
Add change API doc.

Dependencies

None

Testing

Both unit and Tempest tests need to be created to cover the code change that mentioned in “Proposed change” and ensure that Cinder snapshot feature works well while introducing import/export snapshots.

Documentation Impact

The cinder API documentation will need to be updated to reflect the REST API changes.
Add the side effect of export snapshots function.

References

None

Support Modifying Volume Image Metadata

Thu, 16 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/python-cinderclient/+spec/support-modify-volumn-image-metadata

This blueprint intends to support modifying volume image metadata, provide Cinder API to allow a user to modify an image property or add new properties.

Cinder should provide the similar mechanism for protected properties as what Glance did, and it is the agreed approach according to the discussion in the IRC and mailing list.

Problem description

When creating a bootable volume from an image, the image metadata (properties) is copied into a volume property named volume_image_metadata.

Cinder volume_image_metadata is the metadata on bootable volumes that Nova looks at and uses for scheduling as well as for passing along to the individual compute drivers.

Cinder volume_image_metadata is used by nova for things like scheduling and for setting device driver options. nova treat volume_image_metadata the same way as the properties on images. This information may need to change after the volume has been created from an image, besides, the additional properties may also needed to make it available in the scheduler (detailed in the below sections). So, There should be a way to support change/update image metadata.

Use Cases

Here are some types of metadata properties that if set will affect runtime characteristics of how Nova handles the booted volume. Many of them very well could be a user deciding to basically build a new image using a volume, but they want to twiddle with various handling properties as part of the building process. Or they simply may have a bootable volume that they want to modify some of these properties.

Hypervisor selection: OpenStack Compute supports many hypervisors, although most installations use only one hypervisor. For installations with multiple supported hypervisors, you can schedule different hypervisors using the ImagePropertiesFilter. This filters compute nodes that satisfy any architecture, hypervisor type, or virtual machine mode properties specified on the instance’s image properties (also volume_image_properties).

Virtual CPU Topology: This provides the preferred socket/core/thread counts for the virtual CPU instance exposed to guests. This enables the ability to avoid hitting limitations on vCPU topologies that OS vendors place on their products. See also: http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/virt-driver-vcpu-topology.rst

Watchdog Behavior: For the libvirt driver, you can enable and set the behavior of a virtual hardware watchdog device for each flavor. Watchdog devices keep an eye on the guest server, and carry out the configured action, if the server hangs. The watchdog uses the i6300esb device (emulating a PCI Intel 6300ESB). If hw_watchdog_action is not specified, the watchdog is disabled. Watchdog behavior set using a specific image’s properties will override behavior set using flavors.

Shutdown Behavior: Numerous things coming: https://review.openstack.org/#/c/89650/12 What landed in Juno: By default, guests will be given 60 seconds to perform a graceful shutdown. After that, the VM is powered off. The os_shutdown_timeout property allows overriding the amount of time (unit: seconds) to allow a guest OS to cleanly shut down before power off. A value of 0 (zero) means the guest will be powered off immediately with no opportunity for guest OS clean-up.

Minimum Flavor Requirements: Minimum required CPU, Minimum RAM. These are used by the Horizon UI to guide flavor selection. Somebody with a bootable volume may install some new software on it and play around with setting the minimum requirements before cloning to an image.

The new Horizon UI is going to be providing more information about images, flavors, volumes to admins and users in the UI leveraging the metadata and looking up rich information about the metadata from the definition catalog to display information to users and admins. This can include metadata about software on the volume.

Proposed change

This ONLY affects the individual volume, it’s has nothing to do with volume types

We are proposing the changes in Cinder to add update capability and provide new Cinder API to allow a user to update an image properties.

Since Glance has RBAC (role based access control) on some specific properties, the RBAC config file is proposed to copied from Glance to Cinder, and sync the protected properties code from Glance into Cinder (Glance are happy to take code cleanups that make this easier; we can consider OSLO or whatever in future but that’s a heavy weight process for another day, copy and paste will do for now).

Alternatives

As to the strategy on protected properties, one proposal is providing new API to query the properties protected or so per-property from Glance to Cinder, but his approach seems will increase the overload of Glance.

Data model impact

None

REST API impact

Since only image metadata is used by nova for VM scheduling or setting device driver options, we proposed to add new REST APIs into Cinder for the operations on image metadata of volume.

update image metadata referenced with volume

Common http response code(s)

Modify Success: 200 OK
Failure: 400 Bad Request with details.
Forbidden: 403 Forbidden

e.g. no permission to update the specific metadata
Not found: 501 Not Implemented

e.g. The server doesn’t recognize the request method
Not found: 405 Not allowed

e.g. HEAD is not allowed on the resource

Update volume image metadata

Method type PUT
API version
PUT /v2/{project_id}/volumes/{volume_id}/image_metadata
JSON schema definition
{
   "image_metadata": {
       "key": "v2"
   }
}
To unset a image metadata key value, specify only the key name. To set a image metadata key value, specify the key and value pair.

Security impact

None

Notifications impact

None

Other end user impact

We intend to expose this via Horizon and are working on related blueprints.
Glance also need share its properties protection code to Cinder and some code cleanups in Glance’s IMPL.

Provide Cinder API to allow a user to update an image property. CLI-python API that triggers the update.

# Sets or deletes volume image metadata
cinder image-metadata  <volume-id> set <property-name = value>

Performance Impact

None anticipated.

Other deployer impact

Two config file will be added into Cinder, that is property-protections- policies.conf and property-protections-roles.conf These file will be put in “/etc/cinder” by default and is configurable via cinder.conf or point directly at the Glance files in devstack for example.
Deployer will be responsible for keeping the config files in sync with Glance’s
The config files will only take effect when they are present on the system. So it is up to the deployer to ensure they are accurate. Otherwise, there will be no impact to Cinder of the OpenStack environment by default.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Dave Chen (wei-d-chen)
Other contributors:: None

Work Items

Changes to Cinder:

Define property protections config files in Cinder (Deployer need to keep the files in sync with Glance’s)
Sync the properties protection code from Glance into Cinder (The common protection code will be shared in Cinder)
Extend existing volume_image_metadatas(VolumeImageMetadataController) controller extension to add update capability.
Reuse update_volume_metadata method in volume API for updating image metadata and differentiate user/image metadata by introducing a new constant “meta_type”
Add update_volume_image_metadata method to volume API.
Check against property protections config files (property-protections-policies.conf or property-protections-roles.conf) if the property has update protection.
Update DB API and driver to allow image metadata updates.

Changes to Cinder python client:

Provide Cinder API to allow a user to update an image property. CLI-python API that triggers the update.

# Sets or deletes volume image metadata cinder image-metadata <volume-id> set <property-name = value>

Dependencies

Same dependencies as Glance.

Testing

Unit tests will be added for all possible code with a goal of being able to isolate functionality as much as possible.

Tempest tests will be added wherever possible.

Documentation Impact

Since Glance has role based access control to properties. It could be the case that we want to update a property in Cinder that is protected in Glance. Eg: a license key is added in glance and it’s copied to cinder when the volume is created. It should not be changed by an unauthorized user in Cinder because this can be violating the billing policies for that image. Therefore, Property Protections which is similar with Glance is proposed to be adopted into Cinder.

We propose to define two samples config file in favor of Property Protections, that is property-protections-roles.conf and property-protections-policies.conf.

property-protections-policies.conf This is a template file when using policy rule for property protections.

Example: Limit all property interactions to admin only using policy rule context_is_admin defined in policy.json.

[.*]

create = context_is_admin

read = context_is_admin

update = context_is_admin

delete = context_is_admin
property-protections-roles.conf

This is a template file when property protections is based on user’s role. Example: Allow both admins and users with the billing role to read and modify properties prefixed with x_billing_code_.

[^x_billing_code_.*]

create = admin,billing

read = admin, billing

update = admin,billing

delete = admin,billing

[.*]
create = context_is_admin
read = context_is_admin
update = context_is_admin
delete = context_is_admin

[^x_billing_code_.*]
create = admin,billing
read = admin, billing
update = admin,billing
delete = admin,billing

Please refer to here, http://docs.openstack.org/developer/glance/property-protections.html for the details explanation of the format.

In case there is property which is protected strictly in Glance, license key for example, deployer should aware the config files may turn out to be inconsistent between Cinder and Glance, it’s up to deployer’s responsibility to keep the config files in sync with Glance’s

Other docs is also needed for new API extension and usage.

References

This blueprint is actually a partial task of Graffiti project, many parts of this concept have already been implemented for other pieces of OpenStack, but that Cinder is outstanding (already completed for images, flavors, host aggregates)

Youtube summit recap of Graffiti Juno POC demo.

Discussions in the mailing list.

Discussions in the IRC.

The Horizon patch set which depends on this functionality

Property Protections introduction in Glance

Deletion of volumes with associated snapshots

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/del-vols-with-snaps

Allow deletion of volumes with existing snapshots. The proposal is to integrate this with the existing volume delete path, using an additional parameter to request deletion of snapshots as well when deleting a volume.

Problem description

When deleting a volume, the delete operation may fail due to snapshots existing for that volume. The caller is forced to examine snapshot information and make many calls to remove snapshots, even if they are not interested in the snapshots at all, and just want the volume gone.

Since snapshots are “children” of volumes in our model of the world, it is reasonable to allow a volume and its snapshots to be removed in one operation both for usability and performance reasons.

Use Cases

More friendly and expected behavior for end-users.

Currently, if a volume has snapshots, the basic user experience is:

Try to delete volume
Get back error message about it having snapshots
Go delete X snapshots manually, become slightly frustrated
Delete the volume

Simpler for other software integrating with Cinder.

I received a request for this functionality because a project integrating with Cinder would like to be able to just delete volumes without having to handle logic for this. I think that is a reasonable point of view (just as it’s reasonable to figure that a user shouldn’t have to handle this).

Faster and more efficient for some volume drivers.

There are two losses of performance in requiring back and forth between cinder-volume and the backend to delete a volume and snapshots:

Extra time spent checking the status of X requests.

Time spent merging snapshot data into another snapshot or volume which is going to immediately be deleted.

This means that we currently force a “delete it all” operation to take more I/O and time than it really needs to. (The degree of which depends on the particular backend.)

Proposed change

A volume delete operation should handle this by default.

Phase 1:

This is the generic/”non-optimized” case which will work with any volume driver.

When a volume delete request is received:

Look for snapshots belonging to the volume, set them all to “deleting” status.
Set the volume to “deleting” status. (Done after #1 so as not to diverge more than needed from our current model of state transitions.)
Issue a snapshot delete for each snapshot. (This loop happens in the volume manager.)
If any snapshot delete operations fail, fail the operation and ensure the volume returns to available. Any snapshots that were successfully deleted remain so. Any snapshots which failed to be deleted are marked as ‘error_deleting’. It may make sense to continue deleting snapshots if an error occurs, or it may be best to stop, depending on the type of error. We probably need some experience with implementing this to be sure of the details for this.
Volume manager now moves all snapshots in ‘deleting’ state to deleted. (volume_destroy/snapshot_destroy)

Phase 2:

This case is for volume drivers that wish to handle mass volume/snapshot deletion in an optimized fashion.

When a volume delete request is received:

Starting in the volume manager…

Check for a driver capability of ‘volume_with_snapshots_delete’. (Name TBD.) This will be a new abc driver feature.

If the driver supports this, call driver.delete_volume_and_snapshots(). This will be passed the volume, and a list of all relevant snapshots.

No exception thrown by the driver will indicate that everything was successfully deleted. The driver may return information indicating that the volume itself is intact, but snapshot operations failed.

Volume manager now moves all snapshots and the volume from ‘deleting’ to deleted. (volume_destroy/snapshot_destroy)

If an exception occurred, set the volume and all snapshots to ‘error_deleting’. We don’t have enough information to do anything else safely.

The driver returns a list of dicts indicating the new statuses of the volume and each snapshot. This allows handling cases where deletion of some things succeeded but the process did not complete.

Alternatives

Implement as the default behavior in volume delete. - Deemed not a suitable change at this time.
Introduce this as a separate volume_action instead of in the standard volume delete path. - Does not help usability without client modifications.

Data model impact

No direct impact.

In implementation, we need to ensure we don’t end up with strange things like a volume in a “deleting” status that has snapshots in “available” status. Thus, failures to delete a single snapshot in this model may cascade to marking the volume and all other associated snapshots as errored. (Only relevant for phase 2 above. This doesn’t happen if we leave the snapshot and volume delete operations separate internally.)

REST API impact

Add a boolean parameter “delete_snapshots” to the delete volume call, which defaults to false.

A volume delete with snapshots which previously returned 400 will now succeed.

Security impact

None.

Notifications impact

None.

All snapshot/volume delete notifications will still be fired.

Other end user impact

New –delete-snapshots parameter for volume-delete in cinderclient.

Performance Impact

Someone deleting a volume and all snapshots should be able to achieve this more quickly, and with fewer REST calls.
Some storage backends will experience less load due to not having to merge snapshots being deleted.

Other deployer impact

None.

Developer impact

New, optional, driver interface:

def delete_volume_and_snapshots(volume, snapshots[]):
This should take whatever driver-specific steps are needed to delete the snapshots and associated volume data.

The assumption can be made that any failed snapshot delete results in a failed volume, so this does not have to account for partial failures.
Note: None of this has to happen at a level above the volume manager since the volume manager handles all related status updates.

Implementation

Assignee(s)

Primary assignee:: eharney (spec, some implementation)
Other contributors:: Other associates (implementation)

Work Items

Investigation:

Understand interaction w/ public/shared snapshots.

Implementation:

Rough order should be: * Add parsing for new parameter to volume delete API * Implement volume manager logic to delete everything * Create an abc class for the new driver interface * Implement volume manager logic to talk to the new driver interface * Implement an optimized case for the LVM driver

Dependencies

None

Testing

Tempest tests will be added to cover this.

Documentation Impact

Need to document the new behavior of the volume delete call, as well as related client examples, etc.

References

https://review.openstack.org/#/c/133822/ This is not proposing the same thing as this spec! It proposed to orphan the snapshots and transform them into volumes, or similar.
https://bugs.launchpad.net/cinder/+bug/1276101 Bug demonstrating one of the usability issues here

Cinder Volume Active/Active support - General description

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support

Right now cinder-volume service can run only in Active/Passive HA fashion and this spec proposes a possible path to support Active/Active configurations in Cinder Volumes.

This spec will only provide a general description of the problem enumerating the different issues we have to resolve without actually going into too much detail. It’s more an eagle’s eye kind of view of the problem.

Each specific issue will have its own spec that gives a detailed description of the problem with proposed solution to the problem.

Problem description

Right now cinder-volume service only accepts Active/Passive High Availability configurations, and there are a number of things that need to change for it to support Active/Active configurations.

API Races

On API nodes given current code we are open to races in the code that affect resources on the database, and this will be exacerbated when working with Active/Active configurations.

Local Manager Locks

We have multiple local locks in the manager code of the volume nodes to prevent multiple green threads from accessing the same resource on specific operations.

This locking is local to the nodes and doesn’t extend to other nodes, so we need to solve mutual exclusion among volume nodes of the same cluster.

Job distribution

Cinder has no concept of clusters, only has the concept of hosts and each host implements a specific backend/service. A mechanism is needed to group hosts from the same cluster under the same conceptual unit while retaining the individual identities of the nodes belonging to the cluster for differentiation in the clean up of crashed nodes.

Cleanup

Right now only one node can work on a specific backend, and therefore on the resources that it contains, so the cleanup is done by the node itself on startup. And if the node does not come up and the resources are left on a stale state it is not a big deal.

It is different with an Active/Active deployment since multiple nodes are sharing the same storage back-end and a node can only do cleanup for the nodes he was working on when he died/failed.

It is also important to do proper cleanup even when a specific node does not come back to life, since other nodes from the same cluster can still manage those resources.

Data Corruption Prevention

Since multiple nodes will be accessing the same storage back-end we have to be extra careful not to access resources that are accessed by other nodes.

More relevant case is when we lose connection to the DB and we no longer can send Service Heartbeats, since Scheduler’s cleanup process (explained in Cleanup proposed changes) will come into place and we could have 2 different nodes accessing the same resource, one because it’s still working on it and the other because it is trying to do the cleanup.

Drivers’ Locks

Some drivers require mutual exclusion for certain operations or when accessing the same resources.

This mutual exclusion is currently being done using local locks in the same way the manager does and they need to be able to work when multiple nodes are accessing the same storage back-end.

Use Cases

Proposed change

API Races

Races on the API nodes will be removed used compare-and-swap updates to the DB.

Specs: https://review.openstack.org/207101/

Job distribution

Job distribution will add the concept of cluster to cinder and send jobs using a topic message queue using the cluster instead of the host like we are doing now.

Specs: https://review.openstack.org/232595

Cleanup

Cleanup will keep track of resources that are have ongoing operations and will have cleanup mechanisms on the Scheduler as well as the Volume nodes.

Cleanup on the nodes will happen on initialization as it is doing now but we’ll also have an automatic cleanup job on the scheduler for the cases where a node with the same host name is not brought up.

Automatic cleanup mechanism will be disabled by default and it will be possible to trigger it manually.

Specs: https://review.openstack.org/236977

Data Corruption Prevention

Stop listening to new jobs from the Message Broker and halt all ongoing operations so we are no longer accessing resources on the Storage Backend.

Specs: https://review.openstack.org/237076

Manager Local Locks

Default solution will be using a DLM with TooZ as the abstraction layer:

Specs: https://review.openstack.org/202615

An alternative solution, that will be initially left as nice to have, will be available for systems that don’t want to install a DLM solution and are using drivers that don’t require distributed locking for Active-Active configurations. This solution replaces local file locks on c-vol’s manager with a DB locking mechanism using workers DB table (introduced by Cleanup changes).

Specs: https://review.openstack.org/237602

Drivers’ Locks

We will be using a DLM solution with TooZ as the abstraction layer:

Specs: https://review.openstack.org/202615

Alternatives

There are quite a number of alternatives to not only each of the issues we need to fix, and they are discussed in the respective specs except for the Drivers’ lock alternative that creates a generic locking mechanism extending the locking mechanism implemented to remove Manager Local Locks.

Specs: https://review.openstack.org/237604

Data model impact

Discussed in the respective specs.

REST API impact

Discussed in the respective specs.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Discussed in the respective specs.

Other deployer impact

Discussed in the respective specs.

Developer impact

None

Implementation

Assignee(s)

Discussed in the respective specs.

Work Items

API Races
Job distribution
Cleanup
Data Corruption Prevention
Manager Local Locks
Drivers’ Locks

Dependencies

None

Testing

Discussed in the respective specs.

Documentation Impact

Discussed in the respective specs.

References

None

Use Tooz to replace local file locks

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support

Right now cinder-volume service can run only in Active/Passive HA fashion. Some of the resons are:

Local file locks in manager that are protecting resources. For example when creating a volume A from volume B a lock on B is created to protect B from getting deleted during creation of A.
Local file locks in drivers like RemoteFS (and others). These locks are used to block operations that cannot run concurrently (like taking a snapshot in case of RemoteFS-based drivers).

This blueprint proposes a switch to use Tooz library for distributed locking.

Problem description

Currently there can be only one active cinder-volume service per volume backend. This means that to achieve high availability of it we need an external service (like Pacemaker) that’s monitoring state of the c-vol and makes a new instance active in case of a failure of a previous one. Also this makes c-vol completely unscalable, because operator cannot just spin up additional services if load is high and needs to rely on having just one instance.

Local file locks that are used in c-vol’s manager and some drivers are preventing that, because the lock won’t be shared between different c-vol services running on different hosts. This may cause multiple issues, resulting even in data loss.

Use Cases

Operator would want to deploy multiple cinder-volume services. The motivation may be to run multiple cinder-volume attached to a single storage backend to increase throughput of requests and therefore performance. Another advantage is increased resiliency to hardware failure when running multiple instances of the service.

Cloud user will get a better cloud.

Proposed change

At Mitaka Design Summit there was a session about allowing projects to have a hard requirement on DLM software. Main conclusions from the session are:

Projects can hard-depend on having a DLM.
Tooz will be the abstraction layer.

That’s why proposed solution is to convert current locks that are local to use Tooz abstraction layer.

This would require unification of current approaches (as some locks are done through cinder.utils.synchronized method and some are using oslo.concurrency.lockutils directly). By default Tooz would be configured to use file locks, so everything will work as today. If operator would want to run multiple cinder-volume services he would need to configure Tooz backend service and set it in cinder.conf. Currently most reliable Tooz backends are ZooKeeper and Redis.

Redis backend in Tooz requires sending periodical heartbeats, so cinder-volume manager will start a new thread that will take care of that. This will be a regular thread (non-eventlet) to be safe from situations when eventlet greenthreads are blocked. It is intended to spawn this thread only when Tooz needs it, so this won’t be done if FileDriver or ZooKeeper is used as backend.

Alternatives

To solve the problem with locks in RemoteFS-based volume drivers we may simply deprecate them and make operators switch to use lock-free drivers. This may be very hard to achieve because according to OpenStack Users Survey 19% of Cinder deployments use these drivers. Also some non-RemoteFS-based drivers are using local locks too.

We could also replace current locks with some DB-based locking. This was proposed by gegulieo in specs to remove local locks from the manager and from drivers, but increased the complexity of the solution and potentially required more testing than relying on a broadly used DLM software.

In the future we will probably want to try to remove locks from volume manager using other means (for example state-based locking). This will make it possible to run c-vol in A/A manner without DLM (when running a volume driver that doesn’t do locking).

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Using an external service for locking will affect the performance. There are some performance tests done by geguileo. Also having a heartbeat thread may add a tiny performance overhead.

These will be non-existent if user would stick to file locks and decide to run c-vol exactly as it works today.

Other deployer impact

Deployer will have a new options in a section [coordination]:

backend_url=file://$state_path - Tooz backend connection string.
heartbeat=1.0 - number of seconds between heartbeats for distributed coordination.
initial_reconnect_backoff=0.1 - number of seconds to wait after failed reconnection to Tooz backend.
max_reconnect_backoff=60.0 - Maximum number of seconds between sequential reconnection retries to Tooz backend.

Deployment tools maintainers will need to decide if they want to use new possibility. If so - they will need to setup a Tooz backend (ZooKeeper, Redis, or less recommended memcached) in their environment.

Developer impact

Developer would need to use locks in cinder-volume service only through Tooz library abstraction layer and be aware that there can run multiple instances of the service.

Implementation

Assignee(s)

Primary assignee:: Michal Dulko (dulek)
Other contributors:: Szymon Wroblewski (bluex-pl)

Work Items

Tooz locking implementation in Cinder (done).
Switch current locks to use Tooz implementation.
- We have some work already done. We should split it for each driver and work with driver maintainers to get patches merged.
Add DevStack patches to set up a CI testing Cinder with Redis or ZooKeeper as Tooz backend.

Dependencies

None

Testing

Unit tests for Tooz code will be added and a CI configured to test Cinder with Redis or ZooKeeper as lock backend will be set up. Possibly we can do that with multinode Tempest.

Documentation Impact

Documentation and Openstack HA Guide will need to be updated to include instructions on how to configure Tooz and deploy cinder-volume in A/A.

References

Differentiate thick and thin provisioning logic in scheduler

Thu, 16 Nov 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/differentiate-thick-thin-in-scheduler

Currently in the capacity filter and weigher of the scheduler, we use the logic to evaluate whether there is enough capacity to thin provision a volume on a backend if the driver reports thin_provisioning_support to be True. However, a driver may be able to support both thin and non-thin provisioning. The logic does not check whether the user wants the volume to be provisioned as thin or not. This blueprint proposes to fix the problem by checking thick_provisioning_support in extra specs of the volume type.

Problem description

If a driver reports both thin_provisioning_support to True and thick_provisioning_support to True for a pool that can support both thin and thick luns, and the user wants to create a thick lun, the logic in the scheduler would wrongly use the logic for thin provisioning to make decisions. It would make decisions based on provisioned_capacity_gb and max_over_subscription_ratio. This could potentially lead to over provisioning.

In this spec, we are trying to address this problem by checking whether the new volume is thin or thick and then use logic accordingly to make decisions.

Use Cases

Currently a driver can report both thin_provisioning_support to True and thick_provisioning_support to True if it has a pool that can support both thin and thick. However the logic in the scheduler checks capacity for thin provisioning if the driver reports thin_provisioning_support to True even if the volume type specifies the volume to be thick. The proposed spec wants to fix this problem.

Proposed change

The spec proposes to make the following change in the logic in the capacity filter and the capacity weigher.

The volume type of the volume to be provisioned will be checked. If provisioning_type is set to thick in the extra specs of the volume type, it will use the thick provisioning logic to evaluate. Note that this only affects the logic if the driver reports both thin_provisioning_support and thick_provisioning_support to True.

Otherwise, the logic remains the same as before.

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Other deployer impact

When the admin sets up volume types, he/she needs to set the following in the extra specs for a thick volume type:

{'thick_provisioning_support': <is> True>} or
{'capabilities:thick_provisioning_support': <is> True>}

Developer impact

Driver developer should be aware of this extra spec and handle it accordingly in volume creation.

Implementation

Assignee(s)

Primary assignee:: <xing-yang>
Other contributors:: <launchpad-id or None>

Work Items

Modify capacity filter to check thick_provisioning_support.
Modify capacity weigher to check thick_provisioning_support.

Dependencies

None.

Testing

Unit tests will be added for this change.

Documentation Impact

Documentation needs to be changed to include this information.

References

A patch is proposed in Manila to solve a similar problem:: https://review.openstack.org/#/c/315266/

Note that capabilities reporting for thin and thick provisioning in Manila is different from that in Cinder. In Manila, a driver reports thin_provisioning = [True, False] if it supports both thin and thick; In Cinder, a driver reports thin_provisioning_support = True and thick_provisioning_support = True if it supports both thin and thick. Therefore the proposal in this spec is different from the solution in the Manila patch.

Generic Volume Group

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/generic-volume-group

In this spec, we are designing a generic volume group. A generic volume group can be used by a driver for various purposes and it can be extended easily in the future to add new features.

Problem description

In Cinder, the only group construct available today is Consistency Group. Consistency Group only supports consistent group snapshot today. When we started to look at extending this support for group replication, lots of issues came up. Consistency Group implies data consistency. For some drivers, a group replication can ensure data consistency across the volumes in the group, but for other drivers, group replication does not imply data consistency.

In this spec, we are designing a generic volume group. This group construct can form a base for adding group replication support which will be discussed in a different spec. The reason for adding a generic volume group is that it can be used by drivers for different purposes. In this spec, we will be providing basic functions including create, delete, and update a volume group. Once the basic generic volume group is added, we can easily extend it to support more new features without adding complex new APIs.

In the following section, use cases are added to explain why a group construct is needed.

Use Cases

Group volumes together for a common purpose A tenant may want to put volumes used in the same application together in a group so that it is easier to manage them together.
Group replication It was advised during previous spec reviews that generic volume groups should be split out from group replication and covered in separate specs. As suggested, group replication is discussed in details in a separate spec in [1].

Once a volume group construct is available, it provides the base for adding support for group replication. Proposed replication functions in [1] only apply to a group that has group_replication_enabled or consistent_group_replication_enabled spec set to True. This means if you have created a group that does not have group_replication_enabled or consistent_group_replication_enabled spec set to True, you cannot enable replication on that group.
Group snapshot Details on group snapshot are discussed in another spec [2]. Once a volume group construct is available, it provides the base for adding support for group snapshot. We also need to change the existing Consistency Group snapshot API to adopt the new group construct and make sure it does not break rolling upgrades.

There is a difference between group snapshot propsed in [2] versus cgsnapshot which is existing today. Group snapshot supports two capabilities, that is consistent_group_snapshot_enabled and group_snapshot_enabled. A group snapshot with consistent_group_snapshot_enabled spec set to True is equivalent to cgsnapshot that is existing today. A group snapshot with group_snapshot_enabled spec set to True is a group of snapshots that does not guarantee consistency at the storage level. More details are provided in [2].

Existing work flow: 1. consisgroup-create creates a CG which is a group of volumes. Some drivers call array APIs to put volumes in a group on the array so that it can take a consistent snapshot later, but other drivers do not really have an array API at this step so it is just that Cinder puts the volumes in a group in the db. The next command cgsnapshot-create is the one that guarantees point-in-time consistency. 2. cgsnapshot-create creates a CG snapshot which is consistent group snapshot.

New work flow: 1. group-create 2. group-snapshot-create Whether the group snapshot is consistent or not depends on group type and capabilities reported by the driver.

Proposed change

Create a group
- Add an API that allows a tenant to create a volume group.
- Add a Volume Driver API accordingly.
Delete a group
- Add an API that allows a tenant to delete a volume group.
- Add a Volume Driver API accordingly.
List groups
- Add an API to list volume groups.
Show a group
- Add an API to show a volume group.
Update a group
- Add an API that adds existing volumes to a group and removes volumes from the group after it is created.
- Add a Volume Driver API accordingly.
- Note: This is the ability to add an existing volume to a group or remove a volume from a group without deleting it. Some driver maintainers have reported that update group cannot be supported by their drivers. For example, HNAS iSCSI driver cannot support update group while implementing the existing CG features because it means moving files between filesystems and it does not have API to do that yet [3]. When implementing update group, the driver maintainer should evaluate backend capabilities and also check the group type specs to decide whether it can be supported or not.
Group type
- Add a group type for a group just like a volume type for a volume.
- One group can support only one group type.
Group type specs
- Add group type specs to describe a group’s characteristics just like volume type extra specs to a volume type.
- Group type specs will be reported as “capabilities” similar to extra specs of a volume type.
Changes in the scheduler.
- Make changes in the scheduler so that an appropriate backend will be chosen to create a group.
DB Schema Changes
- A new group_type table will be created and will contain the following: * uuid of the group_type * name * description
- A new group_type_specs table will be created.
  - uuid of the spec
  - key
  - value
  - uuid of group_type as a foreign key
- A new group table will be created and will contain the following:
  - uuid of the group
  - uuid of a group_type as a foreign key
  - name
  - description
- A new volume_group_mapping table will be created and will contain the following columns. This is needed because one volume could be in multiple groups.
  - uuid of the mapping entry
  - uuid of a volume
  - uuid of a group
  - Note: Restricting a volume to only one group will limit what we can do with the generic group contruct in the future.
- A new group_volumetypes_mapping table will be created and will contain 3 columns:
  - uuid of a group_volumetype entry
  - uuid of a group
  - uuid of a volume type
- A group quota mechanism will be added, similar to what we have for CG.
Changes will be made to make sure new group contruct can support CG.
- In the Newton release, we should be able to create a CG using the new API and preserve the existing behavior of the CG API.
- In the Ocata release, a group will be created in both the new and old tables and we will read from the old table.
- In the “P” release, we will write to both new and old tables and read from the new table.
- In the “Q” release, the old table will be removed. Both write and read will be from the new table.
Driver needs to report group capabilities. Examples are as follows:
- consistent_group_replication_enabled
- group_replication_enabled
- consistent_group_snapshot_enabled
- group_snapshot_enabled
Group type spec needs to specify capabilities, i.e., {‘group_snapshot_enabled’: <is> True}

Alternatives

Without these proposed changes, we can add replication support to the existing consistency group but won’t have a generic volume group that can serve more purposes.

Data model impact

DB Schema Changes
- A new group_type table will be created and will contain the following:
  - uuid of the group_type
  - name
  - description
- A new group table will be created and will contain the following:
  - uuid of the group
  - uuid of a group_type as a foreign key
  - name
  - description
- A new group_specs table will be created and will contain the following:
  - uuid of the group_spec
  - key
  - value
  - uuid of the group_type as a foreign key
- A new group_volumetypes table will be created and will contain 3 columns:
  - uuid of a group_volumetype entry
  - uuid of a group
  - uuid of a volume type
- The volumes table will have a new column:
  - uuid of the group as a foreign key

REST API impact

New Group Type APIs

Create Group Type

V3/<tenant id of admin>/group_types
Method: POST

JSON schema definition for V3:

{
    "group_type":
    {
        "name": "my_group_type",
        "description": "My group type",
        "group_type_specs": {"key1": "value1", "key2": "value2", ...}
    }
}

Delete Group Type
- V3/<tenant id of admin>/group_types/<group_type_uuid>
- Method: DELETE
- This API has no body.

New Group Type Specs APIs

Create Group Type Spec
- V3/<tenant id of admin>/group_types/<group_type_uuid>/group_type_specs
- Method: POST
- JSON schema definition for V3:
```
{
    "group_type_specs":
    {
        "key": "value"
    }
}
```
Delete Group Type Spec
- V3/<tenant id of admin>/group_types/<group_type_uuid>/group_type_specs/ <spec_uuid>
- Method: DELETE
- This API has no body.
List Group Type Specs
- V3/<tenant id of admin>/group_types/<group_type_uuid>/group_type_specs
- Method: GET
- This API has no body.
Show Group Type Spec
- V3/<tenant id of admin>/group_types/<group_type_uuid>/group_type_specs/ <spec_uuid>
- Method: GET
- This API has no body.

New Group APIs

Create a Group
- V3/<tenant id>/groups
- Method: POST
- JSON schema definition for V3:
```
{
    "group":
    {
        "name": "my_group",
        "description": "My group",
        "group_type": group_type_uuid,
        "volume_types": [volume_type1_uuid, volume_type2_uuid, ...],
        "availability_zone": "az1"
    }
}
```
- Cinder scheduler will find a backend that supports the group type and volume types. One group will be hosted on one backend, similar to CG. An empty group will be created first with the above API. After the group is created, a tenant can create a volume providing the group uuid and volume type and the volume will be provisioned and placed in the group and on the backend where the group resides.
- The reason to include volume types when creating a group is to make sure that the backend selected to place the group will be able to host a volume of the specified volume type at a later time.
- One group can support one group type and multiple volume types.
- Cinder API will be responsible for creating the group entry in the database. Cinder driver may or may not need to do anything special when the empty group is first created. This depends on the backend. Most drivers just need to return success.
Update Group
- V3/<tenant id>/groups/<group uuid>
- Method: PUT
- JSON schema definition for V3:
```
{
    "group":
    {
        "name": "my_group",
        "description": "My group",
        "add_volumes": [volume uuid 1, volume uuid 2,...]
        "remove_volumes": [volume uuid 8, volume uuid 9,...]
    }
}
```
- This method can update name, description, as well as volumes in the group. The list after “add_volumes” will contain UUIDs of volumes to be added to the group and the list after “remove_volumes” will contain UUIDs of volumes to be removed from the group. The API will validate the input name, description, UUIDs in add_volumes and remove_volumes fields against the information in Cinder db and send the request to the volume manager. Manager will call driver to do the update on the backend. The API will update Cinder db.
Delete Group
- V3/<tenant id>/groups/<group uuid>/action
- Method: POST (We need to use “POST” not “DELETE” here because the request body has a flag and therefore is not empty.)
- JSON schema definition for V3:
```
{
    "delete":
    {
        "delete-volumes": False
    }
}
```
- Set delete-volumes flag to True to delete a group with volumes in it. This will delete the group and all the volumes. Deleting an empty group does not need the flag to be True.
List Groups
- V3/<tenant id>/groups
- This API lists summary information for all groups.
- Method: GET
- This API has no body.
List Groups (detailed)
- V3/<tenant id>/groups/detail
- This API lists detailed information for all groups.
- Method: GET
- This API has no body.
Show Group
- V3/<tenant id>/groups/<group uuid>
- Method: GET
- This API has no body.
Changes to Create Volume API
- A new field “group_id” (uuid of the group) will be added to the request body.
Cinder Volume Driver API

The following new volume driver APIs will be added:
- def create_group(self, context, group)
- def update_group(self, context, group, add_volumes=None, remove_volumes=None)
- def delete_group(self, context, group, volumes)

Security impact

None.

Notifications impact

Notifications will be added for create, delete, and update groups.

Other end user impact

python-cinderclient needs to be changed to support the new APIs.

Create Group Type

cinder group-type-create –description <description> <name>
Delete Group Type

cinder group-type-delete <group type uuid>
Show Group Type

cinder group-type-show <group type uuid>
List Group Types

cinder group-type-list
Set/unset Group Type Spec

cinder group-type-key <group type name or uuid> <action> <key-value> Valid action is “set” or “unset”.
List Group Type Specs

cinder group-type-specs-list
Create Group

cinder group-create –name <name> –description <description> –availability-zone <availability-zone> –volume-types <volume type uuid> [<volume type uuid>…] group_type
Update Group

cinder group-update –name <new name> –description <new description> –add-volumes <volume uuid> [<volume uuid> …] –remove-volumes <volume uuid> [<volume uuid> …] <group uuid or name>
Delete Group

cinder group-delete –delete-volumes <group uuid> [<group uuid> …] The delete-volumes flag is needed when a group is not empty.
List Group

cinder group-list
Show Group

cinder group-show <group uuid>

Performance Impact

None

Other deployer impact

None

Developer impact

Driver developers can implement the new driver APIs.

Implementation

Assignee(s)

Primary assignee:: xing-yang

Other contributors:

Work Items

New Group Type APIs:
- Create Group Type
- Delete Group Type
New Group Type Spec APIs:
- Create Group Type Spec
- Delete Group Type Spec
- List Group Type Specs
- Show Group Type Spec
New Group APIs:
- Create Group
- Update Group
- Delete Group
- List Groups
- Show Group
New Volume Driver API changes:
- Create Group
- Update Group
- Delete Group
DB schema changes
Implement group methods in the LVM driver.
Make sure both new and old group APIs work. See details in spec [2] on how to achieve this.

Dependencies

Testing

New unit tests will be added to test the changed code.

Documentation Impact

Documentation changes are needed.

References

[1] The replication group spec:: https://review.openstack.org/#/c/229722/
[2] The group snapshot spec:: https://review.openstack.org/#/c/331397/
[3] HNAS iSCSI driver Consistency Group support code review:: https://review.openstack.org/#/c/327043/4/cinder/volume/drivers/ hitachi/hnas_iscsi.py@939

Group Snapshots

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/generic-volume-group

This spec is dependent on the generic volume group spec [1]. The purpose of proposing this spec is to migrate the existing consistency groups support to use the generic volume group constructs and also provide a common API for group snapshots that works for all storage backends.

Problem description

In Juno, we introduced consistency groups (CG) support into cinder. The existing CG support in cinder only supports CG snapshot.

The existing CG support includes the following tables only used for CG. * consistencygroups * cgsnapshot

The existing CG support provides the following APIs. * Create CG * Delete CG * Update CG * Create CG from source (CG or CG snapshot) * List CG * Show CG * Create CG snapshot * Create CG snapshot * List CG snapshot * Show CG snapshot

The generic volume group spec introduces a groups table which is corresponding to the existing consistencygroups table.

The generic volume group spec introduces the following APIs that are corresponding to some existing CG APIs. * Create group * Delete group * Update group * List group * Show group

The missing pieces to support the existing CG feature using the generic volume contructs are as follows.

A group_snapshots table (for cgsnapshots table)
Create group snapshot API (for create cgsnapshot)
Delete group snapshot API (for delete cgsnapshot)
List group snapshots API (for list cgsnapshots)
Show group snapshot API (for show cgsnapshot)
Create group from source group or source group snapshot API (for create CG from cgsnapshot or source CG)

Therefore we are proposing to add the missing pieces in this spec and also provide a way to migrate data from the CG and CGSnapshot tables to group and group snapshots tables while maintaining the support for rolling upgrade.

The existing CG and CG snapshot APIs can only be supported by a subset of storage backends in cinder. In this spec, we propose to provide group snapshot APIs that can be supported by all storage backends.

Use Cases

Group snapshot supports two capabilities, that is consistent_group_snapshot_enabled and group_snapshot_enabled. A group snapshot with consistent_group_snapshot_enabled spec set to True is equivalent to cgsnapshot that is existing today in Cinder and it can guarantee point-in-time consistency at the storage level. A group snapshot with group_snapshot_enabled spec set to True is a group of snapshots that does not guarantee consistency at the storage level.

Data protection products built on top of cinder, nova, neutron, glance, etc. want to protect all OpenStack resources. They can use the group snapshot API to take snapshots in their solution. Without the group snapshot API, the data protection products will have to take snapshot for each volume inidividually.

Proposed change

Create a group snapshot
- Add an API that allows a tenant to create a group snapshot.
- Add a Volume Driver API accordingly.
Delete a group snapshot
- Add an API that allows a tenant to delete a group snapshot.
- Add a Volume Driver API accordingly.
List group snapshots
- Add an API to list group snapshots.
Show a group snapshot
- Add an API to show a group snapshot.
Create a group from a source group or a source group snapshot
- Add an API that creats a group from a source group or a source group snapshot.
- Add a Volume Driver API accordingly.
DB Schema Changes
- A new group_snapshots table will be created and will contain the following: * uuid of the group_snapshot * name * description * uuid of the original group as a foreign key
- A group_snapshot_id column will be added to the snapshots table.
- Two new columns group_snapshot_id and source_group_id will be added to the groups table.
Changes will be made to make sure generic volume group and group snapshots can support CG and CG snapshots.
- Create a default group type and use it only for the existing CGs.
- Write a migration script to copy data from consistencygroups to groups and from cgsnapshots to group_snapshots tables. All existing consistencygroups moved to groups will use the default group type.
- In the future (i.e., Ocata) we can provide a cinder manage command to allow admin to change the group type.
- In Newton, to support rolling upgrade, all existing CG and CG snapshot APIs will continue to work, and they will write to both existing and new tables. Read will be from the existing tables. So list and show will retrieve data from the existing tables.
- In the Ocata release, using old CG APIs will still work and data will be written in both the new and old tables. Read will be from the new tables. So list and show will retrieve data from the new tables.
- In the “P” release, using old CG APIs will still work and both write and read will be using the new tables. The old tables will be removed in the “P” release.
- Using new APIs will write to and read from the new tables only. This means list groups will list all groups in the new tables including those created using CG APIs. The same applies to group snapshots. By checking the group type of a group, you can tell what kind of group it is.
- When creating a group using the new API, if the following is in the group type spec, the manager will call create_group in the driver first and will call create_consistencygroup in the driver if create_group is not implemented.
```
{'consistent_group_snapshot_enabled': <is> True}
```
  Same applies to delete_group, update_group, create_group_snapshot, delete_group_snapshot, and create_group_from_src. This way the new APIs will work with existing driver implementation of CG functions.
- During the “P” release, we can make a decision on whether to keep the CG and CG snapshots APIs or deprecate them in the “Q” release.

Alternatives

We can continue to use the existing CG and CG snapshot APIs.

Data model impact

DB Schema Changes
- A new group_snapshots table will be created and will contain the following.
  - uuid of the group_snapshot
  - name
  - description
  - uuid of the original group as a foreign key
- A group_snapshot_id column will be added to the snapshots table.
- Two new columns group_snapshot_id and source_group_id will be added to the groups table.

REST API impact

New Group Snapshot APIs

Create a Group Snapshot

V3/<tenant id>/group_snapshots
Method: POST

JSON schema definition for V3:

{
    "group_snapshot":
    {
        "name": "my_group_snapshot",
        "description": "My group snapshot",
        "group_id": group_uuid,
        "user_id": user_id,
        "project_id": project_id,
    }
}

Delete Group Snapshot
- V3/<tenant id>/group_snapshots/<group snapshot uuid>
- Method: DELETE
- This API has no body
List Group Snapshots
- V3/<tenant id>/group_snapshots
- This API lists summary information for all group snapshots.
- Method: GET
- This API has no body.
List Group Snapshots (detailed)
- V3/<tenant id>/group_snapshots/detail
- This API lists detailed information for all group snapshots.
- Method: GET
- This API has no body.
Show Group Snapshot
- V3/<tenant id>/group_snapshots/<group snapshot uuid>
- Method: GET
- This API has no body.
Create Group from Source

V3/<tenant id>/groups/action
Method: POST

JSON schema definition for V3:

{
    "create-from-src":
    {
        "name": "my_group",
        "description": "My group",
        "group_snapshot_id": group_snapshot_uuid,
        "source_group_id": source_group_uuid,
        "user_id": user_id,
        "project_id": project_id,
    }
}

Changes to Create Snapshot API
- A new field “group_snapshot_id” (uuid of the group snapshot) will be added to the request body.
Cinder Volume Driver API

The following new volume driver APIs will be added:
- def create_group_snapshot(self, context, group_snapshot, snapshots)
- def delete_group_snapshot(self, context, group_snapshot, snapshots)
- def create_group_from_src(self, context, group, volumes,
  group_snapshot=None, snapshots=None, source_group=None, source_vols=None)

Security impact

None.

Notifications impact

Notifications will be added for create and delete group snapshots and create group from source.

Other end user impact

python-cinderclient needs to be changed to support the new APIs.

Create Group Snapshot

cinder group-snapshot-create –name <name> –description <description> <group uuid>
Delete Group Snapshot

cinder group-snapshot-delete <group snapshot uuid> [<group snapshot uuid> …]
List Group Snapshot

cinder group-snapshot-list
Show Group Snapshot

cinder group-snapshot-show <group snapshot uuid>
Create Group from Source cinder group-create-from-src –group-snapshot <group snapshot uuid> –source-group <source group uuid> –name <name> –description <description>

Performance Impact

None

Other deployer impact

None

Developer impact

Driver developers can implement the new driver APIs.

Implementation

Assignee(s)

Primary assignee:: xing-yang

Other contributors:

Work Items

New Group Snapshot APIs
- Create Group Snapshot
- Delete Group Snapshot
- List Group Snapshots
- Show Group Snapshot
New Clone Group API
- Create Group from Source Snapshot or Source Group
New Volume Driver API changes
- Create Group Snapshot
- Delete Group Snapshot
- Create Group from Source
New DB schema changes
Implement methods in the LVM driver.
Make sure both new and old APIs work. See details in the Proposed Change section.

Dependencies

Testing

New unit tests will be added to test the changed code. Tempest tests should be added as well. Functional tests could be added if needed.

Documentation Impact

Documentation changes are needed.

References

[1] The generic volume group spec:

https://review.openstack.org/#/c/303893/

Cinder Volume Active/Active support - Manager Local Locks

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support

Right now cinder-volume service can run only in Active/Passive HA fashion.

One of the reasons for this is that we have multiple local locks in the Volume nodes for mutual exclusion of specific operations on the same resource. These locks need to be shared between nodes of the same cluster, removed or replaced with DB operations.

This spec proposes a different mechanism to replace these local locks.

Problem description

We have locks in Volume nodes to prevent things like deleting a volume that is being used to create another volume, or attaching a volume that is already being attached.

Unfortunately these locks are local to the nodes, which works if we only support Active/Passive configurations, but doesn’t on Active/Active configurations when we have more than one node, since locks will not guarantee mutual exclusion of operations on other nodes.

We have different locks for different purposes but we will be using the same mechanism to allow them to handle an Active/Active clusters.

List of locks are:

${VOL_UUID}
${VOL_UUID}-delete_volume
${VOL_UUID}-detach_volume
${SNAPSHOT_UUID}-delete_snapshot

We are adding an abstraction layer to our locking methods in Cinder -for the manager and drivers- using Tooz as explained in Tooz locks for A/A that will use local file locks by default and will use a DLM for Active-Active configurations.

But there are cases where drivers don’t need distributed locks to work, they may just need local locks and we would be forcing them to install a DLM just for these few locks in the manager, which can be considered a bit extreme.

Use Cases

Operators that have hard requirements, SLA or other reasons, to have their cloud operational at all times or have higher throughput requirements will want to have the possibility to configure their deployments with an Active/Active configuration and have the same behavior they currently have. But they don’t want to have to install a DLM when they are using drivers that don’t require distributed locks to operate in Active-Active mode just for 4 locks.

Also in the context of the effort to make Cinder capable of working as an SDS outside of OpenStack, where we no longer can make a hard requirement the presence of a DLM as we can within OpenStack, it makes even more sense for Cinder to be able to work without a DLM present for Active-Active configurations if the storage backend drivers don’t require distributed locking in Active-Active environments.

Proposed change

This spec suggests modifying behavior introduced by Tooz locks for A/A for the case where the drivers don’t need distributed locks. So we would use local file locks in the drivers (if they use any) and for the locks in the manager we would use a locking mechanism based on the workers table that was introduced by the HA A/A Cleanup specs.

This new locking mechanism would be an hybrid between local and distributed locks.

By using a locking mechanism similar to the one that local file locks provide we’ll be able to mimic the same behavior we have:

Assure mutual exclusion between different nodes of the same cluster and in the node itself.
Request queuing.
Lock release on node crash.
Require no additional software in the system.
Require no operator specific configuration.

To assure the mutual exclusion using the workers table we will need to add a new field called lock_name that will store the current operation (method name in most cases since table already has the resource type and UUID) that is being performed on the Volume’s manager and it will be used for locking.

New locking mechanism will use added lock_name field to check if workers table already has an entry for that lock_name on the specific resource and cluster of the node, in which case the lock is acquired and we have to wait and retry later until the lock has been released -row has been deleted- and we can insert the row ourselves to acquire the lock. This means that in the case of attaching a volume we will only proceed with the attach if there is no DB entry for our cluster for the volume ID with operation volume_attach.

To assure mutual exclusion, this lock checking and acquisition needs to be atomic, so we’ll be using a conditional insert of the “lock” in the same way we are doing conditional updates (compare-and-swap) to remove API races. Insert query will be conditioned to the absence of a row with some specific restrictions, in this case conditions will be the lock_name, the cluster and the type. If we couldn’t insert the row acquiring the lock failed and we have to wait, if we inserted the row successfully we have acquired the lock and can proceed with the operation.

This process of trying to acquire, fail, wait and retry, is the exact same mechanism we have today with the local file locks. synchronize method when using external synchronization will try to acquire the file lock with a relatively small timeout and if it fails it will try again until lock is acquired.

So by mimicking the same behavior as the file locks we are preserving the queuing of operations we currently have, and not altering our API’s behavior and “implicit API contract” that some external scripts may be relying on.

Since we will be using the DB for the locking operators will not need to add any software to their systems or carry out specific system configurations.

It is important to notice that timeouts for these new locks will be handled by the cleanup process itself, as rows are removed from the DB when heartbeats are no longer being received, thus releasing the locks and preventing a resource from being stuck.

On a closer look at these 4 locks mentioned before we can classify them in 2 categories:

Locks for the resource of the operation.
- ${VOL_UUID}-detach_volume - Used in detach_volume to prevent multiple simultaneous detaches
- ${VOL_UUID} - Used in attach_volume to prevent multiple simultaneous attaches
Locks that prevent deletion of the source of a volume creation (they are created by create_volume method):
- ${VOL_UUID}-delete_volume - Used in delete_volume
- ${SNAPSHOT_UUID}-delete_snapshot - Used in delete_snapshot

For locks on the resource of the operation -attach and detach- the row in the DB is already been inserted by the cleanup method, so we’ll reuse that same row and condition the writing of the lock_name field to the lock being available.

As for locks preventing deletion we will need to add the row ourselves since cleanup was not adding a row in the workers table for those resources as they didn’t require any cleanup.

Alternatives

We could use a DLM, which is a stand-in replacement for local locks, but there have been operator that have expressed their concern on adding this burden -to their systems and duties- because they are using drivers that don’t require locks for Active-Active and would prefer to avoid adding a DLM to their systems.

Instead of using the new locking mechanism for locks that prevent deletion of resources we could add a filter to the conditional update -the one being used to prevent API Races- that will prevent us from deleting a volume or a snapshot that is being used as the source for a volume adding also the appropriate response error when we try to delete such a volume/snapshot.

Data model impact

Adds a new string field called lock_name to the workers table.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Small, but necessary, performance impact from changing local file locks to DB calls.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Gorka Eguileor (geguileo)
Other contributors:: Anyone is welcome to help

Work Items

Add lock_name field to workers table.
Modify Cinder’s new locking methods/decorators to handle hybrid behavior.

Dependencies

Cleanup for HA A/A: https://review.openstack.org/236977

We need the new workers table and the cleanup mechanism.

Removing API Races: https://review.openstack.org/207101/

We need compare-and-swap mechanism on volume and snapshot deletion to be in place so we can add required filters.

Testing

Unittests for new locking mechanism.

Documentation Impact

This needs to be properly documented, as this locking mechanism will not be appropriate for all drivers.

References

General Description for HA A/A: https://review.openstack.org/#/c/232599/

Cleanup for HA A/A: https://review.openstack.org/236977

Removal of API Races: https://review.openstack.org/207101/

Reset generic volume group and group snapshot statuses

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/reset-cg-and-cgs-status

Problem description

Currently we support the administrator to reset the status of volume, snapshot, backup and so on, but both of the generic volume group and group snapshot could also run into error status, the administrator could only modify the status in database. This is inconvenient for administrator.

Use Cases

The primary use case is for the manual recovery of some sort of issue with generic volume group’s status(snapshot’s inclusive). This is helpful for recovering the two resources from a bad status due to real issues with the system or crashed from bugs.

Proposed change

Take the generic volume group for example, The change can add a new admin generic group action on the resource. The logic could be the same as volume’s action reset_status which already exists, for example, to reset the status of generic group to error, it would be:

POST /v3/{tenant_id}/groups/{group_id}/action
{
    'reset_status': {
       'status': 'error'
    }
}

The ‘status’ should be one of the generic group status,the api will validate the input status and then update the status in database.

Alternatives

We could just leave it as a manual db operation, but that leaves it up to an admin to go manually update the db to achieve the same goal.

Data model impact

None

REST API impact

The API microversion will have to be bumped for the new APIs below.

New GroupAdminController:

POST /v3/{tenant_id}/groups/{group_id}/action

And new admin action for it:

reset_status

New GroupSnapshotAdminController:

POST /v3/{tenant_id}/group_snapshots/{group_snapshot_id}/action

And new admin action for it:

reset_status

They both required the identical request body, and ‘status’ should be passed in as a string:

{
    'reset_status': {
       'status': 'error'
    }
}

For the return codes if the status is not valid we will return a 400 Bad Request, if the resource is not found it will be a 404 Not Found, and will otherwise return a 202.

Cinder-client impact

The cinder-client will add commands to expose these new APIs(use ‘state’ instead of ‘status’ to keep consistency with existed reset commands):

New command to reset generic group status:

group-reset-state <group> –state <state>

the ‘group’ should be name or ID of the specified group.

New command to reset group snapshot status:

group-snapshot-reset-state <group-snapshot> –state <state>

the ‘group-snapshot’ should be name or ID of the specified group snapshot.

Security impact

None

Notifications impact

We will add reset_group_status and reset_group_snapshot_status notifications that will behave the same way as the reset_status notifications. There will be a start and end notification.

Other end user impact

The new API will be exposed to users via the python-cinderclient.

Performance Impact

While it is doing a db operation it should be a very low impact.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: tommylikehu(TommyLikeHu@gmail.com)

Work Items

Implement new admin action and tests in Cinder.
Add support in python-cinderclient.

Dependencies

None

Testing

Unit tests in Cinder.
Tempest tests in Cinder.
Functional tests in Cinder.
Unit tests in python-cinderclient.

Documentation Impact

New admin docs to explain how to use the API.

References

None

Replication with More Granularity (Tiramisu)

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/replication-cg

In Mitaka, replication v2.1 (Cheesecake) [1][2] was implemented to support a disaster recovery scenario when an entire host can be failed over to a secondary site. The Cheesecake API was admin facing.

At the Mitaka mid-cycle meetup, Cinder team decided that after Cheesecake is implemented, Tiramisu will be the next step. Tiramisu is the code name for tenant facing replication APIs with more granularity. It gives tenant more control of what should be replicated together.

Problem description

Replication v2.1, which was implemented in Mitaka as an admin API, was designed to solve a true DR scenario when the entire backend has to be failed over to the backend on the secondary site. This is definitely a very important use case.

There are other cases where the customers also want to have more control over the granularity of replication. The ability to replicate a group of volumes and test failover without affecting the entire backend is also a desired feature. A tenant facing API is needed to meet this requirement because only a tenant knows what volumes should be grouped together for his/her application.

It is true that you can do group based replication with Cheesecake by creating a volume type with replication information and creating volumes of that volume type so all volumes created with that type belongs to the same group. So what can Tiramisu do that Cheesecake cannot?

In Tiramisu, a group construct will be used to manage the group of volumes to be replicated together for the ease of management.

The group constuct designed in the Generic Volume Group spec [3] will provide tenant facing group APIs to allow a tenant to decide what volumes should be grouped together and when and what volumes should be added to or removed from the group.
Enable replication function will be added to allow the tenant to decide when all volumes needed for a particular application have been added to the group and signal the start of the replication process.
Disable replication function will be added to allow the tenant to decide when to stop the replication process and make adjustments to the group.
Failover replication function will be added for the tenant to test a failover of a group and make sure that data for the application will not be compromised after a failover.
A tenant can test a failover without affecting the entire backend.
A tenant can decide when and whether to failover when the disaster strikes.

If a tenant has already failed over a group using the Tiramisu API and then a true DR situation happens, the Cheesecake API can still be used to failover the entire backend. The group of volumes already been failed over should not be failed over again. Tiramisu will be using the config options such as replication_device designed for Cheesecake.

Use Cases

An application may have a database stored on one volume and logs stored on another and other related files on other volumes. The ability to replicate all the volumes used by the application together is important for some applications. So the concept to have a group of volumes replicated together is valuable.
A tenant wants to control what volumes should be grouped together during a failover and only the tenant has knowledge on what volumes belonging to the same application and should be put into the same group.
A tenant can determine when to start the replication process of a group by doing enable replication after all volumes used by an application have been added to the group.
A tenant can determine when to stop the replication process of a group by doing disable replication when he/she needs to make adjustments to the group.
A tenant wants the ability to verify that the data will not be compromised after a failover before a true DR strikes. He/she is only concerned about his/her own application and wants to be able to test the failover of his/her own group of volumes without affecting the entire backend.
A tenant wants to decide when and whether to failover when the disaster happens.

Proposed change

The Generic Volume Group design is in a separate spec [3] and this spec depends on it. The URL for the group action API is from the Generic Volume Group spec:

‘/v3/<tenant_id>/groups/<group_uuid>/action’

The following actions will be added to support replication for a group of volumes.

enable_replication
disable_replication
failover_replication
list_replication_targets

If a volume is “in-use”, a “allow-attached-volume” flag needs to set to True for failover_replication to work. This is to prevent the tenant from abusing the failover API and causing problems when forcefully failing over an attached volume. Either the tenant or the admin should detach the volume before failing over, or it will be handled by a higher layer project such as Karbor.

Group type needs to have the following group_spec to support replication:

'group_replication_enabled': <is> True
or
'consistent_group_replication_enabled': <is> True

Driver also needs to report group_replication_enabled or consistent_group_replication_enabled as capabilities.

Karbor or another data protection solution can be the consumer of the replication API in Cinder. The flow is like this:

Admin configures cinder.conf.
Admin sets up group type and volume types in Cinder.
Tenant creates a group that supports replication in Cinder.
Tenant creates volumes and puts into the group in Cinder.
In Karbor, a tenant chooses what resources to put in a protection group. For Cinder volume, Karbor can detect whether a selected volume belongs to a replication group and automatically selects all volumes in that group. For example, Karbor can check whether a Cinder volume group has group_replication_enabled or consistent_group_replication_enabled in its group specs.
There are APIs to create/delete/update/list/show a group in the Generic Volume Group spec [3]. Karbor or another data protection solution can use those APIs to manage the groups for the tenant.
In Karbor, a protection action can be executed depending on the protection policies. If the protection action triggers a fail over, Karbor can call Cinder’s replication API to failover the replication group.
As discussed during the Cinder Newton midcycle meetup [4], driver can report replication_status as part of volume_stats. The volume manager will check the replication_status in volume_stats and update database if replication_status is error.

Alternatives

Do not add a generic volume group construct. Instead, just use the existing consistency group to implement group replication.

Data model impact

None

REST API impact

Group actions will be added:

enable_replication
disable_replication
failover_replication
list_replication_targets

URL for the group action API from the Generic Volume Group design::: ‘/v2/<tenant_id>/groups/<group_uuid>/action’

Enable replication
- Method: POST
- JSON schema definition for V3:
```
{
    'enable_replication': {}
}
```
- Normal response codes: 202
- Error response codes: 400, 403, 404
Disable replication
- Method: POST
- JSON schema definition for V3:
```
{
    'disable_replication': {}
}
```
- Normal response codes: 202
- Error response codes: 400, 403, 404

Failover replication

Method: POST
JSON schema definition for V3:

{
    'failover_replication':
    {
        'allow_attached_volume': False,
        'secondary_backend_id': 'vendor-id-1',
    }
}

Normal response codes: 202
Error response codes: 400, 403, 404

List replication targets

Method: POST
JSON schema definition for V3:

{
    'list_replication_targets': {}
}

Response example:

{
    'replication_targets': [
        {
            'backend_id': 'vendor-id-1',
            'unique_key': 'val1',
            ......
        },
        {
            'backend_id': 'vendor-id-2',
            'unique_key': 'val2',
            ......
        }
     ]
}

Response example for non-admin:

{
    'replication_targets': [
        {
            'backend_id': 'vendor-id-1'
        },
        {
            'backend_id': 'vendor-id-2'
        }
     ]
}

Security impact

None

Notifications impact

Notifications will be added for enable, disable, and failover replication.

Other end user impact

python-cinderclient needs to support the following actions.

enable_replication
disable_replication
failover_replication
list_replication_targets

Performance Impact

None

Other deployer impact

None

Developer impact

Driver developers need to modify drivers to support Tiramisu.

Implementation

Assignee(s)

Primary assignee:: xing-yang
Other contributors:: jgriffith

Work Items

Change capability reporting. Driver needs to report group_replication_enabled or consistent_group_replication_enabled.
Add group actions to support group replication.

Dependencies

This is built upon the Generic Volume Group spec which is already merged and implemented.

Testing

New unit tests will be added to test the changed code.

Documentation Impact

Documentation changes are needed.

References

[1] Replication v2.1 Cheesecake Spec:: https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/cheesecake.html
[2] Replication v2.1 Cheesecake Devref:: https://github.com/openstack/cinder/blob/master/doc/source/devref/replication.rst
[3] Generic Volume Group:: https://github.com/openstack/cinder-specs/blob/master/specs/newton/generic-volume-group.rst
[4] Newton Midcycle Meetup:: https://etherpad.openstack.org/p/newton-cinder-midcycle-day3

Shared backend config stanza

Thu, 16 Nov 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/shared-backend-config

Add a new config section for shared backend configuration options.

Problem description

Right now there isn’t a great way to share config options between backends in a multi-backend cinder.conf. You can put things in DEFAULT, but drivers self.configuration won’t be able to see it if they were defined in their own stanza. You can just look directly at the DEFAULT section, but you then can’t easily override it in the driver backend section if you did want to have something special. As an example consider the following cinder.conf:

[DEFAULT]
enabled_backends = backend1, backend2, backend3
...
my_driver_param = foo
...

[backend1]
...
my_driver_param = bar

[backend2]
..
# not specifying the param

[backend3]
...
(also not specifying, maybe even a different volume driver, doesn't matter)

What we end up with is backend1 gets ‘bar’ and backend2/3 get whatever the default was for ‘my_driver_param’. To get them to use ‘foo’ we need something like:

[DEFAULT]
enabled_backends = backend1, backend2, backend3
...

[backend1]
...
my_driver_param = bar

[backend2]
..
my_driver_param = foo

[backend3]
...
my_driver_param = foo

Part of the confusion is that if you don’t use the multi-backend format you actually do want the options set in DEFAULT.

Use Cases

The main use-case here is for shared configs relevant to multiple backends in cinder.conf. Also to help lessen some of the confusion about shared config options in DEFAULT.

Proposed change

The new config section called ‘backend_defaults’ will contain config options that each backend has access to. If any options are also in the backend stanza that backend specific value will take precedence. It will be safe to assume that config options in shared will be shared and available for all backends. None of the guesswork from DEFAULT.

A new config then might look something like:

[DEFAULT]
enabled_backends = backend1, backend2, backend3
...

[backend_defaults]
my_driver_param = foo

[backend1]
...
my_driver_param = bar

[backend2]
..
# not defining the value, but it gets picked up

[backend3]
...
# same here..

This is the same as the example above where all backends define the value. This simplistic example doesn’t really show a ton of the benefits, so a slightly more realistic example might be:

[DEFAULT]
enabled_backends = pure1, pure2, pure3
...

[backend_defaults]
volume_driver = cinder.volume.drivers.pure.PureISCSIDriver
driver_ssl_verify = true
driver_ssl_cert = /etc/blah/my_cert
use_multipath_for_image_xfer = true
use_chap_auth = true
image_volume_cache_enabled = true
image_volume_cache_max_count = 15
image_volume_cache_max_size_gb = 200
max_over_subscription_ratio = 10
pure_eradicate_on_delete = false

[pure1]
san_ip = 1.2.3.4
pure_api_token = abcdef123456
max_over_subscription_ratio = 50

[pure2]
san_ip = 1.2.3.5
pure_api_token = abcdef1234567
image_volume_cache_max_count = 25

[pure3]
san_ip = 1.2.3.6
pure_api_token = abcdef12345678

So here we have a ton of shared config options, leaving only a few specific to each backend.

In addition to the shared config section we will deprecate setting backends via the DEFAULT section. We will only allow multi-backend style configs using the “enabled_backends” config option.

Alternatives

We could change the behavior so that things in DEFAULT do this, but it not be backwards compatible with older cinder.confs. Behavior would probably not be what deployers would expect when upgrading.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Would have a new config mechanism to learn, but older style configs would still work the same way. They will eventually need to migrate to using the “enabled_backends” config option, this process will follow the standard deprecation process.

Developer impact

Nothing other than needing to be aware of how drivers can get their config values.

Implementation

Assignee(s)

Primary assignee:: patrick-east

Work Items

Add a warning when initializing c-vol without “enabled_backends”
Modify the driver config utility to look for this section first, and override values with backend specific ones if they are defined.
Add some unit tests
Incorporate into multi-backend tempest test configuration

Dependencies

None

Testing

Unit tests
Tempest test config with devstack (reach goal, depends on timing)

Documentation Impact

Updated driver config references
Driver dev documentation to explain how this works

References

https://blueprints.launchpad.net/cinder/+spec/shared-backend-config

Migrate ConfKeyManager encryption key ID into Barbican

Fri, 13 Oct 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/migrate-fixed-key-to-barbican

Support the transition to Barbican by migrating existing keys associated with the ConfKeyManager into Barbican. The ConfKeyManager key uses a key ID of all zeros that’s associated with the fixed_key configuration parameter in the key_manager section of cinder.conf.

Problem description

The legacy ConfKeyManager is insecure, and the prefered encryption key manager is Barbican. Using Barbican in new deployments is easy, but switching deployments from the ConfKeyManager to Barbican is difficult when there are existing volumes encrypted using the ConfKeyManager.

Use Cases

A user has a deployment based on the ConfKeyManager. They wish to to switch to Barbican, but there are existing volumes encrypted using the ConfKeyManager. These existing volumes must continue to function when Barbican is the key manager. The user needs a migration path to Barbican that doesn’t break existing encrypted volumes.

Proposed change

This spec proposes that all instances of the ConfKeyManager’s key (identifed by an all-zeros key ID) be replaced with an identical key stored in Barbican. This is possible because Barbican provides a mechanism for storing existing encryption keys.

A cinder-volume thread will be used to search for encryption key ID instances in the database that match the ConfKeyManager’s all-zeros ID. This thread will be refered to as the key migration thread.

The key migration thread will search the volume table for encryption_key_id entries matching the all-zeros ConfKeyManager key ID. For each matching entry in the volume table, a new Barbican key ID will be created with the exact same secret derived from the fixed_key config value used by the ConfKeyManager. The volume table is updated with the new Baribican key ID.
The thread will only migrate keys associated with volumes belonging to that host. This will avoid contention when there are multiple cinder-volume hosts.
The key migration thread will also look for ConfKeyManager keys stored in metadata tables. Where appropriate, these entries will be updated with the Barbican key ID stored in the corresponding volume table. Otherwise, a new Barbican key ID will be created.
The key migration thread will be spawned on start-up whenever both the following conditions are true:
1. The ConfKeyManager’s “fixed_key” is configured
2. The key_manager’s “backend” (was “api_class”) is not the ConfKeyManager
The key migration thread will generate logs that indicate a key ID has been migrated to Barbican. It will also generate a log that indicates the thread has run to completion without finding any key IDs associated with the ConfKeyManager.

The user is responsible for deciding when to remove the fixed_key from the deployment. It is recommended the user use the logs messages described above to determine when the key migration process has completed. Once the user removes the fixed_key from the configuration, the key migration thread will not be spawned on startup.

The key migration thread will work in conjunction with an enhancement to Castellan and/or Barbican. During the migration process, cinder services may still encounter the ConfKeyManager’s key ID (all zeros) in the database. This is not a valid Barbican key ID, and Barbican will throw an error if asked to provide the key associated with that ID. The enhancement will essentially intercept requests for the all-zeros key ID, and return the ConfKeyManager’s response. All of this would be totally transparent to the rest of the Cinder code. In short, the all-zeros key ID will continue to work even with Barbican as the designated key manager.

Alternatives

Instead of using a separate migration thread, each piece of code that fetches and uses the encryption key ID could be responsible for migrating the key into Barbican. However, there are multiple places where the code needs to handle the key ID, and each instance would need to be updated to perform the migration. This approach would involve significantly more code change, and would require more long-term maintenance.

We could also not migrate keys to Barbican at all, and simply allow the Castellan-Barbican enhancement to handle requests for the all-zeros key ID. However, this would require the user retain the fixed_key value in the configuration, which essentially misses the point of migrating to Barbican.

Data model impact

None. Existing database entries that contain the ConfKeyManager’s encryption key ID will be updated with references to a Barbican key ID, but the database structure will not change.

REST API impact

None.

Security impact

This change should be considered a security enhancement in that it facilitates transitioning existing ConfKeyManager deployments to Barbican.

Notifications impact

None.

Other end user impact

None.

Performance Impact

Minimal. The key migration thread will not be CPU intensive, and other cinder operations will not need to block until the migration thread completes.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: Alan Bishop<abishop@redhat.com>

Work Items

Develop code framework for scanning tables for encryption_key_id that needs to be migrated to Barbican
Develop utility code for generating new Barbican key ID with ConfKeyManager’s secret
Add/update unit tests
Update documentation

Dependencies

Castellan and/or Barbican, will be enhanced to aid the migration process. The enhancement will allow volumes encrypted with the ConfKeyManager’s key ID to function properly during the migration process.

Testing

Add and update the unit tests.

Documentation Impact

The documentation will be updated to state that volumes encrypted using the ConfKeyManager will continue to function correctly after switching to Barbican. The user needs to feel comfortable knowing the migration to Barbican is seamless and automatic.

The changes will explain the user is responsible for removing the ConfKeyManager’s fixed_key from the deployment, with the recommendation that the decision to do so be guided by the log messages generated by the key migration thread. Those message will indicate when keys are still being migrated, and when the migration process has finished.

References

None.

Add shared_targets column to volumes

Wed, 04 Oct 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/add-shared-targets-flag-to-volume

One problematic thing about dealing with device attachments is that some devices share a single target for multiple volumes while others have a single unique target for every volume.

For example, a device utilizing shared targets, may have a single iSCSI connection shared for multiple volumes attached to a Nova compute node.

This can be troublesome because extra care needs to be taken by consumers to make sure that the target it’s not being used by another volumes on a system when a delete/detach call is made.

Problem description

Currently when os-brick receives a detach operation it attempts to inspect the system and determine if there are multiple volumes attached to the host sharing the same target. Most of the time this works, however it tends to be a bit prone to races, additionally attempting bus scans is not only inefficient but unreliable.

The other problem is that currently there is no indication for os-brick to know if it’s dealing with a device that utilizes shared targets or not, so we go through this scanning routine even if we don’t necessarily need to.

The process of removing targets would be much more robust if we actually knew ahead of time if the volume was hosted on a device that utilizes shared targets, in which case we could do things more efficiently including using locks on a compute node around attach/detach operations for a specific backend.

Use Cases

This improves the existing use case of detaching volumes correctly and efficiently. The basic condition is that you have a device that utilizes shared-targets, and you have a single volume (V-A) from that device attached to a compute node. A user issues a detach call and at the same time another user issues an attach of a different volume (V-B) from the same backend that will land on the same compute node.

It’s possible that the connection response for V-B is completed under the assumption that the target exists (due to V-B) but then the V-A detach process performs it’s checks prior to the attachment of V-B being finalized, and as a result it removes the target and the connection of V-B will appear to have been successful but the volume will not be accessible.

Proposed change

Add a column to the volume reference that indicates if the backend associated with the volume utilizes shared-targets or not. In addition add a backend-name member to the volume detail view. When shared-targets is True, the consumer will know that they’re dealing with a device that uses shared-targets and that it needs to take some extra precautions.

In addition we need to provide a unique identifier for the backend, while at the same time not leaking details of the abstraction to end users. The service UUID should work nicely for this.

For the case of folks that use a single device to back multiple c-vol services we’ll provide the ability to over-ride/set this identifier in the config file. That way an admin can choose to set multiple c-vol configurations to use the same backend_id if needed.

Note that by default we will set the shared_targets flag to True. We use this as the default because there’s no harm or functional issues with treating unique targets as shared, it’s just going to introduce some unnecessary locking but will not impact functionality. Conversely if a device that actually uses shared targets is False and not locked we have a race condition between attach and detach calls that causes problems.

Alternatives

The scan method we use today works most of the time and is reasonable. We could certainly just continue to use that by itself.

Data model impact

This change adds a new boolean column to the Volumes table (shared_targets).

What new data objects and/or database schema changes is this going to require?

This introduces a new column to the Volumes Table
What database migrations will accompany this change.

We’ll perform a DB migration that adds the column and sets its value to True. Setting to True is safe for non-shared targets so while it’s not necessarily the most efficient way to do things, it’s safe and reliable.
How will the initial set of new data objects be generated, for example if you need to take into account existing volumes, or modify other existing data describe how that will work.

During init of the volume service, we’ll query the backend capabilities for the setting “shared_targets”, if the value from the stats structure is False we’ll check any volumes that are set as True and update them accordingly. This way we migrated everything and set it to true, then on service init we verify that the setting is actually correct.

NOTE that we’re already iterating over the volumes of a backend on init, so we’re not introducing any new overhead or lag here.

REST API impact

This change does not impact the arguments or options for any of the existing API calls, however it does add additional fields to the Volume Get response.

When the appropriate micro-version is selected, we’ll add two additional fields to the detailed volume response view:

shared_targets backend_name

Security impact

Notifications impact

Other end user impact

This change mostly just provides the user with additional info. It’s up to the user/consumer to determine if they want to do anything with this extra info or not, but it does not change their current use-cases or workflow.

Performance Impact

Other deployer impact

Developer impact

Be default we’ll set the shared_targets flag for volumes to True, but driver maintainers should add the appropriate stats field to their driver to change this if they don’t use shared_targets.

Implementation

Assignee(s)

Primary assignee:: john-griffith

Work Items

Add the changes to Cinder and bump the max support MV in cinderclient.

Dependencies

Testing

Add a functional test for the specific microversion and ensure the appropriate response.

Documentation Impact

Documenting the new fields in the detailed volume response, and also recommend how it can be used. Or just reference this spec.

References

Use the enginefacade from oslo_db

Sat, 30 Sep 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/use-oslodb-enginefacade

Implement the new oslo.db enginefacade interface described here:

https://blueprints.launchpad.net/oslo.db/+spec/make-enginefacade-a-facade

Problem description

The linked oslo.db spec contains the details of the proposal, including its general advantages to all projects. In summary, we transparently track database transactions using the cinder.RequestContext object. This means that if there is already a transaction in progress we will use it by default, only creating a separate transaction if explicitly requested.

Use Cases

These changes will only affect developers.

Allow a class of database races to be fixed

Cinder currently only exposes db transactions in cinder/db/sqlalchemy/api.py, which means that every db api call is in its own transaction. Although this will remain the same initially, the new interface allows a caller to extend a transaction across several db api calls if they wish. This will enable callers who need these to be atomic to achieve this, which includes the save operation on several Cinder objects.

Reduce connection load on the database

Many database api calls currently create several separate database connections, which increases load on the database. By reducing these to a single connection, load on the db will be decreased.

Improve atomicity of API calls

By ensuring that database api calls use a single transaction, we fix a class of bug where failure can leave a partial result.

Make greater use of slave databases for read-only transactions

The new api marks sections of code as either readers or writers, and enforces this separation. This allows us to automatically use a slave database connection for all read-only transactions. It is currently only used when explicitly requested in code.

Proposed change

Decorate the cinder.RequestContext class

cinder.RequestContext is annotated with the @enginefacade.transaction_context_provider decorator. This adds several code hooks which provide access to the transaction context via the cinder.RequestContext object, including attributes ‘session’, ‘connection’, ‘transaction’.

Ensure that all database apis use context session

Use context session in cinder.db.sqlalchemy.api.model_query().

Update database apis incrementally

Individual calls will be annotated as either readers or writers. Existing transaction management will be replaced.

Database apis will be updated in batches, by function. For example, Volume apis, Snapshot apis, Quota apis. Calls into apis which have not been upgraded yet will continue to explicitly pass the session or connection object.

Alternatives

Alternatives were examined during the design of the oslo.db code. The goal of this change is to implement a solution which is common across OpenStack projects.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

By reducing connection load on the database, the change is expected to provide a small performance improvement. However, the primary purpose is correctness.

Other deployer impact

None

Developer impact

The initial phase of this work will be to implement the new engine facade in cinder/db/sqlalchemy/api.py only. Callers will not have to consider transaction context if they do not currently do so, as it will be created and destroyed automatically.

This change will allow developers to explicitly extend database transaction context to cover several database calls. This allows the caller to make multiple database changes atomically.

Implementation

Assignee(s)

Primary assignee:: Hanxi_Liu <hanxi.liu@easystack.cn>

Work Items

Enable use of the new api in Cinder
Migrate api bundles along functional lines:
- Backup, BackupMetadata
- Cluster
- ConsistencyGroup, Cgsnapshot
- DriverInitiatorData
- Encryption
- Group, GroupSnapshot, GroupTypes, GroupTypeSpecs, GroupTypeProjects, GroupVolumeTypeMapping
- ImageVolumeCacheEntry
- Message
- QualityOfServiceSpecs
- Quota, QuotaClass, QuotaUsage, Reservation
- Service
- Snapshot, SnapshotMetadata
- Transfer
- VolumeAttachment, AttachmentSpecs
- Volume, VolumeMetadata, VolumeAdminMetadata, VolumeGlanceMetadata
- VolumeTypes, VolumeTypeProjects, VolumeTypeExtraSpecs

Dependencies

A version of oslo.db including the new enginefacade api:

https://review.openstack.org/#/c/138215/

Testing

This change is intended to have no immediate functional impact. The current tests should continue to pass.

Documentation Impact

None

References

Implement the oslo.db enginefacade interface described here:

https://blueprints.launchpad.net/oslo.db/+spec/make-enginefacade-a-facade

Use the new oslo_db enginefacade in Nova:

https://blueprints.launchpad.net/nova/+spec/new-oslodb-enginefacade

Use the new oslo_db enginefacade in Neutron:

https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch

Support get volume metadata summary

Thu, 31 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/metadata-for-volume-summary

Support get volumes’ metadata through volume summary API.

Problem description

Currently, Cinder supports filter volumes with metadata. But in some case, users don’t know what metadata all the volumes contain or what metadata is valid to filter volumes. Then users need to show the volumes one by one to get the correct metadata, this is really a heavy and unfriendly way. Imaging that if there are hundreds of volumes, admin users will take a very long time to query all metadata.

Use Cases

1. For users, they can get all the metadata easily just through one API request. 2. For dashboard, such as Horizon, it can use this metadata information to show end users a dropdown list.

Proposed change

DB layer change:
All the volumes’ metadata can be got by the sql query.
API layer change:
Add a new micro version. Add “metadata” to volume-summary API response body. The body will be like:
"metadata": {"key1": ["value1"],"key2": ["value2", "value3"]}

Alternatives

Leave as it is. Let the operators get volumes metadata by themselves through some peripheral ways. Such as, create a script to call volume-list-detail API and then analyse the result one by one.

Data model impact

None

REST API impact

A new microversion will be created.

Cinder-client impact

Now both OpenStackClient and CinderClient don’t support volume-summary command. We can add them as well.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There is a little performance influence about volume-summary API since a new sql query action will be added.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: wangxiyuan(wangxiyuan@huawei.com)

Work Items

Add a new microversion.
Add volumes’ metadata to the volume-summary API’s response body.
Add client side support.

Dependencies

None

Testing

Add unit tests.

Documentation Impact

Update API documentation.

References

None

cinder db purge utility

Wed, 16 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/database-purge

This spec adds the ability to sanely and safely purge deleted rows from the cinder database for all relavent tables. Presently, we keep all deleted rows, or archive them to a ‘shadow’ table. I believe this is unmaintainable as we move towards more upgradable releases. Today, most users depend on manual DB queries to delete this data, but this opens up to human errors.

The goal is to have this be an extension to the cinder-manage db command. Similar specs are being submitted to all the various projects that touch a database.

Problem description

Very long lived Openstack installations will carry around database rows for years and years. To date, there is no “mechanism” to programmatically purge the deleted data. The archive rows feature doesn’t solve this.

Use Cases

Operators should have the ability to purge deleted rows, possibily on a schedule (cronjob) or as needed (Before an upgrade, prior to maintenance) The intended use would be to specify a number of days prior to today for deletion, e.g. “cinder-manage db purge 60” would purge deleted rows that have the “deleted_at” column greater than 60 days ago

Project Priority

Low

Proposed change

The proposal is to add a “purge” method to DbCommands in cinder/cinder/cmd/manage.py This will take a number of days argument, and use that for a data_sub match Like: delete from instances where deleted != 0 and deleted_at > data_sub(NOW()…)

Alternatives

Today, this can be accomplished manually with SQL commands, or via script. There is also the archive_deleted_rows method. However, this won’t satisfy certain data destruction policies that may exist at some companies.

Data model impact

None, all tables presently include a “deleted_at” column.

REST API impact

None, this would be run from cinder-manage

Security impact

Low, This only touches already deleted rows.

Notifications impact

None

Other end user impact

None

Performance Impact

This has the potential to improve performance for very large databases. Very long-lived installations can suffer from inefficient operations on large tables.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

primary author and contact.: Abel Lopez <al592b>
Primary assignee:: <al592b>
Other contributors:: <al592b>

Work Items

Add purge functionality to manage.py db/api.py db/sqlalchemy/api.py Add tests to confirm functionality Add documentation of feature

Dependencies

None

Testing

The test will be written as such. Three rows will be inserted into a test db. Two will be “deleted=1”, one will be “deleted=0” One of the deleted rows will have “deleted_at” be NOW(), the other will be “deleted_at” a few days ago, lets say 10. The test will call the new function with the argument of “7”, to verify that only the row that was deleted at 10 days ago will be purged. The two other rows should remain.

Documentation Impact

will need to add documentation of this feature

References

This was discussed on both the openstack-operators mailing list and the openstack-developers mailing lists with positive feedback from the group.

http://lists.openstack.org/pipermail/openstack-dev/2014-October/049616.html

Add connector object to create_export call

Wed, 16 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-create-export-connector

There are a good number of volume drivers in Cinder today, that are using initialize_connection to do the work of exporting a volume as a target to an initiator (nova compute host). One of the reasons that those drivers are doing that, is because the create_export method API doesn’t include the connector object being passed in. This spec outlines adding the connector object to the standard volume driver API for create_export.

Problem description

There are many drivers in Cinder today that are doing volume target exports inside of initialize_connection, instead of the create_export method. One of the reasons for this is a documented understanding of what the differences between these 2 methods is supposed to be. The other problem that prevents drivers from doing the work inside of create_export, is that create_export is missing the connector object coming from the initiator. Some backends require the connector object, which contains the initiator information, in order to export a target from their array. So, drivers end up moving all of that exporting code into initialize_connection, because that’s the only driver call that has the connector object.

The other issue with Cinder drivers doing the exporting of volume targets in initialize_connection, is that Nova calls initialize_connection repeatedly during other processes not involved with attaching a volume, such as live migration. Nova will call a driver’s initialize_connection simply to fetch the connection_information about an exported volume, not because it wants to attach a volume.

Use Cases

The main use case is attaching a volume. When someone calls Cinder to attach a volume, they eventually call the volume manager’s initialize_connection method. The volume manager’s initialize_connection method first calls create_export, with the intention of asking the driver to create a target that is exported from the array to an initiator. Then the volume manager calls the driver’s initialize_connection to fetch the target information to pass back to the caller, to discover the volume showing up.

Proposed change

The proposed change is simple. Add the already existing connector object in the call to create_export. This makes the connector object universally available to all drivers during create_export time. Then each driver can eventually be changed by each maintainer to do the target exporting, instead of during initialize_connection time.

Alternatives

We can do nothing, and ask that each driver maintainer be more careful about doing any creating of exports inside of initialize_connection if they already exist. This basically makes create_export useless and a noop for all of those drivers. I’d rather see every driver use each method as they were intended, so to make reviewing drivers more consistent across all of Cinder.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

If driver developers move the creating of the exports into create_export, then any calls into initialize_connection should be faster, depending on the driver of course.

Other deployer impact

None

Developer impact

Once the connector object is available inside of create_export, then it’s possible for drivers to migrate their existing export creation code from initialize_connection into create_export. Reviewers should also be looking for this in existing and future reviews, and not approving drivers that ignore create_export.

Implementation

Assignee(s)

Primary assignee:: walter-boring

Work Items

First is to update the base driver.py’s create_export method. Then each other driver that defines create_export will have to accept the new connector object.

Then the volume manager call to create_export will need to pass the connector.

Dependencies

None

Testing

Any and all Unit tests will need to be updated to note the new connector object parameter. Driver’s unit tests will need to be updated as well.

Documentation Impact

None

References

A lot of research has gone into this, due to the efforts of trying to make Nova live migration with cinder volumes work. There is an existing etherpad that talks about the known issues of Nova, Cinder interaction. That etherpad also lists out the Cinder drivers that have potential problems with live migration due to creating exports inside of initialize_connection.

https://etherpad.openstack.org/p/CinderNovaAPI

Cinder volume driver for Huawei SDSHypervisor

Wed, 16 Aug 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/huawei-sdshypervisor-driver

This proposal is to add Huawei SDSHypervisor driver to cinder.

Huawei SDSHypervisor is a storage virtualization solution, is a software running in host, just focus on the data plane, the goal is to facilitate the reuse of customer’s existing old and low-end devices. SDSHypervisor does not interact with the storage device in control plane, such as create volume, create snapshot, do not need third-party device manufacturers to develop any driver. Administrator just attaches Lun to the hypervisor node as hypervisor storage entity and hypervisor will provide virtual volume based user’s QoS and patch advance feature to these Lun such as snapshot, link clone, cache, thin provision and so on.

The purpose of this blue print is to add Huawei SDSHypervisor Driver to Cinder.

Currently there are many heterogeneous storage devices in user product environment which is low-level device without advance feature, e.g. snapshot, clone, cache, thin provision, QoS. So adding the SDShypervisor driver will bring the following advantages: * Better I/O performance using SDSHypervisor cache feature when there is SSD in host. * Facilitate the reuse of customer’s existing old and low-end devices, patch advance feature to these device via SDSHypervisor, e.g. snapshot, clone, cache, thin provision, QoS * Help customer use Openstack build his cloud OS. For Cinder, we can’t allow driver to unlimited expand. For manufacturers, they may not want to spend more time to develop cinder driver for these obsolete and not be maintained devices. But in the customer’s environment there are such devices which need to be accessed to openstack, in this situation customers can access these devices to hypervisor, indirect access to Cinder, use Openstack build his cloud OS.

Problem description

Currently, user can’t access Huawei SDShypervisor by Openstack Cinder.

Use Cases

Proposed change

We will add a new Cinder driver that uses socket API interact with Huawei SDShypervisor storage. SDShypervisor data panel using private Key value protocol, so we also add a new connector to realize attach/detach volume.

The following diagram shows the command and data paths.

                +------------------+
                |                  |
                |  Cinder +        |
                |  Cinder Volume   |
                |                  |
                |                  |
                +------------------+
                |                  |
                |                  |
                |                  |
                |                  |
+---------------+-------+     +----+------------------+
|                       |     |                       |
+                       |     |                       |
|     SDShypervisor     |     | SDShypervisor Driver  |
|       connector       |     |                       |
|                       |     |                       |
+-----------------------+     +-----------------------+
                   |           |
                   |           |
                   |           |
                  CLI     Socket API
                   |           |
                   |           |
                   |           |
                +--+-----------+---+
                |                  |
                |                  |
                |  SDShypervisor   |
                |    storeage      |
                |                  |
                +------------------+

Add new driver in /cinder/volume/drivers path, and realize cinder driver minimum features: * Volume Create/Delete * Volume Attach/Detach * Snapshot Create/Delete * Create Volume from Snapshot * Get Volume Stats * Copy Image to Volume * Copy Volume to Image * Clone Volume * Extend Volume

Add new connector in cinder/brick/initiator path, and realize abstract connector methods: * connect_volume * disconnect_volume

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

User will be able to use Huawei SDSHypervisor with Cinder.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: zhangni <zhangni@huawei.com>
Other contributors:: None

Work Items

Realize Cinder driver minimum features using socket API. Realize new connector using CLI. Add unit test code for Huawei SDShypervisor cinder driver and connector.

Dependencies

Because Huawei SDShypervisor data panel using private Key-Value protocol, we will create a new libvirt volume driver in Nova to realize attach/detach volume to instance. Nova BP page is https://blueprints.launchpad.net/nova/+spec/huawei-sdshypervisor-volume-driver

Testing

3rd party continuous integration will be done for Huawei SDSHypervisor Driver.

Documentation Impact

The CinderSupportMatrix table and Block storage manual should be updated to add Huawei SDShypervisor.

References

https://wiki.openstack.org/wiki/Cinder/HuaweiSDSHypervisorDriver

Add QoS capability to the IBM Storwize driver

Sun, 13 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-storwize-driver-qos

Storwize driver can provide the QoS capability with the parameter of IO throttling, which caps the amount of I/O that is accepted. This blueprints proposes to add the QoS support for Storwize driver.

Problem description

Storwize backend storage can be enabled with the QoS support by limiting the amount of I/O for a specific volume. QoS has been implemented for some cinder storage drivers, and it is feasible for the storwize driver to add this feature as well.

Use Cases

Proposed change

To enable the QoS for storwize driver, an extra spec with IOThrottling as the key needs to bind to a volume type. All the following changes apply to the Storwize driver code.

Create volume:

If QoS is enabled and IOThrottling is available as the QoS key, set the I/O throttling of the volume into the specified IOThrottling value.

Create clone volume:

If the QoS is set for the original volume, the target volume needs to set to the same QoS.

Create snapshot:

The QoS attributes will be copied to the snapshot.

Create volume from snapshot:

If QoS is enabled and IOThrottling is available as the QoS key, set the IO throttling of the volume into the specified IOThrottling value.

Re-type volume:

If a volume type is changed, a different IOThrottling value will apply to the volume and the I/O of the volume needs to be set to the new IO throttling.

Alternatives

The proposed change follows the pattern, in which other drivers implement the QoS feature.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Other deployer impact

The QoS for Storwize driver can be configurable with a configuration option in cinder.conf. An extra spec with the key of IOThrottling can bind to a volume type, so that the volumes with this volume type are guaranteed with an I/O throttling rate.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: Vincent Hou
Other contributors:: TBD

Work Items

Add a configuration option to allow the user the use the QoS for Storwize driver.
Add the QoS enablement check and set the I/O throttling in create volume.
Add the QoS enablement check and set the I/O throttling in create volume from snapshot.
Set the I/O throttling to the new volume according to the original volume in create clone volume.
Copy the QoS attributes to the snapshot in create snapshot.
Change the I/O throttling of the volume if the volume type is changed.

Dependencies

None.

Testing

Unit tests need to be added to test the QoS for Storwize driver.

Documentation Impact

Add how to configure the QoS for the Storwize driver in the document.

References

Add QoS capability to the IBM storwize driver https://blueprints.launchpad.net/cinder/+spec/cinder-storwize-driver-qos

Oracle ZFS Storage Appliance iSCSI Driver

Sun, 13 Aug 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/oracle-zfssa-cinder-driver

ZFSSA ISCSI Driver is designed for ZFS Storage Appliance product line (ZS3-2, ZS3-4, ZS3-ES, 7420 and 7320). The driver provides the ability to create iSCSI volumes which are exposed from the ZFS Storage Appliance for use by VM instantiated by Openstack’s Nova module.

Problem description

Currently there is no support for ZFS Storage Appliance product line from Openstack Cinder.

Use Cases

Proposed change

iSCSI driver uses REST API to communicate out of band with the storage controller. The new driver would be located under cinder/volume/drivers/zfssa, and it would be able to perform the following:

Create/Delete Volume
Extend Volume
Create/Delete Snaphost
Create Volume from Snapshot
Delete Volume Snapshot
Attach/Detach Volume
Get Volume Stats

Additionally a ZFS Storage Appliance workflow (cinder.akwf) is provided to help the admin to setup a user and role in the appliance with enought privileges to do cinder operations. Also, cinder.conf has to be configured properly with zfssa specific properties for the driver to work.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

User will be able to use ZFS Storage Appliance product line with Openstack Cinder.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: juan-c-zuluaga <juan.c.zuluaga@oracle.com>
Other contributors:: brian-ruff <brian.ruff@oracle.com>

Work Items

All the features that ZFS Storage Appliance iSCSI does. Add CI unit test for ZFS Storage Appliance iSCSI Cinder Driver

Dependencies

Minimum ZFS Storage Appliance with OS8.2

Testing

CI will be performed for ZFS Storage Appliance iSCSI Driver.

Documentation Impact

Cinder Support Matrix should be updated. https://wiki.openstack.org/wiki/CinderSupportMatrix

References

http://www.oracle.com/us/products/servers-storage/storage/nas/overview/index.html

ZFS Storage Appliance Workflow.

http://docs.oracle.com/cd/E26765_01/html/E26397/maintenance__workflows.html#maintenance__workflows__bui

Support Volume Num Weighter

Sun, 13 Aug 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/support-volume-num-weighter

Provide a mean to help improve volume-backends’ IO balance and volumes’ IO performance.

Problem description

Currently cinder support choosing volume backend according to free_capacity and allocated_capacity. Volume Num Weighter is that scheduler could choose volume backend based on volume number in volume backend, which could provide another mean to help improve volume-backends’ IO balance and volumes’ IO performance.

Explain the benefit from volume number weighter by this use case.

Assume we have volume-backend-A with 300G and volume-backend-B with 100G. Volume-backend-A’s IO capabilities is the same volume-backend-B IO capabilities. Each volume’s IO usage are almost the same. Use CapacityWeigher as weighter class.

Concrete Use Case: If we create six 10G volumes, these volumes would placed in volume-backend A. All the six volume IO stream has been push on volume-backend-A, which would cause volume-backend-A does much IO scheduling work. At the same time, volume-backend-B has no volume and its IO capabilites has been wasted.

If we have volume number weighter, scheduler could do proper initial placement for these volumes—-three on volume-backend A, three on volume-backend-B. So that we can make full use of all volume-backends’ IO capabilities to help improve volume-backends’ IO balance and volumes’ IO performance.

Use Cases

Proposed change

Implement a volume number weighter:VolumeNumberWeighter.

_weigh_object function return volume-backend’s non-deleted volume number by using db api volume_get_all_by_host.
Add a new config item volume_num_weight_multiplier and its default value is -1, which means to spread volume among volume backend according to volume-backend’s non-deleted volume number.

Since VolumeNumberWeighter is mutually exclusive with CapacityWeigher/AllocatedCapacityWeigher and cinder’s scheduler_default_weighers is CapacityWeigher, we could set scheduler_default_weighers=VolumeNumberWeighter in /etc/cinder/cinder.conf and restart cinder-scheduler to make VolumeNumberWeighter effect.

VolumeNumberWeighter, whichi provides a mean to help improve volume-backends’ IO balance and volumes’ IO performance, could not replace CapacityWeigher/AllocatedCapacityWeigher, because CapacityWeigher/AllocatedCapacityWeigher could be used to provide balance of volume-backends’ free storage space when user focus more on free space balance between volume-bakends.

Alternatives

None.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: ling-yun

Work Items

Implement Volume Number Weighter
Add weighter option of Volume Number Weighter to OPENSTACK CONFIGURATION REFERENCE

Dependencies

None

Testing

Set up volume-backend-A with 300G and volume-backend-B with 100G. Create six 10G volumes, the expected result is 3 volumes in volume-backend A and 3 volumes in volume-backend B.

Documentation Impact

Add weighter option of Volume Number Weighter to OPENSTACK CONFIGURATION REFERENCE.

References

None

XtremIO storage Cinder volume driver

Sun, 13 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/xtremio-cinder-volume-driver

This is to enable OpenStack to work on top of XtremIO storage.

Problem description

This is a new Cinder driver that would enable Open Stack to work on top of XtremIO storage. The following diagram shows the command and data paths.

+----------------+                 +--------+---------+
|                |  Command        |                  |
|                |  Path           |  Cinder +        |
|  Nova          +---------------> |  Cinder Volume   |
|                |                 |                  |
|                |                 |                  |
+-----+----------+                 +--------+---------+
      |                                     |
      |                                     |
      |                                     |
      |                                     |
      |                                     |  +------------------+
      |                                     |  |                  |
 Command                                    +--+                  |
 Path +                                        |  XtremIO Driver  |
      |                                        |                  |
      |                                        |                  |
      |                                        +------+-----------+
      |                                               |
      |                                               |
      |                                               +
      |                                        XtremIO Rest API
      |                                               |
      v                                               |
                                                      |
+----------------+                                    |    +-----------------+
|                |                                    |    |                 |
|  Compute       |                                    |    |                 |
|                |                                    +---->    XtremIO      |
|                |           Data Link                     |    storeage     |
|                +-----------------------------------------+                 |
+----------------+                                         +-----------------+

Use Cases

Proposed change

2 new volume drivers for iSCSI and FC should be developed, bridging Open stack commands to XtremIO management system (XMS) using XMS Rest API. The drivers should support the following Open stack actions:

Volume Create/Delete

Volume Attach/Detach

Snapshot Create/Delete

Create Volume from Snapshot

Get Volume Stats

Copy Image to Volume

Copy Volume to Image

Clone Volume

Extend Volume

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

This feature will use native snapshot so the user can expect great speed up to all snapshot/clone related actions.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: shay-halsband
Other contributors:: None

Work Items

Implement REST API client to XMS Implement Logic for each functionality to support both iSCSI and FC

Dependencies

Testing

Continuous integration as required for all drivers in the Juno timeframe

Documentation Impact

Add documntation on how to install and use the drivers.

References

Add DB table for driver private data

Sun, 13 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/driver-private-data

As discussed at the mid-cycle meetup (https://etherpad.openstack.org/p/cinder- meetup-winter-2015) we want to add a DB table where drivers can store info they need to operate. A specific use case is for iSCSI CHAP credentials for initiators, but it should be generic enough for other drivers to store data they depend on too.

Problem description

Currently there is not a standard solution for drivers to store information they need to operate. For things volume related we have some “provider” fields on the volume table that can be used, but for things not associated with the volume’s life-cycle they are insufficient. This has lead to some drivers storing information on the backend which works for some but not all backends have this capability.

A use case for this is storing target CHAP secrets so iSCSI initiators can do CHAP authentication against targets.

Use Cases

Proposed change

A solution for this problem is to have a new table in the Cinder database which can hold initiator information for drivers. The model would look like:

+--------------+--------------+--------------------------------------------+
|   Field      |     Type     |            Description                     |
+--------------+--------------+--------------------------------------------+
| created_at   | datetime     |                                            |
| updated_at   | datetime     |                                            |
| id           | int(11)      | Auto incremented ID                        |
| initiator    | varchar(255) | Unique constraint with "key" +             |
|              |              | "namespace"                                |
| namespace    | varchar(255) | Unique constraint with "key" + "initiator" |
|              |              | This comes from cinder.conf for each back  |
|              |              | end configuration.                         |
| key          | varchar(255) | Unique constraint with "initiator" +       |
|              |              | "namespace"                                |
| value        | varchar(255) | This could be anything a driver wants to   |
|              |              | keep track of for this initiator. In the   |
|              |              | CHAP credential usecase this could be a    |
|              |              | barbican key uuid.                         |
+--------------+--------------+--------------------------------------------+

In initialize_connection for VolumeManager if there is an initiator (in the case of iSCSI) we can look up the rows for it filtering by the namespace and pass them in to the driver as a new optional parameter for initialize_connection.

The namespace field will come from a new configuration option on the driver base class called driver_data_namespace. This config option will default to None. If this option is not set, we will check the backend_name config option, and finally if that is not set, we fall back to the driver class name. This will allow for being able to share data between multiple different backends and a single backend in HA configuration.

The driver then can return a model update dictionary as an “initiator_update” field on the connection info. These changes will then be applied to the initiator table.

The initiator_update dictionary can contain “set_values” dictionary and/or “remove_values” list. The set_values are key value pairs which are to be set or updated in the database. The remove_values is a list of keys to be deleted from the database. They will be “hard” deleted so the data will not remain in the database once specified in remove_values.

The return value from initialize_connection might look somthing like:

{
    "driver_volume_type": "iscsi",
    "data": {
        "target_iqn": "iqn.2010-06.com.purestorage:flasharray.12345abc",
        "target_portal": 1.2.3.4:5678,
        "target_lun": 1,
        "target_discovered": True,
        "access_mode": "rw",
        "auth_method": "CHAP",
        "auth_username": "someUserName",
        "auth_password": "somePassword"
    },
    "initiator_updates":
        {
            "set_values": {
                "purearray1ChapSecretId": "someUUID",
                "someOtherThing": "someOtherValue"
            }
            "remove_values": [
                "myOldKey",
                "anotherKey"
            ]
        }
}

The manager’s initialize_connection method can then pull the updates out of the return object, do the database operations (as needed) and pass on the connection info with the field stripped out so it does not propagate through the system.

Alternatives

These key values pairs are scoped for the initiator, so there is a chance for two different backends to get access to the values. This is helpful for backends that are configured in active/active HA, but there is a risk of two different vendors being used by the same initiator and having key collisions. We could introduce a vendor-prefix field to use as part of the key as well to avoid that.
- We could also use the cinder host to scope these instead of a vendor prefix
We could make drivers keep track of this outside of Cinder. This could be left up to a driver backend to store data, or with another external database somewhere. Some drawbacks to this approach include having different databases to maintain, update, and keep in sync which would be a less than desirable user experience for a cloud admin, having multiple ways of doing synchronization/locking in drivers can and will introduce bugs and maintenance problems, and makes it more difficult for new driver developers to make a good choice on where to store data.
We could allow the drivers direct(ish) access to the database through some get/set api’s to store arbitrary key-value pairs. Some alternatives for the database table in that model include:
- Have a column for the host name and using it as part of the unique constraint for the “key”. The benefit being that it is much less likely to have naming collisions between drivers using this. The downside is that two hosts using the same driver may not be able to easily share data, which makes active/active HA for the same backend difficult.
- Have a column for the driver class name and using it as part of the unique constraint for the “key”. Again the benefit being to help avoid collisions with key fields. This one is better for sharing between hosts that are using the same driver type, but the issue then is where does the name come from? If it is reported by the driver as a string it might as well be any generic prefix as there is nothing to stop someone from just making up their name. If it is automatically gathered from the driver class via python internal variables on the object then there are issues with any driver that changes their class name later on.
- Have a column that is just a provider/vendor namespace and is part of the unique constraint for the “key”. Yet again the goal is to avoid collisions with key fields. Similar to the last one the only downside is where it comes from and that there isn’t really anything that would prevent drivers from using the same one, but it would be one more layer of protection.
- Have no restriction and let the keys be more like a global scope, this allows drivers to share data easily, but has issues with collisions.
Return a tuple instead of adding the data to the return object of initialize_connection. This requires having an optional second return value and conditional calling/return values for methods, or modifying every usage of driver.initialize_connection(…).

Data model impact

The proposed solution will be adding in a new database table named driver_initiator_data and associated model DriverInitiatorData. There will be a migration to add and remove the table but will not seed or modify any data.

Access to the database will be used through api’s outlined in the Proposed Change.

Table rows will be hard deleted, so we don’t keep around potentially sensitive credentials/information.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Potential impact for drivers that heavily utilize this table with extra database queries.

Other deployer impact

New config option to potentially set for each backend.

Developer impact

Drivers that store data on their backend may want to utilize this feature.

This change will require modifying the signature for all drivers initialize_connection method implementations.

Implementation

Assignee(s)

Primary assignee:: patrick-east
Other contributors:: None
Interested parties:: walter-boring

Work Items

Add new Model
Add new db API’s
Add new db Migration scripts
Modify the volume manager’s initialize_connection method to query/save data

Dependencies

None

Testing

DB migration tests
DB API tests
BaseDriver unit tests
No new tempest tests

Documentation Impact

None

References

Support for incremental backup in Cinder

Sun, 13 Aug 2017 00:00:00

Launchpad Blueprint:: https://blueprints.launchpad.net/cinder/+spec/incremental-backup

Problem description

The current implementation of Cinder Backup functionality only supports full backup and restore of a given volume. There is no provision to backup changes only that happened since last backup. As the volumes grow bigger and over, all changes to volumes between backups stay relatively small, copying entire volumes during backups will be resource intensive and do not scale well for larger deployments. This specification discusses implementation of incremental backup feature in detail.

Use Cases

Proposed change

Cinder backup API, by default uses Swift as its backend. When a volume is backed up to Swift, Swift creates a manifest file that describes the contents of the backup volume. The manifest file contains header (metadata) and array of pointers to the volume backup files. Since Swift has an upper limit on the object size, Cinder backup API splits the volume data into individual chunks of Swift object size and uploads these individual chunks to Swift. Cinder volume backup manifest file includes these list of objects, their corresponding objects, the logical offset of each object within the volume and a message digest of each chunk to detect any unwarranted changes to objects. During restore operation, Cinder reconstructs the volume based on the manifest and individual chunks referenced in the manifest.

To support incremental backup functionality, we introduce another object called shafile to the list of backup files. shafile file helps track changes to the volume since last backup. This new object holds SHA256s of the volume. A brief description of SHA-2 or SHA256 can be found at http://en.wikipedia.org/wiki/SHA-2. The backup manifest file will have a reference to this object. During a full backup operation, Cinder divides up the volume into fixed blocks of user configurable block size. It calculates SHA256 of each block and compiles a list of SHAs and uploads the shafile to the backup container.

To keep the incremental backup implementation simple, an incremental operation is only performed with respect to a full backup. During incremental backup, cinder reads the shafile of the full backup. It creates a new shafile from the current volume data and compares the new shafile with full backup shafile to calculate the blocks that are changed since last full backup.

We will use existing manifest mechanism to capture the delta. Since full backups do not contain any holes, offset+lengths of each chunk of the volume describe the full length of the volume logical address. However with incremental backup, this model is challenged and the offset/chunk of individual files become sparse. The absence of offset/length in a manifest represents the data that is not modified since last backup. One potential drawback of this approach is if changes to volume are fragmented, incremental backup may result in too many objects in Swift. However object stores like Swift are built to handle many small objects effectively.

The new shafile is uploaded as part of the incremental backup.

The manifest header identifies this backup as incremental backup and hence contains a reference to the full backup container.

Following changes are made to the manifest header of the backup

metadata['version'] = self.DRIVER_VERSION
metadata['backup_id'] = backup['id']
metadata['volume_id'] = volume_id
metadata['backup_name'] = backup['display_name']
metadata['backup_description'] = backup['display_description']
metadata['created_at'] = str(backup['created_at'])

# Changes to metadata section of manifest
metadata['shafile'] = <shafilename>  # Path to shafile name. Or
                                     # can be hardcoded to "shafile"
                                     # in the container

metadata['backup_type'] = "incrementa/full" # backup type
metadata['full_container'] = <object path> # path of full backup

Restore API is not expected to change, however restore implementation will be changed to handle incremental backup. To keep the restore from incremental backup simple and easy to test, the restore operation first performs restore of the full volume from the full backup copy and then apply incremental changes at offset and length as described in the incremental backup manifest.

Snapshot based backups:

Since existing backup implementation copies the data directly from the volume,
it requires the volume to be detached from the instance. For most cloud
workloads this may be sufficient but other workloads that cannot tolerate
prolonged downtimes, a snapshot based backup solution can be a viable
alternative. Snapshot based backup will perform a point in time copy of the
volume and backup the data from point in time copy. This approach does not
require volume to be detached from the instance. Rest of the backup and
restore functionality remain the same.

As an alternative, snapshot based backup can be implemented by extending
existing backup functionality to snapshot volumes. This approach can be lot
more simpler than backup API taking snapshot of the volume and then managing
the snapshots.

Alternatives

Incremental backup offers two important benefits:

Use less storage when storing backup images
Use less network bandwidth and improve overall efficiency of backup process in terms of CPU and time utilization

The first benefit can be achieved as a post processing of the backup images to remove duplication or by using dedupe enabled backup storage. However the second benefit cannot be achieved unless Cinder backup supports incremental backup.

Data model impact

No perceived data model changes

REST API impact

No new APIs are proposed. Instead existing backup API will be enhanced to accept additional option called “–incr” with <path to full backup container>” as its argument.

cinder backup-create <volumeid> --incr <full backup container>
  Performs incremental backup

cinder backup-create <volumeid> --snapshot
  Optionally backup-create will backup a snapshot of the volume. Snapshot
  based backups can be performed while the volume is still attached to the
  instance.

cinder backup-create <volumeid> --snapshot --incr <full backup container>
  Optionally backup-create will perform incremental backup from volume
  snapshot

No anticipated changes to restore api

Security impact

None

Notifications impact

None

Other end user impact

python-cinderclient will be modified to accept “–incr” option. It may include some validation code to validate if the full backup container path is valid

Currently backup functionality is not integrated with OpenStack dashboard. When it happens, the dashboard will provide an option for user to choose incremental backup

Performance Impact

Except for calculating SHAs during full backup operation, there is no other performance impact on existing API. The performance penalty can be easily offset by the efficiency gained by incremental backup. Also new hardware support CPU instructions to calculate SHAs which alleviates some stress on the CPU cycles.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: muralibalcha(murali.balcha@triliodata.com)

Other contributors: giribasava(giri.basava@triliodata.com)

Work Items

python-cinderclient That accepts “–incr” option and some validation code
cinder/api Which parses the “–incr” option
cinder/backup/api.py backup api signature is modified
cinder/backup/manager.py
cinder/backup/driver/swift.py Heavy lifting is done here. Both backup and restore apis will be modified.

Dependencies

None

Testing

Unit tests will be added for incremental backup.

Testing will primarily focus on the following:

SHA file generation
Creating various changes to the original volume. These include

Changes to first block

Changes to last block

Changes to odd number of successive blocks

Changes to even number of successive blocks

Changes spread across multiple sections of the volume

Perform 1 incremental
Peform multiple incremental backups
Restore series of incremental backups and compare the contents
Perform full backup, then incremental, then full and then incremenal restore the volume from various backups.

Documentation Impact

Need to document new option in the block storage manual.

References

None

Over Subscription in Thin Provisioning

Sun, 13 Aug 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/over-subscription-in-thin-provisioning

This proposal is to introduce a mechanism to allow over subscription in thin provisioning and also use reserved percentage to prevent over provisioning.

Problem description

There are a couple of problems in capacity reporting in thin provisioning:

Some drivers are still using ‘infinite’ and ‘unknown’ to report capacity. This could lead to over provisioning.
There’s no mechanism to allow over subscription in thin provisioning. Over subscription allows flexibility in storage allocation and usage and it is an important concept in providing thin provisioning support.

Terminologies

The following terminologies will be used in this spec:

total_capacity: This is an existing parameter already reported by the driver. It is the total physical capacity. Example: Assume backend A has a total physical capacity of 100G.

free_capacity: This is an existing parameter already reported by the driver. It is the real physical capacity available to be used. Example: Assume backend A has a total physical capacity of 100G. There are 10G thick luns and 20G thin luns (10G out of the 20G thin luns are written). In this case, free_capacity = 100 - 10 -10 = 80G.

free: This is calculated in the scheduler by subtracting reserved space from free_capacity.

volume_size: This is an existing parameter. It is the size of the volume to be provisioned.

provisioned_capacity: This is a new parameter. It is the apparent allocated space indicating how much capacity has been provisioned. Example: User A created 2x10G volumes in Cinder from backend A, and user B created 3x10G volumes from backend A directly, without using Cinder. Assume those are all the volumes provisioned on backend A. The total provisioned_capacity will be 50G and that is what the driver should be reporting.

allocated_capacity: This is an existing parameter. Cinder uses this to keep track of how much capacity has been allocated through Cinder. Example: Using the same example above for provisioned_capacity, the allocated_capacity will be 20G because that is what has been provisioned through Cinder. allocated_capacity is documented here to differentiate from the new parameter provisioned_capacity. If a driver can guarantee that the backend is used solely for a specific Cinder installation, then provisioned_capacity could be the same as allocated_capacity. Only driver has knowledge of this and therefore driver can make a decision on how to report provisioned_capacity.

max_over_subscription_ratio: This is a new parameter. It is the float representation of the over subscription ratio when thin provisioning is involved. It is a ratio of provisioned capacity over total capacity. Default ratio is 1.0, meaning provisioned capacity cannot exceed the total capacity. Note that this ratio is per backend or per pool depending on driver implementation.

reserved_percentage: This is an existing parameter. It represents the percentage of backend capacity that is reserved. It ranges from 0 to 100. Default is 0. Note this refers to the real capacity. This is per backend or per pool depending on driver implementation.

reserved: This is a ratio of reserved_percentage over 100 calculated by the scheduler.

virtual_free_capacity: This is a parameter calculated based on other parameters. It refers to how much space a user can still provision apparently. Note that this is different from free_capacity. free_capacity is about the real available physical capacity. virtual_free_capacity is about apparent available virtual capacity. Example: Assume the total capacity of backend A is 100G. The max over subscription ratio is 2.0. If no volumes have been provisioned yet, the virtual_free_capacity is 100 x 2.0 = 200. If 50G volumes have already been provisioned, the virtual_free_capacity is 200 - 50 = 150.

Use Cases

Proposed change

New parameters in get_volume_stats

One new configuration options “max_over_subscription_ratio” will be added to cinder/volume/driver.py. This configuration option will be added to cinder.conf. It can be configured for each backend when multiple-backend is enabled.

Note: This configuration option is provided as a reference implementation and will be used by the LVM driver. However, it is not a requirement for a driver to use this option from cinder.conf. Driver can choose its own way if that makes more sense.

In scheduler/host_manager.py, the following additional information sent by each backend will be saved and will be used later by the scheduler to make decisions. Note these are reported for a backend or a pool, depending on driver implementation. In the LVM driver, these will be reported for a backend (which is treated as one pool).

provisioned_capacity
max_over_subscription_ratio

There is a change on how reserved_percentage is used. It was measured against free capacity in the past. Now it will be measured against total capacity.

Note: The configuration option max_over_subscription_ratio added in cinder.conf is for configuring a backend. For a driver that supports multiple pools per backend, it can report these ratios for each pool in get_volume_stats.

Driver can get this option in cinder.conf and report the same ratio for the pools that belonging to the same backend.
Alternatively driver can choose to report different ratio for each pool in get_volume_stats, without using this option in cinder.conf.

Capabilities

Driver can report the following capabilities for a backend or a pool:

thin_provisioning_support = True (or False) thick_provisioning_support = True (or False)

Two capabilities are added here to allow a backend or pool to claim support for thin provisioning, or thick provisioning, or both.

Volume type extra specs

If volume type is provided as part of the volume creation request, it can have the following extra specs defined:

‘capabilities:thin_provisioning_support’: ‘<is> True’ or ‘<is> False’ ‘capabilities:thick_provisioning_support’: ‘<is> True’ or ‘<is> False’

Note: ‘capabilities’ scope key before ‘thin_provisioning_support’ and ‘thick_provisioning_support’ is not required. So the following works too:

‘thin_provisioning_support’: ‘<is> True’ or ‘<is> False’ ‘thick_provisioning_support’: ‘<is> True’ or ‘<is> False’

The above extra specs are used by the scheduler to find a backend that supports thin provisioning, thick provisioning, or both to match the needs of a specific volume type.

If an extra spec scope key “provisioning:type” is defined, it can be used by the driver to detemine whether the lun to be provisioned is thin or thick. The value of this extra spec is either “thin” or “thick”. Note this extra spec is not used by the scheduler to find a backend.

Capacity filter

In the capacity filter, the following will be evaluated in the decision making when choosing a backend that fits the criteria:

If (provisioned_capacity + volume_size) / total_capacity >= max_over_subscription_ratio, the backend will not be chosen to provision the volume. Note: This formula will be executed only if “thin_provisioning_support” is True and max_over_subscription_ratio >= 1.

If ((free_capacity - total_capacity * reserved) * max_over_subscription_ratio) < volume_size, the backend will not be chosen to provision the volume. Note: This formula will be executed only if “thin_provisioning_support” is True and max_over_subscription_ratio >= 1.

If (free_capacity - total_capacity * reserved) < volume_size, the backend will not be chosen to provision the volume. Note this check was already in the capacity filter, but the formula is changed to use total_capacity * reserved instead of free_capacity * reserved.

Capacity weigher

In the capacity weigher, virtual_free_capacity should be used for ranking if “thin_provisioning_support” is True. Otherwise, real free_capacity will be used as before. A change is made to measured reserved space against the total_capacity. virtual_free_capacity = total_capacity * max_over_subscription_ratio - provisioned_capacity - total_capacity * reserved

LVM driver

In the default LVM driver, changes will be made in get_volume_stats which periodically reports capabilities and the information will be received by the scheduler.

Changes will be made in the LVM driver to report provisioned_capacity. It makes calls to the LVM class in brick to retrieve volume information including capacities.
The LVM driver will also report max_over_subscription_ratio. This will be from the configuration parameters set in cinder.conf.
While other drivers need to report max_over_subscription_ratio, they are not required to read those ratios from cinder.conf.

Changes will also be made in the following LVM driver functions to make sure over provisioning will not happen even when a request didn’t go through the scheduler:

create_volume
extend_volume

The following will be evaluated in the above LVM driver functions:

If the ratio of the apparent provisioned capacity over real total capacity has exceeded the over subscription ratio, the operation will fail.
If the free space is smaller than the volume size, the operation will fail.

Use cases

The design of this feature will support the following use cases.

Use case 1: Each volume type has a separate backend or pool. For example, Gold volume type uses pool gold, Silver volume type uses pool silver, and Bronze volume type uses pool bronze. Each pool can have a different max over subscription ratio.

Use case 2: One volume type is associated with multiple backends or pools. For example, Silver volume type uses pool 1 and pool 2. Both pools can have the same max over subscription ratio. Note that capacities for each pool can be different at any given time.

Use case 3: One backend or pool is used by multiple volume types. For example, pool 3 is used by volume types Gold, Silver, and Bronze. Assume Gold volume type uses thick luns only, Silver volume type can have either thick or thin luns, and Bronze volume type has thin luns only. Because the over subscription ratio is calculated by the ratio of provisioned_capacity over total_capacity and all three volume types are sharing the same pool, the ratio will be the same for all volume types. Gold volume type can guarantee its space reservation by creating thick luns. The apparent size and the used size of a Gold volume will always be the same. For a thin lun created as Silver or Bronze volume type, the apparent size can be bigger than the real size. Some detailed examples are shown at this etherpad: https://etherpad.openstack.org/p/cinder-over-subscription-white-board

Alternatives

Without this, we cannot support over subscription in thin provisioning and there’s also no upper limit that prevents over provisioning from happening.

Data model impact

N/A

REST API impact

N/A

Security impact

N/A

Notifications impact

If the capacity usage has exceeded the used ratio or if the provisioned capacity has exceeded the over subscription ratio, a notification should be sent. The notification should report the name of the backend or pool and the capacity information from the backend or pool. The purpose of the notification is for the storage administrator to take notice and take actions to fix the problem.

Notification will also be sent periodically whenever the scheduler receives an update of capacities from the backend. This will be consumed by Ceilometer. This was discussed for the Capacity Headroom topic at the summit. The Ceilometer team will be responsible for the changes required on the Ceilometer side for this.

Other end user impact

There is a new parameter in cinder.conf that end user needs to be aware of.

Performance Impact

N/A

Other deployer impact

New parameters over_subscription_ratio will be added to cinder.conf.

Developer impact

Drivers should report provisioning capabilities (thin_provisioning_support and thick_provisioning_support).

Drivers supporting thin provisioning should report provisioned capacity in addition to free capacity in get_volume_stats.

For drivers supporting thick provisioning only, free capacity will be used just as before.

For drivers supporting both thin and thick provisioning, provisioned capacity and free capacity should both be reported.

If there is a range regarding capacity and you are not sure how to report, please be conservative. For example, if the available capacity is in the range of 80 to 100 GB, be conservative and report the lower bound 80 GB.

Driver developers can take a look of _update_volume_stats in the LVM driver as a reference implementation.

Note: This work is also needed for Cinder to use ThinLVM as the default driver in Kilo.

Implementation

Assignee(s)

Primary assignee:: xing-yang

Other contributors:

Work Items

Add max_over_subscription_ratio in driver.py.
Modify host_manager.py to update provisioned capacity, over subscription ratio by the backends.
Modify capacity filter to check whether over subscription ratio has been exceeded in a backend.
New parameters max_over_subscription_ratio will be added to cinder.conf.
LVM driver will be changed to report virtual capacity and over subscrption ratio.
LVM class in brick will be updated to calculate provisioned capacity.
LVM driver functions will be changed to check whether over subscription ratio has been exceeded.

Dependencies

N/A

Testing

New unit tests will be added to test the changed code. Testing will be done using the LVM driver for thin provisioning. Testing will be done to cover the 3 use cases described above.

Documentation Impact

Documentation changes are needed for the following: New parameter max_over_subscription_ratio will be added to cinder.conf. Driver needs to add provisioning capabilities (thick_provisioning_suppot, thin_provisioning_support) and report provisioned_capacity.

References

Examples: https://etherpad.openstack.org/p/cinder-over-subscription-white-board

Virtual capacity “provisioned_capacity_gb” was discussed in Winston’s spec https://review.openstack.org/#/c/105190/6/specs/juno/volume-statistics-reporting.rst

Kilo design summit session on this topic: https://etherpad.openstack.org/p/kilo-cinder-over-subscription https://etherpad.openstack.org/p/kilo-cinder-capacity-headroom

Documentation on the filter scheduler: http://docs.openstack.org/developer/nova/devref/filter_scheduler.html Note: This is a document on Nova filter scheduler, but it is very similar to the Cinder filter scheduler.

RemoteFS Config Improvements

Sun, 13 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/remotefs-share-cfg-improvements

RemoteFS drivers (NFS, GlusterFS, etc.) are currently configured by adding a list of shares to a text config file which is referenced by cinder.conf. This means that one driver instance manages a handful of storage locations for the driver. This work will a) have these drivers configured like most other Cinder drivers are, and b) leverage the Cinder scheduler for selection between different storage backends rather than having the driver act as a pseudo-scheduler.

Problem description

The configuration system for NFS/GlusterFS/etc drivers:

is different from other drivers
is more complex than necessary
limits functionality such as migration

Use Cases

Proposed change

Replace the <x>_shares_config setting with settings that can be used to login to the storage platforms. This means that an nfs_shares_config file such as:

192.168.1.10:/export1 -o sync
192.168.1.11:/export2 -o vers=nfs4

would become, in cinder.conf:

[nfs1]
address = 192.168.1.10
export_path = /export1
options = -o sync

[nfs2]
address = 192.168.1.11
export_path = /export2
options = -o vers=nfs4

Each Cinder backend will then only manage one export rather than a handful of exports. This brings the RemoteFS drivers closer to how other Cinder drivers operate.

Alternatives

Leave things as they are today. (Not desirable.)

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

It will be possible to use Cinder volume migration to move volumes between all NFS exports when previously this was not always possible (since the different exports were managed by the same driver instance).

Performance Impact

None

Other deployer impact

nfs_shares_config, glusterfs_shares_config, etc., will be deprecated (but still functional for Kilo). Setting the new options will cause these settings to be ignored.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: eharney
Other contributors:: Other interested parties?

Work Items

Create new options for address, export, mount options
Mark options and code to be removed in L as deprecated in Kilo

Dependencies

None

Testing

The NFS driver and GlusterFS drivers will be gaining CI during the Kilo cycle which will cover this.

Manual testing should cover both the current and new configuration paths.

Documentation Impact

New configuration options and possibly guide changes for configuring the NFS driver.

References

None

Backup Snapshots

Sun, 13 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/backup-snapshots

Currently we can backup a volume, but not a snapshot. This spec proposes to provide a way to backup snapshots.

Problem description

Today we can backup a volume, but we cannot directly backup a snapshot. The ability to backup a volume directly is valuable because it allows a volume to be backed up in one step. Also volume backup has been available ever since backup was introduced in Cinder. Users also take snapshots from the volumes as a way to protect their data. These snapshots reside on the storage backend itself. Providing a way to backup snapshots directly will allow the user to protect the snapshots taken from the volumes on a backup device, separately from the storage backend.

Use Cases

There are users who have taken many snapshots and would like a way to protect these snapshots. This proposal to backup snapshots provides another layer of data protection.

There are other projects in OpenStack focusing on data protection such as Freezer, Smaug, Raksha, etc. They are all in different stages of design, development, or adoption. The backup API in Cinder is not a replacement of those projects which are doing a full blown orchestration of data protection for all resources in OpenStack (not just block storage). Instead, Cinder APIs can be consumed by those higher level projects for data protection and can also be used directly by users who do not need those higher level projects.

Proposed change

In summary, the following changes will happen:

A field snapshot_id will be added to the request body of the existing backup API.
A new column snapshot_id will be added to the backups table.
The backup_volume logic in driver.py and lvm.py will be changed to use the snapshot_id if it is not None.
The logic to find the latest parent during the incremental backup will be adjusted to take into account backups from snapshots.

Note: No new driver API will be introduced in this proposal.

Steps to create a backup from snapshot are as follows by default using the internal tenant:

Create a temporary volume from the snapshot
Attach the temporary volume
Do backup from the temporary volume
Detach the temporary volume
Cleanup temporary volume

If the driver has implemented the attach snapshot interface introduced in the Liberty release (see the developer impact section), backing up a snapshot will be done using the following steps:

Attach the snapshot
Do backup from the snapshot
Detach the snapshot

For incremental backups, because the latest parent is calculated automatically by looking at the timestamps, the logic has to be changed to accommodate backups from snapshots. For backups from snapshots, we need to look at the timestamps of the snapshots; for backups from volumes, we still look at the timestamps of the backups as before. The parent of an incremental backup of a snapshot could be a backup from a previous snapshot or a backup from the volume, depending on the timestamps when the previous snapshot was taken vs when the volume backup was taken. The backup with the latest timestamp will be chosen as the parent. A new column will be created to record the timestamp of the data in the backups table. For a backup from volume, the data timestamp field will be the same as the created_at field in the backups table. For a backup from snapshot, the data timestamp field will be the same as the created_at field of the snapshot.

Alternatives

Here is a manual alternative:

Create a volume from the snapshot
Backup the volume
Delete the volume

Data model impact

Add the following new column to the backups table for snapshot id. This field will be null if the backup is from a volume:

snapshot_id = Column(String(36))

Add the following new column to the backups table to record the timestamp of the data:

data_timestamp = Column(DateTime)

Note that the following column will still be required for a backup from snapshot:

volume_id = Column(String(36), nullable=False)

REST API impact

Change the existing create backup API to take a snapshot id. Either volume_id or snapshot_id has to be provided for the create backup API, but not both. The snapshot_id is required for backing up a snapshot.

Create backup

V2/<tenant id>/backups
Method: POST

JSON schema definition for V2:

{
    "backup":
    {
        "display_name": "nightly001",  # existing
        "display_description": "Nightly backup",  # existing
        "volume_id": "xxxxxxxx",  # existing
        "snapshot_id": "xxxxxxxx",  # new
        "container": "nightlybackups",
        ......
    }
}

Security impact

None

Notifications impact

Currently notifications are sent out when a backup is created, restored, and deleted. The notification data needs to be updated with the snapshot_id if necessary.

Other end user impact

End user will be able to create a backup from a snapshot.

Performance Impact

No obvious performance impact.

Other deployer impact

The deployer will be able to backup a snapshot.

Developer impact

All volume drivers will get the backup from snapshot feature with this proposal. No additional changes are required.

If a driver wants to use a more optimal way by attaching the snapshot, it can implement the following interfaces that were added in the Liberty release to support non-disruptive backups:

initialize_connection_snapshot
terminate_connection_snapshot
create_export_snapshot
remove_export_snapshot

The following function can also be overridden by the driver which returns False by default:

backup_use_temp_snapshot

Note: All of the driver APIs specified above were added in the Liberty release. No new driver APIs are introduced by this spec.

Implementation

Assignee(s)

Primary assignee:: <xing-yang>
Other contributors:: <None>

Work Items

Make changes to the backup API to support backup snapshot.
Make changes to the backups db table to add a snapshot_id column.
Make changes to the backup_volume function in driver.py and lvm.py to support backing up a snapshot.
Make changes to the incremental backups to take into account backups created from snapshots.
Make sure the code has good comments to explain different code paths.

Dependencies

None

Testing

Unit tests and tempest tests will be provided.

Documentation Impact

Documentation will be modified to describe how to use this feature. We will make sure both the existing use cases and the new use cases are clearly documented to avoid any confusion. The following should be covered:

Do a full backup of a volume with status being ‘available’ or ‘in-use’.
Do an incremental backup of a volume with status being ‘available’ or ‘in-use’.
Do a full backup of a snapshot.
Do an incremental backup of a snapshot.

Developer documentation should also be created to explain how the different backup cases are handled and how it would impact the developers working on drivers.

References

Code is submitted here: https://review.openstack.org/#/c/243406/

Add API to os-brick to allow extending an attached volume

Sun, 13 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/brick-extend-attached-volume

Add extend_volume API to os-brick connector objects to allow updating the host kernel size information for an attached volume.

Problem description

Currently, Cinder has the ability to extend volumes that aren’t attached. Unfortunately, this doens’t work for attached volumes, due to the host kernel that the volume is attached to, won’t see the new size unless actions are taken on the host. The linux kernel doesn’t automatically see the size changing on the remote storage server. This spec outlines the idea of adding those required actions on the host server when an attached volume is extended in size.

Use Cases

An OpenStack user has a Cinder volume attached to a host/VM and they want to extend/grow the volume while it’s attached.

Proposed change

Add a new os-brick Connector api called “extend_volume” that does the work on the host to get the volume size updated. This will also have to take into account the raw device and the multipath device, if available. It will return the size of the volume after the resize is complete or None if it failed.

This spec doesn’t cover the work needed by Cinder and Nova to make the solution complete. This spec simply covers the work needed by os-brick, to ensure the volume that’s attached to the nova compute host can update it’s host kernel to see the volume’s new size. Cinder will have to change, to allow attached volumes to be extended, and when complete call a new Nova API to initiate the work on the compute host. Then Nova will have to issue a blockresize in virt to get the VM’s virtual device resized. For the volume types such as RBD, where Nova doesn’t use os-brick as the library for managing the volume, Nova will have to do the work of issuing the commands to extend the volume itself.

Alternatives

Unfortunately, if a volume is attached to a VM, there is nothing an end user can do, because the Nova compute host kernel has to see the changes before the VM can see the new size.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

The end user, that owns the VM will may have to issue a filesystem resize to get the native filesystem on the volume to recognize the new volume size. For ext* style filesystems, users can use resize2fs.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: walter-boring

Work Items

Add the new API to the base Connector class and then add the common required code to the linuxscsi.py to do the real work.

Dependencies

In order for this to be used by the OpenStack end user, Cinder will have to optionally allow for attached volumes to be extended.
There will also need to be some Nova work done to initiate the call into os-brick’s new API to do the work, and then notify the VM after it’s successful.

Testing

Until the end to end Cinder -> Nova capability is in place, there isn’t a way to automate the testing of this. For now, it’s a manual process.

Documentation Impact

References

A related cinder-spec that has been up for a while here:

https://review.openstack.org/#/c/200627/

Related information on how to manually do the work of notifying the host kernel:

https://help.ubuntu.com/lts/serverguide/multipath-admin-and-troubleshooting.html

Scalable Backup Service

Sun, 13 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/scalable-backup-service

Because cinder backup workloads run as a dedicated service, they have potential to be run independently of other cinder volume services. Ideally, backup services would be fired up on demand in the cloud as backup workloads are generated, and elastically de-commissioned as workloads go away. Ideally, backup tasks would farmed out to the least busy of the available backup services.

Pursuit of this ideal is blocked today by a tight coupling of cinder backup service to cinder volume service.

This spec is dedicated to breaking this tight coupling of the cinder backup and volume services.

That is the first step in working towards a truly elastic, horizontally scalable backup service. When this coupling is loosened, multiple backup services can be fired up concurrently and run without any requirement that they they be colocated with the volume services or with one another for that matter.

We expect that there will be followup work to address subsequent steps such as scheduler support for backup tasks and for dynamic generation of backup service processes using VMs or containers rather than dedicated physical nodes, but those subjects would be addressed by followup specs and are not in scope here.

One thing at a time.

Problem description

When the backup service backs up or restores a volume, it uses a backup driver specific to a chosen and configured backup repository to interact with e.g. a swift or Ceph object store, an NFS, glusterfs, or generic POSIX filesystem, or Tivoli Storage Manager. This backup driver plugs in to the backup manager and provides functionality to put data in the backup repository or to retrieve it from the repository. A particular backup service process has only one backup driver and therefore only one backup repository. It may, however, perform backup and restore operations for volumes for any of the enabled backends handled by the volume service.

The backup service requires backend-specific functionality to attach to any particular volume in order to read it for backup or to write to it for restore.

Today, this backend-specific information is provided as follows. When the backup service process starts up, the backup manager loads the volume manager module for each enabled backend configured in cinder.conf. When an OpenStack tenant or administrator makes a request to create a backup from a cinder volume or to restore a backup to a cinder volume, the backup api looks at the host field for the volume in question and directs the rpc for the backup or restore operation to the backup manager on that node. There, the backup manager selects the volume manager for the relevant volume backend and invokes its driver’s backup_volume or restore_volume method, passing that method the backup object and the backup driver (swift, ceph, NFS, etc.) for the configured backup repository. Then the volume driver attaches to the volume in question, opens it, and invokes the backup driver’s backup or restore method using the file handle from the volume driver’s attach and open operation.

Thus today the backup manager does not invoke its backup driver methods directly; it finds an appropriate volume driver via the appropriate volume manager, and the volume driver invokes the appropriate backup driver. The volume driver by definition runs on the cinder node that runs the volume service for the relevant backend, so since the backup manager, volume driver, and backup driver code all run in the context of a single operating system process on a single node, the backup service is necessarily tied to the node that also runs the volume service.

When multiple pools and backends run on the same node, this means that concurrent backups end up running into the resource constraints of a single node. Since backup tasks do not flow through a scheduler, they typically also run with the resource constraints of a single operating system process.

Use Cases

An OpenStack tenant or administrator needs to do backups on a schedule or restores on demand such that to get the work done in a timely manner many backups and restores need to run concurrently.
Resource requirements to handle peak backup and restore loads exceed the capabilities of a standard physical node.

Note that the need for concurrent backup and restore operations, especially to the same backend, may be artificially suppressed historically because of the lack of support for live backups. We anticipate that now that backups of in-use volumes are supported ([3] [5]) this need will significantly increase.

Proposed change

Leverage remote attach capability to move the functionality provided by the volume driver into the backup manager. That way the backup manager can invoke the backup driver directly rather than having to load the volume manager at startup and run the relevant volume manager’s driver.

Nowadays attaching a volume is done using a brick library that builds an appropriate connector for the attachment given information about the backend (iSCSI, FibreChannel, NFS, whatever) obtained by running a volume driver initialize_connection method – either directly or via an rpc to the appropriate volume service. In the case of a remote attachment, initialize_connection is invoked by rpc. This is the only part of the workflow that actually requires the volume service, and since it can be an rpc invocation, this means that the direct load of volume manager and volume driver can be eliminated in the backup service itself and the backup service can therefore run on a different node than the node for the backend’s volume service.

Now that backups of in-use volumes are supported, volume drivers can supply an attach_snapshot method, which is then used as optimization instead of attaching a temporary volume copy of the source volume. In the initial implementation [6], an attach_snapshot method was added that only allows for local attaches and the reference lvm driver explicitly uses local_path when getting volumes for backup operations [5]. As part of the work implementing this blueprint spec, we will need to revisit snapshot attachment to allow for remote attaches. Drivers like lvm driver that cannot support remote attach for snapshots will need to fall back to using temporary volumes instead [5]. * Create a temporary volume from the original volume. * Backup the temporary volume. * Clean up the temporary volume.

Alternatives

Keep the existing scheme. Backup service continues to work but cannot scale out.

The node running backup service has to be scaled up to handle projected peak backup workload and is likely to be either underutilized or underpowered for most actual backup workloads.

Instead of loading the volume manager directly from the backup manager, expose backup_volume and restore_volume methods in the volume manager, have these invoke the corresponding volume driver’s methods by the same name, and have the backup manager use rpc to the volume service to trigger the volume manager’s backup_volume and restore_volume methods.

This approach would reduce memory requirements for the backup service process since it would no longer load the volume managers for all enabled backends. And the linkage between backup manager and volume service is now loosely coupled. However, the bulk of the work - data transfer, encryption, compression - is done by the backup driver, which still remains tightly coupled to the volume service.

What this specification does not solve

A backup task scheduler.

The backup api process can get a list of active backup services from the database and choose as an rpc destination e.g. the first service, make a random choice, or round-robin among the choices. This is where a call to a scheduler could go in the future, but a scheduler is not itself in scope for this spec.
Elastic backup service placement.

Backup services will be started more or less manually by an administrator or configured to start on boot of a node. One can imagine a dynamic mechanism for starting service VMs or containers triggered by the backup api as workloads arrive. The decoupling of backup and volume services addressed by this spec is a pre-condition for such an elastic backup service placement capability but it is only a small step towards enabling such a capability.

Service `init_host` cleanup

At startup, the current backup service code makes an attempt to discover and cleanup orphaned, incomplete backup and restore operations (e.g., they were in process when the backup process itself was terminated). The backup service assumes that it is the only backup process, so that if it finds backups in creating or restoring state at startup it can safely reset their state and detach the volumes that were being backed up or restored.

This assumption is not safe if multiple backup processes can run concurrently, and on separate nodes. At startup, a backup service needs to distinguish between in-flight operations that are owned by another backup-service instance and orphaned operations.

Eventually, it will make sense for a backup service process to cleanup stuff left behind either by earlier incarnations of itself or by other abnormally terminated backup processes. A solution to this general problem, however, requires a reliable capability to auto-fence oneself on connection loss as being developed as part of the solution for Active-Active HA for the cinder volume service [7].

Here we will align with the community decision at the Mitaka design summit to defer the auto-fencing capability and start on Active-Active HA for the cinder volume service without automatic cleanup, by restricting backup service initialization cleanup to leftovers from the same backup service.

The host field for a backup object will be set to the host for the backup service to which the backup operation is cast. The status update of the backup and host update will be handled in an transaction. Cleanup at initialization can then be restricted to leftover objects that chain through their corresponding backup object to a host field matching oneself. Compare-and-swap DB operation will be used to prevent race conditions.

For an example of how the host field will be set, consider a volume with a backend handled by volume service on node A where backup service processes are running on node B and node C. When a backup is created using the service on node B, the host field for the backup object will be set to B. When restoring from that backup using the backup service on node C, the host field for the backup object will be set to C.

Cleanup of associated volumes, temporary volumes, and temporary snapshots will be done via rpc to the appropriate volume service host.

Note that the backup object contains a volume_id field for the volume it backs up, as well as temp_volume_id and temp_snapshot_id fields for live backups, but it does not currently keep the id of volumes to which it is restoring backups. We will need to add this field in order to determine orphaned restore-operation volumes.

Special Volume Driver Backup/Restore Considerations

Since the functionality of the current volume driver backup_volume and restore_backup methods will in this proposal move into the backup manager, these methods will no longer be needed and can be removed from the codebase. That said, some volume drivers override these with methods that apparently have a bit more “special sauce” than just preparing their volume for presentation as a block device.

We will need to analyze the codebase to root out any of these and determine how to accommodate any special needs.

An example is the vmware volume driver [4], where a “backing” and temporary vmdk file are created for the cinder volume and the temporary vmdk file is used as the backup source. We will have to determine whether all this can be done in the volume driver’s initialize_connection method during attach, or whether we will require an additional rpc hook to a prepare_backup_volume method or some such for volume drivers of this sort.

Data model impact

None.

REST API impact

None.

Security impact

TBD.

We need to understand exactly where it is necessary to elevate privileges in running backup and restore operations and ensure that there is no unnecessary elevation above the normal privileges for the admin or tentant requesting the these operations.

Notifications impact

We should be able to do exactly the same backup service notifications as those done now.

Other end user impact

No change in function or client interaction.

Performance Impact

Backup process will be lighter weight since volume manager and drivers are no longer loaded.
The proposed change enables running multiple backup processes as required.

Other deployer impact

Backup service can now run on multiple nodes and no longer has to run on the same node as the volume service handling a volume’s backend.

Developer impact

Enables potentially valuable future features such as backup scheduler or elastic backup service placement.

Implementation

Assignee(s)

Primary assignee:

Tom Barron (tbarron, tpb@dyncloud.net)

Other contributors: * LisaLi (lixiaoy11, xiaoyan.li@intel.com) * Huang Zhiteng (winston-d, winston.d@gmail.com)

Work Items

Write the code. A POC is available now [1].
Determine and address any impact on existing volume drivers that have their own backup or restore methods.
Run/test the new code with multiple backup processes running on multiple nodes, other than the node or nodes where the volume services run for enabled backends.

Dependencies

None

Testing

Unit tests will be extended to cover new backup code for functionality formerly provided by volume driver.
Unit tests for backup manager and volume drivers will be modified to reflect code removed from the backup service to load volume manager, run volume drivers, etc. and from the volume driver to run backup and restore operations.
Existing tempest tests should provide sufficient coverage to ensure that current functionality does not regress. Potentially new multi-node tempest tests could be added to verify distributed interactions. We should take advantage of opportunities to extend current tempest coverage for backup and add functional tests for backup when this is feasible.

Documentation Impact

Update with new deployment options.

References

User facing “Failure” Message and Event Viewer

Sun, 13 Aug 2017 00:00:00

For quite some time, OpenStack services have wanted to be able to send messages, especially error messages, to API end users (by user I do not mean the operator, but the user that is interacting with the API).

If user performs some operation and operation fails or goes in hang state, then there must be some interface for user to see reason for this behavior.

So, this is basically regarding facilitating user to see error messages for asynchronous operations either directly through APIs or through new “Event Viewer” tab in horizon or horizon pluggin.

This is more specific to failed operations and cause of their failure.

https://blueprints.launchpad.net/cinder/+spec/summarymessage

Problem description

If operations like create volume, create snapshot etc fails, user gets no detailed information in case operation fails. Sometime only operation status is updated to error, failed etc without any update to user.

In some case it is worse than this. For ex, 1. In case rabbitmq is inactive and user tries to create volume, horizon hangs on waiting response from API forever. 2. In case rabbitmq is active but recipient service is not running and user tries to create volume, API hangs on waiting response from rabbitMQ forever.

A few resources among all the OpenStack projects handle reporting errors to end users for asynchronous operations and those that do are inconsistent with each other. In addition to a mechanism to enable error reporting in a consistent way across OpenStack, the solution must also be able to accommodate a deployment of Cinder that contains no other OpenStack service.

Use Cases

Motivation for this Blue Print:

To help admin in debugging failed operations with less log filtering.
To notify admin to start services which failed due to some abnormal conditions.
To inform user about failure in case operation fails.
To provide enough information to user about failure.

General Use Cases:

Cinder volume/snapshot creation goes to ERROR status due to lack of capacity. (Scheduling error)
Cinder add volume to CG fails, how to tell user the volume and group are not on the same backend?
Cinder volume goes from attaching to available. why?
Volume retype fails.
Volume extend fails, I’d like to know why and be able to still use my volume instead of it being in error_extending.
ETC..

Proposed change

User can fetch operation failure details through direct API calls or through new horizon tab “Event Viewer”. From CLI, cli client could be used to display same kind of information.

Suggested implementation is based on 2 way approach

Push Information: During user operations, messages will be pushed to database using component specific notifications.

These messages will be generated and pushed to database by component for operation start, operation completion or operation failure.

Message generation will be based on eventID to eventMessage mapping using message constant files which keeps different notification messages mapping in it. This way deployer can easily modify notification messages as per requirement.
Pull Information: In case user needs to check operation status, user can pull details using CLI client or Horizon tab.

Results may be shown in tabular way as shown below

Tenant	EventID	NodeName	ReqID	Level	Resource	Time	Summary
Sheel	UKN_ERR	BS-cind1	{…}	Error	Volume	{..}	{….}

Every ‘operation’ initiated by the user has a request ID returned as an HTTP header in the context. These notification messages will be tied to operation request ID. (This request ID will be used for mapping a request to what happened in cinder for that operation.)

Summary message will contain operation specific failure message. For ex, “Volume create operation failed - {Reason of failure}”

Filters:

Results can be filtered depending upon TenantID, HostID, UserID, Operation Outcome/Result(Fail/Pass) etc.

Type of Messages:

API Events : messages for failed operations.
Service Logs: information for failed services either stopped by user or stopped due to any abnormal conditions.
System Logs: any other logs than API and service logs.

Suggested Architecture:

The proposed change is to add a new /v3/<tenant>/messages API resource backed by a messages table in the Cinder DB. This endpoint will return a list of error messages that are intended for the end-user regarding failed asynchronous operations.

In short: * /v3/<tenant>/messages API resource, exposes notifications messages depending upon filters * message_ttl config option that dictates message minimum life in seconds * messages DB table

Questions

None

Alternatives

User facing notifications Use the existing notification framework in combination with an AMQP consumer to pull messages off and provide an endpoint for the user. Faults with this approach are that we do not want to display the current information in notifications to the user and it will require many more services as dependencies.
Per resource faults This alternative suggests adding a sub-resource to each resource, such as volumes/<volume-id>/faults, similar to Nova’s instance faults. This makes it difficult to poll for messages for more than a single resource or resource type. It also adds significant complexity to the api as every resource must add /faults in order to support messages.
Exposing user messages via a separate service (such as Zaqar) This approach suggests storing user messages in another service that the user could query for messages or the service could utilize webhooks to notify the user. One major drawback to this approach is the complexity in writing bindings for the separate service(s) and the need for a separate service as a dependency.

What this specification does not solve

State change notifications. This solution does not intend to solve the use-case of alerting users when a volume or any other resource changes state. For example, when a volume changes from creating to available.

REST API impact

New APIs: * GET /v3/<tenant>/messages With filters by attribute. Ex: GET /v3/<tenant>/messages?resource_type=volume * GET /v3/<tenant>/messages/<message-id> * DELETE /v3/<tenant>/messages/<message-id>

Message schema

Message:
  type: object
  required:
  - user_message
  - id
  - project_id
  - request_id
  - event_id
  - created_at
  - message_level
  - expires_at
  properties:
    id:
      type: string
      description: UUID will be stored in 'id' field.
    message_level:
      type: string
      enum:
      - ERROR
      description: The level of the message. In the future we may expand to
      sending information to the user that is not an error.
    user_message:
      type: string
    event_id:
      type: string
      description: Event ID can be used to
      a. update message text at deployer end for some specific situation.
      b. to report errors by user.
      c. to debug fast as it is easy to search where specific eventID is
      used for reporting error.
    resource_uuid:
      type: string
      description: The uuid of the offending resource.
    resource_type:
      type: string
      description: The type of resource this message pertains to.
      For ex, volume, snapshot, backup etc
    request_id:
      type: string
    created_at:
      type: string
    expires_at:
      type: string
      description: After this time the message may no longer exist

Data model impact

New messages table in the DB to store all messages. This table may prove to grow large in a cloud with lots of errors. The admin will be able to utilize the expires_at column to reap messages.

Security impact

Messages must be highly scrutinized before becoming visible to the user in order to avoid any sensitive data from being shown. This will be mitigated by having all user visible messages defined in a single module. The messaging mechanism will assert that any message it will create comes from the sanctioned location.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

New configuration option message_ttl that will dictate the number of seconds after the messages creation time to set the ‘guaranteed_until’ attribute on generated messages.
New configuration option message_reap_interval that will dictate the number of seconds between calls to delete old messages. A value of -1 will never run. DocImpact: This option should not be set on a large number of nodes, since too many nodes trying this delete at the same time will cause transaction bouncing and degraded DB performance.
New configuration option message_reap_batch_size that dictates the number of expired messages to delete each interval. This allows a deployer to limit DB performance impact by setting a ceiling for the number of messages deleted at a time.
The messages table will be potentially large and may be reaped based on the ‘guaranteed_until’ column. Where all messages with a expires_at date earlier than the current time can be safely deleted.

Developer impact

Developers should be aware of use-cases where the user needs information about an error. In these situations, an appropriate user message should be written and creation of the message added in the specific code path(s).

Implementation

Assignee(s)

Primary assignee:: Sheel Rana Alex Meade

Work Items

This whole implementation depends upon message generation, transport, collection, storage and analytics of different failure messages.

cinder: Implementation to generate notification messages at the time of failure for all existing operations.
cinder: notification listener is required which will serve as basis for handling event messages from different components.
cinder: collector is required to collect, validate and store event messages to database.
cinder: new API to fetch details form database depending upon filters.
cinder: Add pagination to messages
cinder-manage: Add mechanism to automatically, and via a cinder-manage command, reap expired messages in the db depending upon ttl value.
cinder: Documentation for new API details.
cinder: Update “Getting started Guide”.
cinder: Database schema preparation to store notification messages.
cinder: Need to implement “delete messages as per message life” from database after message expiry time. For ex, if user has set message_ttl to 7 days, then all messages older than 7 days will be purged from database.
horizon: Separate tab for cinder to display event messages.
cinder-client: cinder cli to communicate with API and fetch event messages.
cinder-client: Update to CLI reference Guide.
Tempest tests

Implementation Phases:

This whole feature will be implemented in multiple phases:

Phase 1. Basic implementation regarding notification generation and storage into database with “/messages” exposed to view notification messages. This spec targets Phase 1 first, other phases will be implemented after acceptance of phase 1.

Phase 2. Implementation for facilitating admin to configure notification storage like db or zaqar or both. If both RPC/DB are configured by admin, notification message would be stored in zaqar along with storing information to database.

Phase 3. Implementation for consuming information from zaqar directly.

Phase 4. Horizon and CLI implementations to view notifications in more formatted manner.

Phase 5. Handling of some special cases where generation of notifications requires separate handling like rabbitMQ related implementations for showing notifications in case rabbitMQ is in failed state or rabbitMQ recipient is in inactive state.

Dependencies

None

Testing

Tempest tests should be written and run in the gate. It may prove difficult to implement complete functional testing of the feature as messages will not be created unless there is an error, which may be difficult to trigger. However, some operations are easy to trigger failure with unlimited quotas. One example is creating a thick provisioned volume too big to be stored on the backend.

Example Test Cases

# List messages with no messages # Attempt creation of a TOO LARGE volume and verify appropriate scheduling error message is created # List messages with filters, especially resource_type

Documentation Impact

REST API documentation
New config option, message_ttl (time to live)
New config option, message_reap_interval (number of seconds between calls to delete old messages)
New config option, message_reap_batch_size (number of messages which could be deleted in one batch)
New API policies for messages

References

Mitaka Midcycle discussion: https://etherpad.openstack.org/p/mitaka-cinder-midcycle-user-notifications https://etherpad.openstack.org/p/mitaka-cinder-midcycle-day-1
Kilo Summit Discussion: https://etherpad.openstack.org/p/kilo-cinder-async-reporting
Liberty Summit Discussion (in conjunction with HEAT) -: https://etherpad.openstack.org/p/liberty-cross-project-user-notifications

Backup driver initialization

Sun, 13 Aug 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/backup-init

We don’t initialize Cinder Backup driver during service startup. It means that we’ve got cinder-backup service up and running even if it can’t connect to the backup storage.

Problem description

Cinder backup manager does not verify that backup driver is initialized successfully. If cinder-backup is started successfully we can create a volume backup. Such backups always will be in ‘error’ status and tenant user won’t be able to delete them.

Use Cases

Cinder backup service should be marked as ‘down’ if it can’t connect to the backup storage.

Proposed change

We should introduce for cinder backups the same mechanism as we’ve got for volume manager and drivers:

Introduce ‘init_host’ method in backup manager which will be called on service startup and verify that backup driver is initialized: verify driver’s configuration is correct, depends on driver, we could check connection to storage, list of available backups. etc.
If backup driver initialization fails, manager will mark backup service as ‘down’.

In case of initialization failure, cinder will try to do it periodically depends on ‘periodic_interval’ config option value.

Backup service should be initialized in ‘service_down_time’ time interval or will be marked as ‘down’.

Alternatives

Check for backup storage is available on backup create call. If storage is not available, remove ‘host’ field from backup object. We could try to re-schedule backup creation on the other host.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

New notifications for backup initialization failure and success will be added.

Other end user impact

User will be able to delete backup in error state if it was not created on backend.

Performance Impact

None

Other deployer impact

New ‘backup_periodic_initialization’ and ‘backup_initialization_timeout’ config option will be added. Deployer have to enable ‘backup_periodic_initialization’ if needed.

Developer impact

Backup driver developers should implement new APIs.

Implementation

Assignee(s)

Primary assignee:: Ivan Kolodyazhny <e0ne@e0ne.info>
Other contributors:: Backup drivers maintainers.

Work Items

Implement ‘do_setup’ method in a base backup driver which won’t do anything
Implement ‘do_setup’ in each backup driver.
Call driver’s ‘do_setup’ during backup-manager ‘init_host’ call.

Dependencies

None

Testing

Both unit and Tempest tests should be implemented to cover new feature.

Documentation Impact

None

References

https://bugs.launchpad.net/cinder/+bug/1598709

Support backup import on another Storage database

Fri, 28 Jul 2017 00:00:00

URL of launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/support-backup-import-on-another-storage-database

This backup service can use a backup extra metadata in order to import a backup on a different Block Storage.

Problem description

Currently a volume backup can only be restored on the same Block Storage service. This is because restoring a volume from a backup requires metadata available on the database used by the Block Storage service.
In order to import backup metadata on another Block Storage database (i.e disaster recovery site), one has to save the metadata of a volume backup available on the source database, and then replicate it together with the data to the other Block Storage site.
Combination of the local backup service that exports and stores this metadata, together with replication to another Block Storage site, allows you to completely restore the backup even in the event of a catastrophic database failure.
In addition, having a backup metadata together with a volume backup, also provides volume portability. Specifically, backing up a volume and exporting its metadata will allow you to restore the volume on a completely different Block Storage database, or even on a different cloud service.

Use Cases

When a user wants to save backup metadata together with volume backup for volume portability purpose, or for replication purpose to disaster recovery site, and be able to restore a volume from a backup on the other Block Storage site.

Proposed change

In order to support backup import on a different Block Storage database, we need to extend chunked backup driver:
- Enabling to save backup metadata together with the data.
- Add a cinder client api command, that will parse backup metadata from backup_metadata file, and import that metadata to the other Block Storage database.
- User will be able to restore a volume backup on the other Block Storage, using cinder backup restore api.

Alternatives

Support only storage on local storage. Use slow manual backup and restore methods.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: ronen-mesonzhnik
Other contributors:: None

Work Items

Implement get_extra_metadata that will return backup’s corresponding database information as encoded string metadata.
Implement a cinder client api command that can take a backup metadata file, and do the import from it.

Dependencies

None.

Testing

None.

Documentation Impact

In addition to the existing command ‘cinder backup-import <METADATA>’, there will be a command that can accept a file: ‘backup-import-record-from-backup-metadata-file <file_path>’

References

Link to Export and import backup metadata documentation: http://docs.openstack.org/admin-guide-cloud/blockstorage_volume_backups_export_import.html

Support metadata for backup

Wed, 12 Apr 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/metadata-for-backup

Add a new “metadata” property for backup resource.

Problem description

Backup resource lost the ability for getting/setting metadata property.

Use Cases

The metadata here for backup is the descriptive metadata. It’s used for discovery and identification. Users could add key-value pairs for the backups to describe them. And users can also filter backups with specified metadata.

Proposed change

The “metadata” property will be added to backup object.

2. A new DB table “backup_metadata” will be created.

-----------------------------------------------------------------------
| created_at | updated_at | deleted_at | id | backup_id | key | value |
-----------------------------------------------------------------------
|            |            |            |    |           |     |       |
-----------------------------------------------------------------------

The primary key is “id”.

3. The backup create/update API will be updated to support “metadata”.

POST /v3/{project_id}/backups
PUT  /v3/{project_id}/backups/{backup_id}

the request body can contain "metadata".
{
    "metadata":{
        "key1": "value1",
        "key2": "value2"
    }
}

4. A set of new APIs will be created. It’s used for backup metadata’s CRUD.

GET    /v3/{project_id}/backups/{backup_id}/metadata
show a backup's metadata

POST   /v3/{project_id}/backups/{backup_id}/metadata
create or replaces metadata for a backup

PUT    /v3/{project_id}/backups/{backup_id}/metadata
replace all the backup's metadata

GET    /v3/{project_id}/backups/{backup_id}/metadata/{key}
show a backup's metadata for a specific key

DELETE /v3/{project_id}/backups/{backup_id}/metadata/{key}
delete a specified metadata

PUT    /v3/{project_id}/backups/{backup_id}/metadata/{key}
update a specified metadata

Alternatives

Leave as it is.

Data model impact

Backup model will be updated with new property “metadata”.

REST API impact

The backup create/update API’s request body will be updated.
A set of new APIs related to backup metadata will be created.

Security impact

None

Notifications impact

The new APIs will send new notifications as well.

Other end user impact

None

Performance Impact

A new “backup_metadata” table will be created so that the DB conjunction action may let the search performance reduce a little.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: wangxiyuan(wxy)

Work Items

Add metadata property to backup object and bump its version.
Create a new DB table “backup_metadata” and add db upgrade script.
Update backup create/update API.
Add a tuple of new APIs for backup metadata.

Dependencies

None

Testing

Unit tests

Documentation Impact

Api-ref need update.

References

None

Capacity-based QoS

Wed, 29 Mar 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/capacity-based-qos

QoS values in Cinder currently are able to be set to static values. This work proposes a way to derive QoS limit values based on volume capacities rather than static values.

Problem description

This proposes a mechanism to provision IOPS on a per-volume basis with the IOPS values adjusted based on the volume’s size. (IOPS per GB)

Use Cases

A deployer wishes to cap “usage” of this system to limits based on space usage as well as throughput, in order to bill customers and not exceed limits of the backend.

Associating IOPS and size allows you to provide tiers such as

Gold: 1000 GB at 10000 IOPS per GB Silver: 1000 GB at 5000 IOPS per GB Bronze: 500 GB at 5000 IOPS per GB

Proposed change

Allow creation of qos_keys:: read_iops_sec_per_gb write_iops_sec_per_gb total_iops_sec_per_gb

These function the same as our current <x>_iops_sec keys, except they are scaled by the volume size.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

New optional qos spec values.

Off by default, opt-in.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: eharney

Work Items

https://review.openstack.org/#/c/447127/

Dependencies

Testing

Documentation Impact

Document new fields available in qos types.

References

Code: https://review.openstack.org/#/c/447127/

Client reset-state improvements

Wed, 15 Feb 2017 00:00:00

https://blueprints.launchpad.net/cinder/+spec/client-reset-state-improvements

Improve “reset-state” in the cinder client shell.

Problem description

The reset-state implementation in the client shell has a few issues that merit cleaning up:

We have and are adding an <x>-reset-state operation for many objects in Cinder. This results in numerous commands when we could consolidate these into one command. Consolidating things into a single command makes more sense because this is intended to be a rarely-used admin fix-up tool, and not a prominent feature of our client.

The current reset-state model also defaults to unsafe behavior. It will reset an object to “available” if a user runs it on an object without other parameters. This is not a safe path to use as a default because we have no way to know if the object is actually in an “available” state and resetting it to “available” will result in odd problems later.

Use Cases

This functionality addresses admins who need to repair broken things in Cinder to workaround bugs or failed operations.

Proposed change

Add a new argument to “cinder reset-state” so that it can handle all of the types of objects we would like to reset the state of.

See https://review.openstack.org/#/c/413156/

Require state to be specified rather than defaulting to “available”.

This is trickier to keep from breaking compatibility, but worthwhile. The defaulting is done in the client. We could handle this by adding a stricter check in the client using the microversion checks, so that using microversion 3.30+ requires the user to specify the desired state, but when using older API versions, the previous behavior is kept. This does not correspond with an actual API change on the server, but should work as a way to handle compat here.

Decide that we will no longer add <x>-reset-state operations to the client shell.
(Optional:) We should consider looking at attach-status and
migration-status resets as well. These are required to be specified manually but there may be cases where it would be more consistent to handle this for the caller.

i.e. does it ever make sense to reset a volume to “available” and not clear its attach status?

Alternatives

Leave things as they are and keep adding new reset operations to the client which default to hazardous behavior.
Start fixing things in Cinder that require use of reset state in the first place. (We should do this, but it’s a long, hard, project, and out of scope here.)

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

The cinderclient CLI changes.

Performance Impact

None

Other deployer impact

None

Developer impact

We should design Cinder to not require use of reset-state as a “regular” thing.
Currently there’s a model, for some operations, of:
1. Request operation
2. Operation fails
3. User is notified that the operation failed by the object now being in “error” state.
4. For some operations, the admin is now expected to reset the state back to “available” to keep things functioning.
This could be done as:
1. Request operation
2. Operation fails
3. Object is put back into previous state instead of “error”
4. Client polls for operation status via our async messaging
5. User now knows what happened and the admin doesn’t have to perform a reset. The object is left in its “true” state.
Should consider what the intersection of reset-state and force-detach looks like.

Implementation

Assignee(s)

Primary assignee:: eharney
Other contributors:: tommylikehu

Work Items

Continue on https://review.openstack.org/#/c/413156/

Dependencies

None

Testing

Covered by unit tests in the cinder client.

Documentation Impact

Cinder shell client documentation will need updates.

References

cinderclient change: https://review.openstack.org/#/c/413156/
reset-state in openstackclient: https://review.openstack.org/#/c/268907/
Obsoletes some of pike/support-reset-generic-group-and-group-snapshot-status.rst
IRC meeting discussion: http://eavesdrop.openstack.org/meetings/cinder/2017/cinder.2017-01-04-16.02.log.html#l-153

Generic Volume Migration

Tue, 20 Dec 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/generic-volume-migration

The intent is to allow volume drivers that do not support iSCSI and use other means for data transport to also participate in volume migration operations. At the moment, the existing code uses create_export to create and attach a volume via iSCSI to perform I/O operations. By making this more generic, we can allow other drivers to take part in volume migration as well.

This change is necessary to support volume migration for the Ceph driver.

Problem description

When migrating a volume between two backends, the copy_volume_data routine in the source volume’s driver is executed to move the blocks from one volume to another. This routine assumes that both source and destination volumes can be attached locally via iSCSI and calls create_export to attach the volume to the local cinder-volume instance. This is technically not necessary for local volumes and also prevents drivers such as Ceph from participating in volume migration operations.

Use Cases

Proposed change

By using a file-like object abstraction similar to the backup volume driver approach, we can allow the volume driver to determine how best to attach its volume locally. For most drivers, this will continue to be iSCSI. And for Ceph, a RBD object that support file semantics is returned. In both cases, the copy_volume_data routine operates on these file objects as it does now, but without explicit knowledge of the underlying transport mechanism.

Specifically, I would like to add a routine to the volume driver (cinder/volume/driver.py):

::: open_volume() close_volume()

These routines are called when doing local volume attachment as in copy_volume_data(). I think these routines are necessary because local file I/O operations may require different transport than the standard create_export/initialize_connection approach. In the case of Ceph, create_export does nothing and initialize_connection returns the necessary RBD connection info for Nova to communicate with the Ceph volume directly. We want driver authors to retain this flexibility.

The open_volume routine will return a volume file object that supports file operations such as read, write, seek, etc. The close_volume routine will handle teardown and cleanup of the volume file object.

This will allow Ceph to return a file-like object that communicates directly with the volume. Other backends, such as LVM, will continue to create an iSCSI export, attach it, open the resulting block device, and return a handle to this file. Additionally, local LVM volumes could skip the iSCSI step and open the local block device directly. This change could be made later, this effort is to make the minimum amount of changes necessary to support the feature.

With these driver routines in place, copy_volume_data is modified to make use of these calls instead of the using _attach_volume directly. For non-Ceph drivers the result is exactly the same as it was. And the RBD driver can now implement these routines to add support for volume migration.

Alternatives

The most obvious alternative is to change the Ceph driver to implement the export routines and allow volume attachment via iSCSI. This would mean using the rbd kernel driver which is not ideal, as it introduces code into the running kernel on the cinder-volume node and lags a bit in features as compared to librbd. We would like to avoid this if at all possible.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

If everything works correctly, volume migration between Ceph and some other backend should function as expected. In both directions.

Performance Impact

The additional abstraction only enables alternatives to iSCSI attachment, no impact on existing functionality is introduced.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: jbernard

Work Items

Introduce the file-object abstraction and change existing code to use iSCSI for volume attachment - just as it does now. Ceph migration will continue to fail.
Implement the Ceph-specific attach logic, that should allow volume migration to succeed between other backends.
Add test(s) to tempest to verify code paths are executed correctly and yield a block-for-block identicaly migration result.

Dependencies

None

Testing

I don’t think volume migration is covered in the gate, but this could be tested in tempest. Ceph volume migration to a non-Ceph backend should be successful in both directions. If that test passes, this effort was a success.

Documentation Impact

I’m don’t think anything is necessary here.

References

None

Dynamic Log Level control via REST API

Tue, 29 Nov 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/dynamic-log-levels

Add REST API to control Cinder services’ log levels dynamically.

Problem description

To change log levels in a service the service’s configuration needs to be changed and the service restarted. The restart can be done by restarting the service itself or by requesting an internal restart via SIGHUP signal.

In some services a restart is not a big deal, API and scheduler, because they only operate in the control plane and they don’t perform long running operations, but in other services, Volume and Backup, this is a bigger deal, because they are in the data plane as well and restarting of a service may take a long time.

We should be able to change service log levels dynamically as needed, even if they will revert back to the defaults on restart.

A downside to being able to dynamically change log levels is that we’ll no longer be sure of what log level a service is running at a given time, so we’ll also need a mechanism to query current log levels of a service.

Use Cases

Cloud users are encountering problems when using the cloud and they contact support, so the system operator starts looking at the logs only to find out that correct log levels are insufficient to determine the root cause of the problem and the log levels need to be changed to DEBUG.

Another use case that would be satisfied by the implementation of this spec as a side product would be when a system administrator wants to confirm Message Broker connectivity in a service, as the log level query mechanism can be used as a ping to the service via the Message Broker.

Proposed change

The proposal is to introduce 2 new service REST APIs actions, one to modify debug levels at runtime and another to query them. The life of the log level changes will be the current service run, as they will revert to those defined in the configuration file upon restart.

Setting the log levels will be possible for all Volume, Scheduler, and Backup services, but limited in the API service to only the service process that receives the request since there is no mechanism in place right now to propagate the request to other API nodes and adding such mechanism for this feature would be an unnecessary complexity at this point.

This is a reasonable limitation, since API services can be easily restarted without impacting the cloud because they are only in the control plane and are usually deployed in an Active/Active configuration. And if they are not in an Active/Active configuration then there’s only 1 API service running and not being able to propagate the API log level change isn’t such a big deal.

While some operators may prefer to restart the API services to change the log levels, there may be others that prefer to directly make the dynamic log level changes to the all the API nodes skipping the load balancer to avoid restarts, and some others that will just change one API node dynamically skipping the load balancer and make the test request to that one API node.

The mechanism to set the log level should be versatile enough that no scripting is necessary when we want to do multiple changes. The way to achieve this will be to allow changing log levels to all addressable services or limit by binary and/or server.

It’ll also be possible to decide which log levels to change in the service, so we’ll be able to not only change the log levels of the cinder service itself, but also those of its libraries (ie. SQLAlchemy library).

Both mechanism will allow setting/querying multiple services but will only work on services that are up as per DB heartbeats.

Alternatives

An alternative would be to support Dynamic Reconfiguration after modifying cinder.conf, but that is a considerably bigger problem that will require more code changes, and while it’ll be more powerful it has also some drawbacks, since it requires access to the nodes to change the configuration of each of the services and also trigger the reload of each of them.

The benefit of having an API for the log levels is that you don’t have to have access to the infrastructure as you can request the change through the REST API and then check the logs in the log monitoring service.

Data model impact

None

REST API impact

Set log level: This will be implemented as a service action like enable and disable, but will use the set-log identifier. Effective URL /v3/{tenant_id}/os-services/set-log will take following parameters in the body:
- binary (optional): A string parameter indicating the binary of the service to change, it can take following values, cinder-volume, cinder-scheduler, cinder-backup, cinder-api, *, null, empty string or be missing. The last four possibilities being equivalent to all services.
- server (optional): A string parameter indicating the server to change, Can be a host or cluster reference - host@backend or cluster@backend -, or null, empty string, or be missing for all servers matching the binary.
- prefix (optional): A string indicating the prefix for the log path, for example cinder. or sqlalchemy.engine. When not present all logs will be changed.
- level (required): A string with the log level to set, case insensitive, accepted values are INFO, WARNING, ERROR, DEBUG.
Get log level: Service action with get-log identifier. Effective URL /v3/{tenant_id}/os-services/get-log will accept the following parameters in the body:
- binary (optional): A string parameter indicating the binary of the service to query, it can take following values, *, empty string, null, cinder-volume, cinder-scheduler, cinder-backup, and cinder-api. If missing or * or the an empty string is passed then all binaries will be used.
- server (optional): A string parameter indicating the server to query, Can be a host or a cluster reference - host@backend or cluster@backend.
- prefix (optional): A string indicating the prefix for the log path we are querying, for example cinder. or sqlalchemy.engine. When not present or the empty string is passed all log levels will be retrieved.

Example response to get-log:

{
   "log_levels":[
       {
          "binary": "cinder-api",
          "host": "hostname1",
          "levels":{
             "cinder.api": "DEBUG",
             "cinder.api.common": "DEBUG"
             "cinder.db.sqlalchemy.api": "DEBUG"
       },
       {
          "binary": "cinder-scheduler",
          "host": "hostname1",
          "levels":{
             "cinder": "DEBUG",
             "cinder.scheduler.manager": "DEBUG"
             "eventlet": "ERROR"
          }
       },
       {
          "binary": "cinder-volume",
          "host": "hostname2@backend#pool",
          "levels":{
             "cinder": "DEBUG",
             "cinder.volume.drivers.rbd": "DEBUG",
             "sqlalchemy": "WARNING"
          }
       }
   ]
}

Security impact

None, since it will be using the service update Access Control policy used for operations like enable, disable, and freeze…

Notifications impact

For audit purposes a new notification will be emitted with every dynamic log level change.

Other end user impact

None

Performance Impact

None besides the possible increase in log quantity when changed to a greater log level, for example debug.

Other deployer impact

None.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Gorka Eguileor (geguileo)

Work Items

Add the set API endpoint and mechanism on the services
Cinder client support for set action
Add the get API endpoint and mechanism on the services
Cinder client support for get action

Dependencies

None

Testing

Unittests for new API behavior.

Documentation Impact

Only the changes to the API need to be documented.

References

Introduce new volume attach APIs

Fri, 26 Aug 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-new-attach-apis

Problem description

The current volume attach process for Cinder Volumes is somewhat inconsistent and rather unclear in terms of exactly how it works. Some of this has been a result of growth and feature adds, things like optional parameters have made the process a bit more complex than it needs to be.

The interpretation of things like initialize_connection has sort of diverged a bit, and due to some requirements for different backends things have become a bit hacky. We have the potential for race conditions during the attach/detach process and most of all we seem to have a situation where adding things like multi-attach support are extremely complex and difficult.

Rather than continuing to add on to the monolith, it might be a good time to step back and simplify the flow altogether.

Use Cases

Attaching Cinder volumes to Nova Instances, Bare Metal or whatever.

Proposed change

Rather than using reserve, initialize and attach with numerous parameters to chain together an attach, and try to modify things to make multi-attach work this specs proposes that we instead introduce a new simplified and more robust set of attach API’s.

API Operations:

attachment_create

This call is used to create a volume attachment.

If the connector is not specified this call will only complete the volume reservation step and create an empty volume attachment. This can be used by Nova to call it from the API and start the volume attaching process by checking that the volume is available and creates the empty attachment. To finalize the operation Nova will call ‘attachment_update’.

In case the connector is specified it will reserve the volume and create the connection between the volume and the host and update the attachment record. This can be used for instance for bare metal use cases, when the connector is available immediately.

In both cases the call returns the attachment_id to the caller.

Args:
volume_id instance_uuid connector (optional)

Returns:
attachment_id
attachment_update

This call is used to update an existing volume attachment. For this call the connector is mandatory.

In case of an empty attachment it also finalizes attaching the volume and updates the attachment record accordingly.

As the attachment exists in every scenario this call is invoked the attachment_id is mandatory.

Args:
attachment_id connector

Returns:
connection_info
attachment_get

This call will fetch the specified attachment object (by ID). In addition this call will go out to the Cinder backend driver and query it for any shared connection info. The purpose being that we return the attachment object that was requested as well as a list of any shared attachment_id’s.

Args:
attachment_id

Returns:
Attachment info

{id: <uuid-of-the-attachment-object>,
project_id: <project-id-associated-with-attachment>, volume_id: <uuid-of-volume-associated-with-this-attachment-record>, instance_uuid: <uuid-of-instance-attachment-is-used-for>, attached_host: <hostname-of-compute-node-attached-to>, mountpoint: <mountpoint-inside-instance-that-is-expected-to-be-used>, attach_time: <time-we-created-the-attachment>, detach_time: <time-we-detached>, attach_status: <status-of-the-attachment>, attach_mode: <ro|rw>, shared_connections_by_attachment: [<attachment_id>, …]}
attachment_remove

Removes/Detaches an existing attachment, marks a volume as available if no more attachments to it left.

Args:
attachment_id

Returns:
List of attachment_id’s for any shared connections

Alternatives

Keep trying to hack our way around the existing attach/detach flows we have. Deal with the split brain issue we have between Nova and Cinder.

It is possible that we could rewrite the existing methods and enforce stricter interface usage and we’d probably be ok, but the code has become pretty convoluted and ugly to deal with and it might be easier to wrap the existing calls up into a managing wrapper and then over time iterate over the internal pieces.

Data model impact

We need to store the connector when making attachments. As the connector can vary based on the back end and can also easily exceed 255 chars in length adding it to the ‘volume_attachment’ table is not a viable option. As opposed to that we’ll add an ‘attachment_specs’ table that will slurp in the k/v entries in the connector.

REST API impact

This will introduce at least the 4 new calls mentioned in the proposal section, these will be part of a new API object named attachment. We’ll treat the attachment as a first class object in Cinder.

The new API will be released under a new microversion.

Security impact

Notifications impact

Other end user impact

The old API’s will still work until we decide to deprecate them.

Performance Impact

Other deployer impact

Developer impact

For now this is avoiding changes to the drivers in Cinder and focusing on just the API for the attach/detach call up until the Manager layer.

Implementation

Condense initialize_connection and attach down into a single attachment_create call.

The intent here is that Cinder shouldn’t know or care so much about what a consumer is doing with a volume or attachment. We should only care that they desire to attach or detach and that’s about it. In order to do this properly we will need an attachment_id that the consumer should store and reference going forward when they are done with the attachment. This means it’s up to the consumer to decide when they are truly done and they want to destroy the attachment as opposed to now where we try and guess for them based on clues.

Assignee(s)

Primary assignee:: john-griffith

Work Items

Implement API calls and wrappers in Cinder
Implement cinderclient calls to expose them
Implement changes in Nova

Dependencies

Testing

Once the API design is finalized the same Tempest test coverage will be added as we have for the current attach/detach API. In parallel to this activity the Nova implementation to pick up the new API shall be started as well which will add an additional layer of testing.

Documentation Impact

Will need to update any and all documents about how attach/detach of volumes in Cinder works.

References

Nova spec

Parameter combinations for delete (vol, snap, etc.)

Mon, 23 May 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/volume-delete-parameters

This spec outlines how to improve our volume delete functionality with regards to optional parameters which request non-default behaviors.

Problem description

It is not possible to combine both volume force delete and cascade delete.

It is also difficult to add more parameters to volume delete, since they have to interact in a reasonable way with things like os-force-delete, which is part of volume delete from a user’s point of view, but not the same API action.

Use Cases

This makes it easier to delete a volume which may be in an odd state.

It also simplifies our API by passing optional parameters to delete rather than separate action calls. This allows us to combine the parameters in meaningful ways (force + cascade), as well as extend the same combinations to snapshot-delete, cg-delete, etc., without having to make X delete API actions for each parameter. (eg. os-force-delete, os-force-delete-snapshot, os-force-delete-cg, etc.)

Also consider if we wish to add the option later to delete an attached volume with a “force-detach” parameter, etc.

This also should reduce the number of cases where an admin/user needs to use reset-state operations.

Proposed change

Deprecate use of os-force-delete and make “force” a parameter passed to volume delete like “cascade” is.

The ability to default “force” to be an admin-only operation via config will be maintained.

Alternatives

Change nothing.

Data model impact

None

REST API impact

New boolean “force” parameter for volume delete, which defaults to False.

This will behave the same as os-force-delete if not combined with other arguments.

If combined with cascade, a cascade delete which ignores the volume and snapshot states will be performed.

Security impact

None

Notifications impact

None

Other end user impact

$ cinder delete –force –cascade <volume>

will be accepted. This gives a “delete this volume regardless of the state of things” operation which does not exist today.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: eharney

Work Items

Add “force” as a parameter to volume delete API
Add logic to handle combination of force and cascade
(eventually) remove os-force-delete with a new API microversion
Look at what to do, if anything, in this same regard for “unmanage”, e.g., “unmanage –cascade”.

Dependencies

None

Testing

New tempest test for volume delete which uses the parameterized version rather than os-force-delete.

Tempest test for cascade + force volume delete.

Documentation Impact

New arguments for cinderclient volume delete.

References

Cascade delete https://review.openstack.org/#/c/201748/

Inspection Mechanism For Capacity Limited Host

Thu, 21 Apr 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/inspection-mechanism-for-capacity-limited-host

Cinder now has a scheduler filter called CapacityFilter which allow users to create volume under capacity limit. But some APIs which don’t go through cinder-scheduler will break the destination host’s capacity.

Problem description

Some backends now support the thin provisioning capability. But only Cinder gets and checks this capability in cinder-scheduler’s filter: CapacityFilter. It means that some APIs which go to the backend directly without cinder-scheduler doesn’t get and check the reserved_percentage and the max_over_subscription_ratio of the backend. Then it may break the backend’s capacity. Such as “extend volume”, “create volume from snapshot”, “copy volume” and so on.

Use Cases

Usually, the capacity is a hard limit for the specified backend when Cinder works with the CapacityFilter (It is one of the default filters in Cinder). It means that every action to this kind of backend should keep its capacity. We should not break the capability of the backend which support the thin provisioning.

Proposed change

Add some common functions to check the backend’s capacity at the cinder-volume layer. The patch[1] is a POC to solve volume creation problem.
The APIs which 1) change the backend’s capacity and, 2) don’t go through cinder-scheduler, should be directly checked. It includes:
- POST /v3/{project_id}/volumes/{volume_id}/action (os-extend) (SOLVED)[2]
- POST /v3/{project_id}/volumes (create from snapshot, volume or replica)
- POST /v3/{project_id}/groups/action (create_from_src)
- POST /v3/{project_id}/snapshots

NOTE:

Create volume requests for a snapshot, volume, or replica will change the backend’s capacity, and will therefore cause the volume’s status to switch to “error”.
The cinder admin isn’t required to use the CapacityFilter, since the volume capacity limit doesn’t need to be checked at the driver layer.

Alternatives

For the APIs which will change the backend’s capacity and not go through cinder-scheduler, we could route them through the scheduler instead of jumping right to a volume-service.

The problem for the extend API is solved by a bug fix[2] recently. In the patch, it passes a “volume” parameter which includes the host name to cinder-scheduler to indicate that the destination host is specified. Then cinder-scheduler will only check the specified host with backend_passes_filters function. If no error raises, it means that the POST action is allowed.

NOTE:

We need to bump the RPC call’s version between cinder-api and cinder-scheduler to keep backward compatibility for each related API.
The create actions will always call through the cinder-scheduler, even if the destination host is specified. If the cinder-scheduler can’t match the new RPC version, the old behavior is kept.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

Instead of sending notifications, we’d better send user messages if the actions fail.

Other end user impact

The create requests which will exceed the backend’s capacity will make the volume’s status to “error”.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: wangxiyuan <wangxiyuan@huawei.com> Huang Zhiteng <winston.d@gmail.com> Erlon R. Cruz <sombrafam@gmail.com> tommylikehu<tommylikehu@gmail.com>

Work Items

Add some common check functions and call them in:
- create volume from volume, snapshot or replica
- create cg from source
- create group from source
- create snapshot
Add and update the unit test.

Dependencies

None

Testing

Unit tests will be added and updated.

Documentation Impact

None

References

[1] Huang Zhiteng: https://review.openstack.org/#/c/437677/ [2] Erlon R.Cruz: https://review.openstack.org/#/c/405578/

Stochastic Weighing Scheduler

Mon, 18 Apr 2016 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/stochastic-weighing-scheduler

The filter scheduler is the de-facto standard scheduler for Cinder and has a lot of desirable properties. However there are certain scenarios where it’s hard or impossible to get it to do the right thing. I think some small tweaks could help make admin’s lives easier.

Problem description

I’m concerned about 2 specific problems:

When there are multiple backends that are not identical, it can be hard to ensure that load is spread across all the backends. Consider the case of a few “large” backends mixed with some “small” backends. Even if they’re from the same vendor by default new volumes will go exclusively to the large backends until free space decreases to the same level as the small backends. This can be worked around by using something other than free space to weigh hosts, but no matter what you choose, you’ll have a similar issue whenever the backends aren’t homogeneous.
Even if the admin is able to ensure that all the backends are identical in every way, at some point the cloud will probably need to grow, by adding new storage backends. When this happens there will be a mix of brand new empty backends and mostly full backends. No matter what kind of weighing function you use, initially 100% of new requests will be scheduled on the new backends. Depending on how good or bad the weighing function is, it could take a long time before the old backends start receiving new requests and during this period system performance is likely to drop dramatically. The problem is particularly bad if the upgrade is a small one: consider adding 1 new backend to a system with 10 existing backends. If 100% of new volumes go to the new backend, then for some period, there will be 10x load on the single backend.

There is one existing partial solution to the above problems – the goodness weigher – but that has some limitations worth mentioning. Consider an ideal goodness function – an oracle that always returns the right value such that the best backend for new volumes is sorted to the top. Because the inputs to the goodness function (other than available space) are only evaluated every 60 seconds, bursts of creation requests will nearly always go to the same backend within a 60 second window. While we could shrink the time window of this problem by sending more frequent updates, that has its own costs and also has diminishing returns. In the more realistic case of a goodness function that’s non-ideal, it may take longer than 60 seconds for the goodness function output to reflect changes based on recent creation requests.

Use Cases

The existing scheduler handles homogeneous backends best, and relies on a relatively low rate of creation requests compared to the capacity of the whole system, so that it can get keep up to date information with which to make optimal decisions. It also deals best with cases when you don’t add capacity over time.

I’m interested in making the scheduler perform well across a broad range of deployment scenarios:

Mixed vendor scenarios
A mix of generations of hardware from a single vendor
A mix of capacities of hardware (small vs. large configs)
Adding new capacity to a running cloud to deal with growth

These are all deployer/administrator concerns. Part of the proposed solution is to enable certain things which are impossible today, but mostly the goal is to make the average case “just work” so that administrators don’t have to babysit the system to get reasonable behavior.

Proposed change

Currently the filter scheduler does 3 things:

Takes a list of all pools and filters out the ones that are unsuitable for a given creation request.
Generates a weight for each pool based on one of the available weighers.
Sorts the pools and chooses the one with the highest weight.

I call the above system “winner-take-all” because whether the top 2 weights are 100 and 0 or 49 and 51, the winner gets the request 100% of the time.

I propose adding a new option to the filter scheduler called “weighted_stochastic”. It should default to False, which would give the current winner-take-all behavior. If set to True however, the scheduling algorithm would change as follows:

In step 3 above, rather than simply selecting the highest weight, the scheduler would sum up the weight of all choices, assign each pool a subset of that range with a size equal to that host’s weight, then generate a random number across the whole range and choose the pool mapped to that range.

An easier way to visualize the above algorithm is to imagine a raffle drawing. Each pool is given a number of raffle tickets equal to the pool’s weight (assume weights normalized from 0-100). The winning pool is chosen by a raffle drawing. Every creation request results in a new raffle being held.

Pools with higher weights get more raffle tickets and thus have a higher chance to win, but any pool with a weight higher than 0 has some chance to win.

The advantage to the above algorithm is that it distinguishes between weights that are close (49 and 51) vs weights that are far (0 and 100) so just because one pools is slightly better than another pool, it doesn’t always win. Also, it can give different results within a 60 second window of time when the inputs to the weigher aren’t updated, significantly decreasing the pain of slow volume stats updates.

It should be pointed out that this algorithm not only requires that weights are properly normalized (the current goodness weigher also requires this) but that the weight should be roughly linear across the range of possible values. Any deviation from linear “goodness” can result in bad decisions being made, due to the randomness inherent in this approach.

Alternatives

There aren’t many good options to deal with the problem of bursty requests relative to the update frequency of volume stats. You can update stats faster but there’s a limit. The limit is to have the scheduler synchronously request absolute latest volume stats from every backend for every request. Clearly that approach won’t scale.

To deal with the heterogeneous backends problem, we have the goodness function, but it’s challenging to pick a goodness function that yields acceptable results across all types of variation in backends. This proposal keeps the goodness function and builds upon it to both make it stronger, and also more tolerant to imperfection.

Data model impact

No database changes.

REST API impact

No REST API changes.

Security impact

No security impact.

Notifications impact

No notification impact.

Other end user impact

End users may indirectly experience better (or conceivably worse) scheduling choices made by the modified scheduler.

Performance Impact

No performance impact. In fact this approach is proposed expressly because alternative solutions would have a performance impact and I want to avoid that.

Other deployer impact

I propose a single new config option for the scheduler. The default value for this option is to act like the existing scheduler does. An administrator would need to intentionally enable to option to observe changed behavior.

Developer impact

Developers wouldn’t be directly impacted, but anyone working on goodness functions or other weighers would need to be aware of the linearity requirement for getting good behavior out of this new scheduler mode.

In order to avoid accidentally feeding nonlinear goodness values into the stochastic weighing scheduler, we may want to create alternatively-named version of the various weights or weighers, forcing driver authors to explicitly opt-in to the new scheme and thus indicate that the weights they’re returning are suitably linear.

Implementation

Assignee(s)

Primary assignee:: bswartz

Work Items

This should be doable in a single patch.

Dependencies

Filter scheduler (cinder)
Goodness weigher (cinder)

Testing

Testing this feature will require a multibackend configuration (otherwise scheduling is just a no-op).

Because randomness is inherently required for the correctness of the algorithm, it will be challenging to write automated functional test cases without subverting the random number generation. I propose that we rely on unit tests to ensure correctness because it’s easy to “fake” random numbers in unit tests.

Documentation Impact

Dev docs need updated to explain to driver authors what the expectations are for goodness functions.

Config ref needs to explain to deployers what the new config option does.

References

No references.

Add Generic Volume Group Into Quota Management

Mon, 18 Apr 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-volumegroup-into-quota-management

Generic Volume Group currently has its own quota mechanism. But we can’t get any information about the Generic Volume Group quota at the API layer.

Problem description

Cinder already achieved Generic Volume Group quota mechanism. It means that there is a Generic Volume Group quota class(the hard_limit is 10) at the DB layer. But the Generic Volume Group’s quota information wasn’t contained in any quota API’s response body, so that users can’t get or update it.

Use Cases

Cinder should allow users to get and update the Generic Volume Group’s quota. It’s terrible that users could only use the Generic Volume Group quota’s default value and can’t update it. With this change, users could 1) change the value of Generic Volume Group’s hard limit, 2) get the Generic Volume Group’s quota usage, through the Cinder’s quota API.

Proposed change

Let quota management undertake Generic Volume Group quota. And these five APIs will be changed: 1) quota-class-show

GET /os-quota-class-sets/default

Add a new line in the response body.

{
    "quota_class_set": {
        "groups": 10
    }
}

2) quota-show, quota-usage

GET /os-quota-sets/{project_id}?usage={False, True}

Add a new line in the response body.

{
    "quota_set": {
        "groups": {
            "reserved": 0,
            "allocated": 0,
            "limit": 10,
            "in_use": 0
        }
    }
}

3) quota-defaults

GET /os-quota-sets/{project_id}/defaults

Add a new line in the response body.

{
    "quota_set": {
        "groups": 10
    }
}

4) quota-class-update

PUT /os-quota-class-sets/default

Allow update “groups” and Add a new line in the response body.

{
    "quota_class_set": {
        "groups": 10
    }
}

5) quota-update

PUT /os-quota-sets/{project_id}

Allow update “groups” and Add a new line in the response body.

{
    "quota_set": {
        "groups": 10
    }
}

Alternatives

Leave as is.

Data model impact

None

REST API impact

The response body of “quota-defaults”, “quota-usage”, “quota-show” and “quota-class-show” APIs will contain Generic Volume Group.
The “quota-update” and “quota-class-update” APIs will accept the “groups” parameter and the response body will contain Generic Volume Group.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: wangxiyuan

Work Items

Add Generic Volume Group’s quota to quota APIs
Add and update the unit tests
Update the CinderClient’s quota commands

Dependencies

None

Testing

Standard unit tests and manual testing.

Documentation Impact

The response body of quota-defaults, quota-usage, quota-show and quota-class-show should be updated.
The request body of quota-update, quota-class-update should be updated.

References

None

Support volume migration in Ceph driver

Wed, 23 Mar 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/ceph-volume-migrate

Currently cinder supports volume migration with backend assistance. Some backends like IPSANs support migrating volume in backend level, but Ceph does not. Though Ceph volume migration has already been supported through cinder generic migration process in Liberty, the migration efficiency is not so good. It is necessary to support volume migration in the Ceph driver to improve the efficiency of migration.

Problem description

Suppose there are two cinder volume backends which are in the same Ceph storage cluster.

Now volume migration between two Ceph volume backends is implemented by generic migration logic. Migration operation can proceed with file operations on handles returned from the os-brick connectors, but the migration efficiency is limited on file I/O speed. If we offload migration operation from cinder-volume host to Ceph storage cluster, we would get the following benefits:

Improve the volume migration efficiency between two Ceph storage pools, especially when the volume’s capacity and data size is big.
Reduce the IO pressure on cinder-volume host when doing the volume migration.

Use Cases

There are three cases for volume migration. The scope of this spec is for the available volumes only and targets to resolve the issues within the following migration case 1:

Within the scope of this spec:

1.Available volume migration between two pools from the same Ceph cluster.

Out of the scope of the spec:

2.Available volume migration between Ceph and other vendor driver.

3.In-use(attached) volume migration using Cinder generic migration.

Proposed change

Solution A: use rbd.Image.copy(dest_ioctx, dest_name) function to migrate volume from one pool to another pool.

To offload migration operation from cinder-volume host to ceph cluster, we need to do the following changes in migrate_volume routine in RBD driver.

Check whether source volume backend and destination volume backend are in the same ceph storage cluster or not.
If not, return (False, None).
If yes, use rbd.Image.copy(dest_ioctx, dest_name) function to copy volume from one pool to another pool.
Delete the old volume.

Alternatives

Solution B: use ceph’s functions of clone image and flatten clone image to migrate volume from one pool to another pool.

Solution B contains the following steps: * Create source volume snapshot snap_a and protect the snapshot snap_a. * Clone a child image image_a of snap_a to the destination pool. * Flatten the child image image_a, thus snap_a has not been depended on. * Unprotect the snapshot snap_a and delete it.

Using a volume which’s capacity and data size is 12GB to show the time-consuming comparison between solution A and solution B.

[Solution-A]Copy volumes1/volume-777617f2-e286-44b8-baff-d1e8b792cc52 from “volumes1” pool to “volumes2” pool.

root@2C5_19_CG1# time rbd cp volumes1/volume-777617f2-e286-44b8-baff-d1e8b792cc52 volumes2/test1

Image copy: 100% complete…done.

real 2m3.513s user 0m9.983s sys 0m25.213s
[Solution-B-step-1]Create a snapshot of volume 777617f2-e286-44b8-baff-d1e8b792cc52 and protect it.

root@2C5_19_CG1# time rbd snap create –pool volumes1 –image volume-777617f2-e286-44b8-baff-d1e8b792cc52 –snap snap_test

real 0m0.465s user 0m0.050s sys 0m0.016s

root@2C5_19_CG1# time rbd snap protect volumes1/volume-777617f2-e286-44b8-baff-d1e8b792cc52@snap_test

real 0m0.128s user 0m0.057s sys 0m0.006s
[Solution-B-step-2]Do clone operation on volumes1/volume-777617f2-e286-44b8-baff-d1e8b792cc52@snap_test.

root@2C5_19_CG1# time rbd clone volumes1/volume-777617f2-e286-44b8-baff-d1e8b792cc52@snap_test volumes2/snap_test_clone

real 0m0.336s user 0m0.058s sys 0m0.012s
[Solution-B-step-3]Flatten the clone image volumes2/snap_test_clone.

root@2C5_19_CG1# time rbd flatten volumes2/snap_test_clone Image flatten: 100% complete…done.

real 1m58.469s user 0m4.513s sys 0m17.181s
[Solution-B-step-4]Unprotect the snap volumes1/volume-777617f2-e286-44b8-baff-d1e8b792cc52@snap_test and delete it.

root@2C5_19_CG1# time rbd snap unprotect volumes1/volume-777617f2-e286-44b8-baff-d1e8b792cc52@snap_test

real 0m0.150s user 0m0.058s sys 0m0.013s

root@2C5_19_CG1# time rbd snap rm volumes1/volume-777617f2-e286-44b8-baff-d1e8b792cc52@snap_test

real 0m0.418s user 0m0.054s sys 0m0.011s

By the above test results of solution A and solution B, solution A needs (real:2m3.513s, user:0m9.983s, sys:0m25.213s) to finish the volume copy operation and solution B needs (real:1m59.966s, user:0m4.790s, sys:0m17.239s) to do that. The time-consuming for the both two solutions are not much difference, but solution A is more simpler than solution B. So we intend to use solution A to offload volume migration from cinder-volume host to ceph cluster.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

The performance of volume migration between two Ceph storage pools in the same Ceph cluster will be improved greatly.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: chen-xueying1<chen.xueying1@zte.com.cn>
Other contributors:: ji-xuepeng<ji.xuepeng@zte.com.cn>

Work Items

Add location info of back-end:

Add location_info in state of Ceph volume service, it should include the Ceph cluster name(or id) and storage pool name.
Implement volume migration:

1.Check whether the requirements of volume migration are met. If source back-end and destination back-end are in the same Ceph cluster and volume status is not ‘in-use’ state, the volume can be migrated.

2.Copy volume from one pool to another pool and keep it’s original image name.

3.Delete the old volume.

Dependencies

None

Testing

Unit tests will be added. Volume migration test case will be added.

Both unit and Tempest tests need to be created to cover the code change that mentioned in “Proposed change” and ensure that volume migration feature works well while introducing the feature of RBD volume migration.

Documentation Impact

None

References

None

Add Pagination To Other Resources

Tue, 15 Mar 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/add-pagination-to-other-resource

This spec aims to continue the work that we did in Liberty, adding pagination support to other cinder resources.

Problem description

In Liberty release, we have added pagination to backups and snapshots according bp[1]. There are still some work that hasn’t been done yet. In Mitaka, we intend to add pagination support to CG, Volume Type and Qos specs.

Use Cases

In large scale cloud systems, end users and manage system that’s on top of Cinder could make quick querying by using pagination, filter and sort functions to improve performance of querying.

Proposed change

Consistency Group: Refactor current implementation, using DB pagination querying, and add support to filter and sort in querying request.
Volume Type: Add pagination and sort support in querying request.
Qos Specs: Add pagination, filter and sort in querying request.
Add sql pagination querying support as we did in backup and snapshot.

Alternatives

None

Data model impact

None

REST API impact

According the API-wg guideline about pagination, filter and sort[2]:

GET /v2/{project_id}/{resource}?limit=xxx&marker=xxx&sort=xxx&{filter}=xxx
RESP BODY: {"resource_links": [{xxx}],
            "resource": [{xxx}, {xxx}, ..., {xxx}]
           }

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: wanghao<wanghao749@huawei.com>
Other contributors:: None

Work Items

Add pagination support to three resources. * Implement code in db pagination query. * Implement code in list querying api. * Test code.
Update cinderclient to support this functionality for those resources.
Add change to API doc.

Dependencies

None

Testing

Both unit and Tempest tests are needed to be created to cover the code change.

Documentation Impact

The cinder API documention will need to be updated to reflect the REST API changes.

References

[1]https://blueprints.launchpad.net/cinder/+spec/extend-limit-implementations [2]https://github.com/openstack/api-wg/blob/master/guidelines/pagination_filter_sort.rst

Cinder FC Zone Manager - User Friendly Zone Names

Tue, 15 Mar 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/brocade-zone-driver-friendly-zone-names

Introduce user friendly names for zones to include host name and storage system along with ids to easily identify the host and storage port details.

Problem description

FC Zone Manager will be enhanced to configure friendly names for zones. At present, zone manager uses the WWNs of Adapter port and Target port to configure the zone names.

Use Cases

This addresses the need for zone names to be user friendly so that humans can easily read the name and understand what host and/or storage system is represented by that zone. This feature only has impact on volume driver developers that need to add the host name and storage system name into the connection info being passed to the FC zone manager if they wish to use this feature.

Proposed change

As per current design, the zone names include a default prefix ‘openstack’ which can be customized and host port wwn and target wwn (optional) based on the zoning_policy config option: initiator-target or initiator as follows:

Initiator-Target Zones:: <openstack><host WWPN><target WWPN>
Initiator Zones:: <openstack><host WWPN>

These names are not user friendly.

This proposal is to create friendly zone names with host and storage names as follows:

Initiator-target zones: <Host Name>_<host WWPN>_<storage System>_<target WWPN>
Initiator zones: <Host Name>_<host WWPN>

The required host name and storage system are populated in connection_info dictionary returned by fibre channel volume driver initialize_connection method call. The volume driver will extract the host name from the connector object passed as parameter to the initialize_connection method. This blueprint is not documenting the ways the volume driver fetch the storage system value. It is up to the volume driver to implement a mechanism to retrieve the storage system value.

Connection Info samples:

{
   'driver_volume_type': 'fibre_channel'
     'data': {
        **'storage_system': 'AMCE_Array',**
        **'host_name': 'OS_Host100',**
        'target_discovered': True,
        'target_lun': 1,
        'target_wwn': '1234567890123',
        'access_mode': 'rw'
     }
}

or:

{
   'driver_volume_type': 'fibre_channel'
     'data': {
        **'storage_system': 'AMCE_Array',**
        **'host_name': 'OS_Host100',**
        'target_discovered': True,
        'target_lun': 1,
        'target_wwn': ['1234567890123', '0987654321321'],
        'access_mode': 'rw'
     }
}

On attach/detach, Zone Manager will be initialized with connection_info which is provided by volume driver. Zone Manager in turn will extract the host name and storage system and will pass this information to the zone driver to create/delete friendly zone names.

There is a 64 characters limit for zone names. Hence zone driver implements a mechanism to normalize the host name and storage system to best fit the zone name. A WWN is 16 characters long and 32 characters are reserved for host and storage WWNs. For Initiator and Target policy, the max number of characters available for host name and storage system is 14 characters each. Zone driver will trim the names to a max of 14 characters. For Initiator only policy, the max size allowed for host name is 47 characters. Remaining characters are reserved for ‘_’.

Zone Driver updates:

Zone driver’s add_connection and delete_connection methods will be enhanced to take host and storage params. These names are defaulted to None. If names are None, then zone driver will fall back to existing naming mechanism.

The Zone manager will extract the host name and storage system from connection_info dictionary using host_name and storage_system keys. If keys are not found or None, then Zone manager will initialize host name and storage system local variables to None.

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

There is no noticeable performance impact provided FC volume driver will be able to fetch storage system value swiftly.

Other deployer impact

None.

Developer impact

FC Volume drivers need to enhance their existing drivers to support friendly zone names. This is optional.

The storage and host name in the connection_info object returned by the initialize_connection method of FC Volume driver. See the proposed change section above for details.

Implementation

Assignee(s)

Primary assignee:: Prakash Kaligotla
Other contributors:: Nagendra Rao Jaladanki Angela Smith

Work Items

Enhance the zone manager to pass connection_info object to zone driver.
Implement Brocade Zone Driver to create zones as per new format using the host name and storage system in the connector object.
Implement Cisco Zone Driver to create zones as per new format using the host name and storage system in the connector object.
Volume drivers are expected to add storage system information to connector object.
Unit test the zone driver and client code.

Dependencies

Has dependency on FC Volume Drivers to provide host name and storage system as part of connection_info return dictionary on attach/detach calls.

If driver does not provide host name and storage system, the existing zone naming mechanism will be used.

Testing

Attach/Detach unit tests will be performed to verify zone manager.

Documentation Impact

None.

References

http://www.brocade.com/downloads/documents/html_product_manuals/FOS_740_CLI/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Title.Fabric_OS.html http://www.cisco.com/en/US/docs/storage/san_switches/mds9000/sw/rel_2_x/san-os/command/reference/CR02_z.html

Brocade Cinder Zone Driver - Virtual Fabric Support

Tue, 15 Mar 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/brocade-zone-driver-virtualfabrics-support

Introduce Virtual Fabric (VF) support for Brocade FC SAN Switches.

Problem description

As of Juno, there is no support for zone management in Brocade Virtual Fabrics thus preventing the administrator to add this automated zone access control in Brocade Virtual Fabrics (VF) environment.

Use Cases

This addresses the case where an end user had the Brocade Virtual Fabrics feature enabled on the FC fabric and zoning needs to be performed in a VF other than the default. The impact of this feature is on end users who now may configure the Virtual Fabric ID in the cinder.conf file for each Brocade FC fabric block.

Proposed change

A Brocade FC switch(fixed/modular) can be partitioned into multiple virtual switches. A unique ID called VFID(Virtual Fabric ID) is assigned to each virtual switch and all the configurations, including zones, happen within the context of this VFID. User can configure a particular VFID to be ‘default’ which will be the default context for when users login to the switch.

As of Juno, the Brocade Zone Driver and look-up service does not set the VF context while establishing a session to the switch because of which it has access only to the zones in the ‘default’ VFID.

The proposal is to enhance the driver and look-up service to set the VFID context to support any virtual fabric configured in the chassis.

A new ‘fc_virtual_fabric_id’ config option is added to zoning fabrics configuration options as follows:

fc_virtual_fabric_id (StrOpt) VFID of virtual fabric. Default value is ‘None’.

The zone driver and look-up service will read the fabric configuration and extract the VFID to read/apply the configurations on the specific virtual fabric.

Multiple Virtual Fabrics Support:

Cinder already has support for configuring multiple fabrics. In this release, we are adding the support to configure multiple virtual fabrics.

Configuring multiple virtual fabrics is similar to configuring regular fabrics. In case of a virtual fabric, the cinder admin will configure fc_virtual_fabric_id config option in addition to all other fabric configuration options.

On volume attach/detach, the fibre channel volume driver will use SAN lookup service to traverse through the fabrics configured in cinder.conf to identify the targets connected to a host and returns a map of initiator and target port WWNs for each fabric.

Sample map object returned by the look up service:

{
    <Fabric1>: {
        'initiator_port_wwn_list':
        ('200000051e55a100', '200000051e55a121'..)
        'target_port_wwn_list':
        ('100000051e55a100', '100000051e55a121'..)
    }
    <Virtual_Fabric_2>: {
        'initiator_port_wwn_list':
        ('300000051e55a100', '300000051e55a121'..)
        'target_port_wwn_list':
        ('400000051e55a100', '400000051e55a121'..)
    }
}

The volume driver will process this information to extract initiator and target map for each fabric and builds a new initiator_target_map object.

Sample initiator_target_map object:

{
    'host WWPN 1': ['target WWPN 1', 'target WWPN 2']
    'host WWPN 2': ['target WWPN 3', 'target WWPN 4']
}

The zone manager will be provided with the above map of initiator and it’s corresponding target port WWNs by the volume driver to create/delete zones.

Alternatives

None.

Data model impact

None.

REST API impact

None.

Security impact

Zoning enforces ACL in SAN. Zoning is same in either regular or virtual fabrics. There are no specific security impacts with respect to virtual fabrics.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Other deployer impact

Configuration of virtual fabric requires providing the fc_virtual_fabric_id config option with valid virtual fabric VF ID in zoning fabric configuration.

Developer impact

Volume drivers will NOT have to be modified to support virtual fabric.

Implementation

Assignee(s)

Primary assignee:: Prakash Kaligotla
Other contributors:: Nagendra Rao Jaladanki Angela Smith

Work Items

Enhance the Brocade look-up service to find host and targets connected to a VF.
Enhance the Brocade Zone Driver to execute commands in the context of a VFID.
Unit test the zone driver and look-up service.

Dependencies

None.

Testing

Unit tests will be performed to make sure all CRUD operations are successful on virtual and physical SAN fabrics.

Documentation Impact

Configuration details of fc_virtual_fabric_id config option will be added to the fabric zoning configuration of Brocade Fibre Channel Zone Driver.

References

http://www.brocade.com/downloads/documents/html_product_manuals/FOS_730_CLI/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Title.Fabric_OS.html

capacity headroom

Tue, 15 Mar 2016 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/capacity-headroom

The proposal is to provide a mechanism surfacing the visibility of the total remaining block storage capacity which is available to allocate or resize volumes. It will be an indication for deployer to have the future plan.

Problem description

Currently, there is not a clear way for admin users to know how much block storage capacity left totally in cinder can be deployed.

condition with several backends
condition with a backend supporting “over subscription” which temporary capacity not used can be utilized to deploy new volumes
condition with a backend which reports capacity as “infinite” or “unknown”

Use Cases

The proposal calculates the virtual free capacity for each pool and also sum for each backend. It will notify the info together with other useful capacity info which already exist to ceilometer service. The ceilometer service will utilize the notification to generate capacity samples. The admin users may have a hint about the trend of cinder capacity usage, and it will be useful for future capacity planning.

Proposed change

Calculates the virtual free capacity for each pool which reports capcity normally and sums the total for the backend.

How to calculate the virtual free capacity(virtual_free):

For thin provisioning, according to the over subscription mechanism implemented for LVM, the remaining virtual capacity for a pool can be used as terminology: virtual_free = apparent_available_virtual_capacity. It can be calculated by following formula:

virtual_free = apparent_available_virtual_capacity = total_capacity * max_over_subscription_ratio - provisioned_capacity

For thick provisioning, just use physical capacity.
Notify the pool capacity and also the total capacity for a backend to ceilometer service. The capacity notification includes total/free/allocated/provisioned/virtual_free.

For backend which reports “unknown/infinite”, just report it as “unknown/infinite”.

New methods in HostState

get_capacity() - call get_pools() - calculate the capacity info for pool and then sum for each backend

Alternatives

Another alternative could be:

create a new data table which describe capacity info into database in scheduler.
provide a cinder api to retrieve the capacity info from database.

Compare with the proposal, database write operations in scheduler may cost more.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

The method update_service_capabilities() in host_manager will call get_capacity() and send notifications to ceilometer service. Then ceilometer service can produce capacity samples.

Other end user impact

N/A.

Performance Impact

N/A.

Other deployer impact

N/A.

Developer impact

It will be better if Cinder volume drivers can report provisioned_capacity if it has. This will improve the efficiency of the storage utilization. Note: This proposal will calculate virtual capacity thru provisioned_capacity. Otherwise we will calculate physical capacity instead.

Implementation

Assignee(s)

Primary assignee:: XinXiaohui

Other contributors:

Work Items

Calculate the capacity:
- Calculates the total virtual_free capacity for a pool if thin provisioning is supported.
  
  virtual_free = apparent_available_virtual_capacity = total_capacity * max_over_subscription_ratio - provisioned_capacity
  
  Otherwise, just use the physical capacity.
notify capacity (total/free/allocated/provisioned/virtual_free) for each pool and also notify total capacity for the backend to ceilometer service, if the backend reports “unknown/infinite”, just report it as it is.

Dependencies

The proposal depends on the over subscription mechanism of backend drivers.

Testing

Unit tests will be added.

Documentation Impact

None.

References

https://etherpad.openstack.org/p/kilo-cinder-over-subscription https://etherpad.openstack.org/p/kilo-cinder-capacity-headroom

Cinder Volume Active/Active support - API Races

Tue, 15 Mar 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support

Right now cinder-volume service can run only in Active/Passive HA fashion.

One of the reasons for this is non-atomic state transitions in c-api that may cause race conditions.

This spec proposes a change to do atomic DB changes in Cinder API using compare-and-swap to prevent them.

Problem description

Current Cinder API Service code has sections where contents from the DB are retrieved at one point and then checked for validity and in case all checks pass and the operation can proceed DB is changed accordingly.

Elapsed time between the retrieval and the DB changes depends on the nature of the validation, in some cases it’s just a simple check of the ‘status’ field, while in others it requires checking other tables for example to confirm that a volume is not attached or has no snapshots.

This way of handling checks and changes in the API creates a window of opportunity for code races to occur. These races would happen when changes are made to the DB after we have retrieved its contents, leading us to make a decision based on outdated data and not only change the DB using obsolete data as reference, but in most cases also incorrectly calling Volume Manager’s Service to perform an operation.

One example of these API races is the extend method in cinder/volume/api.py where we check at one point the status:

if volume[‘status’] != ‘available’:

Then later on we change the status:

self.update(context, volume, {‘status’: ‘extending’})

And we finally make an RPC request to perform the operation:

self.volume_rpcapi.extend_volume(context, volume, new_size, reservations)

Use Cases

Operator would want to avoid unexpected behavior in their clouds, moreover since these race conditions could lead to data corruption in the storage back-end. And these are probably situations more likely to happen in an Active/Active configuration that can handle a higher volume workload.

Proposed change

Proposed solution is to use compare-and-swap updates, with retries on Deadlocks, to ensure that we only update the DB when all required conditions are met.

Basically a compare-and-swap is just a complex DB query that has a condition added to the update query to ensure that modifications to the DB only occurs if those conditions are met. They can be simple conditions like status == ‘available’ or more complex conditions referring to other tables.

This solution has been chosen over other alternatives because performance numbers in tests show that even under the most extreme conditions in a multi-master cluster DB configuration they will have great performance results.

The idea is to keep Cinder API service behaving as close as possible to the original code, so we will fail and succeed under the same conditions. Even though reported errors will be slightly different, the only relevant difference will be when we previously would have had a race condition, and possibly data corruption, that both API caller would have received an affirmative response to their request and now one of them will receive a failure as if both operations had been performed sequentially.

The small difference in how we report errors comes from the fact that on failure we no longer know which of the conditions triggered the failure. So in name of efficiency and code clarity we will be returning a generic error stating that some of the required conditions were not met and therefore requested operation could not be completed.

Alternatives

An alternative to returning a generic error on failure, that was rejected for its complexity, would be to return the exact same errors we are returning now. For that, we would get updated data from the DB and give it to the validation method that will check which of the conditions is not met and raise the right exception with the right error message. For example this method could be checking the status of a volume and raise a message if it is not available, and then check that the volume has not snapshots and if it does then raise a different exception or the same exception with a different message.

Skeleton of the procedure:

result = resource.conditional_update(values, conditions)
if not result:
    resource = db.get_resource(resource.id)
    raise_right_validation_error(resource)
rpc_call(resource)

But there is a potential race here, we could be requesting a conditional update that fails (because the conditions in the DB are not met), and when we retrieve the data from the DB it has just changed (now it would meet the conditions for the update), so when we call the method in charge of raising the right validation error with this changed data it will not find any reason why we cannot perform the operation, so it will not raise any exception and will just return control to the caller.

In that case, since we didn’t raise an error, we would continue with the rpc call without having changed the DB, which is a problem. That is why we need a loop in the conditional update, to make sure we either succeed on the update or we raise the error regardless of racing conditions on the data.

Updated skeleton of the procedure:

while not resource.conditional_update(values, conditions):
    resource = db.get_resource(resource.id)
    raise_right_validation_error(resource)
rpc_call(resource)

In this implementation it would be a good idea to have a maximum number of retries instead of just looping until we either update the DB or can properly raise an error. That way we would not have an infinite loop if we have an error in our conditional update or our validation code.

There are also multiple alternatives to compare-and-swap that have been explored and discarded for being less efficient - as in being slower or requiring more queries to the DB.

Discarded alternatives are:

SELECT … FOR UPDATE, with retries on Deadlocks, which even though it works with multi-master configurations it has more Deadlock retries than proposed solution.
Using a Distributed Locking Manager with different backends to enforce exclusive access to resources.

More information on the tests can be found in performance numbers.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Performance impact will be negligible, and may even have better performance since there will be only one query to the DB instead of multiple queries. For example on volume deletion, where we now have one query to see the status and another to get the count of snapshots for that volume. With the new compare-and-swap we will only have 1 query that will update the DB if the status is correct and no snapshot exist.

Other deployer impact

None

Developer impact

Once changes are made to Cinder API methods, all new API methods that are added will need to conform to this new compare-and-swap way of updating the DB to prevent new race conditions from entering the code base.

Implementation

Assignee(s)

Primary assignee:: Gorka Eguileor (geguileo)
Other contributors:: Anyone is welcome to help

Work Items

Atomic update method for DB.

This method will require certain capabilities:

Basic comparing: status = ‘available’ status != ‘available’
Comparing of multiple values: status in [‘available’, ‘error’] status not in [‘attaching’, ‘detaching’]
Handling None values like Python does: When doing a negative comparison against a non None value it should return None values as well. So checking for ‘migration_status’ != ‘migrating’ would also return values that have ‘migration_status’ set to None.
Using additional complex filters: In some cases, like checking for attachments, more complex queries may be needed.
Set update values depending on other fields. For example when detaching a volume you will have to set the status to ‘available’ if there are no more volume attachments for that volume, and set it to ‘in-use’ if there are more attachments (it was multi attached).
Set update values based on fields on the DB: previous_status = status, status = ‘retyping’ size = size + 10

Atomic update method in Versioned Objects.

Besides exposing the functionality provided by above DB conditional update item it also needs to:

Automatically add the ID of the resource to the conditional update.
If no conditions are given it must assume we want the DB record to be unaltered.
It must allow saving dirty attributes together with the update.
It must be able to update the versioned object with data that has been written to the DB, even on conditional values.

Update Cinder API Service methods to use compare-and-swap.

Dependencies

No openstack dependencies, but it has a SQLAlchemy depencency on version 1.0.10 that includes Parameter-Ordered Updates (issue #3541).

On some DBs’ update method is order dependent, so they behave differently depending on the order of the values, example on a volume with ‘available’ status:

UPDATE volumes SET previous_status=status, status=’retyping’ WHERE id=’44f284f9-877d-4fce-9eb4-67a052410054’;

Will result in a volume with ‘retyping’ status and ‘available’ previous_status both on SQLite and MariaDB, but

UPDATE volumes SET status=’retyping’, previous_status=status WHERE id=’44f284f9-877d-4fce-9eb4-67a052410054’;

Will yield the same result in SQLite but will result in a volume with status and previous_status set to ‘retyping’ in MariaDB, which is not what we want, so order must be taken into consideration.

Testing

Unit tests will be added for Atomic Update methods.

Documentation Impact

None

References

Support query volume detail with glance metadata

Tue, 15 Mar 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/support-volume-glance-metadata-query

Provide a function to support query volume detail filter by glance metadata.

Problem description

The purpose of this feature is to make user to query volume detail more conveniently. User can query a specific bootable volume quickly filtering by image_name or other glance metadata.

Use Cases

At large scale deployment, there could be many bootable volumes in a tenant. So if user want query some bootable volume detail filtering by the image name or other info came from glance metadata, they could use this feature to query it more conveniently. No need to list all volumes and find what you want tiring.

Proposed change

Add DB query filter using volume_glance_metadata in api of sqlaclchemy.
User can use glance metadata to filter volume detail in cinder api. The query url is like this:
```
"volumes/detail?glance_metadata={"image_name":"xxx"}"
```

Alternatives

None

Data model impact

None

REST API impact

Add query filter support using glance metadata:

GET /v2/{project_id}/volumes/detail?glance_metadata={“image_name”:”xxx”}

Security impact

None

Notifications impact

None.

Other end user impact

None

Performance Impact

Search a lot of glance metadata may take longer than other querying. It may need add new index to column key and value in volume_glance_metadata table to improve searching performance.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: wanghao<wanghao749@huawei.com>

Work Items

Implement code in db query filter.
Update cinderclient to support this function.
Add change API doc.

Dependencies

None

Testing

Both unit and Tempest tests need to be created to cover the code change that mentioned in “Proposed change”.

Documentation Impact

The cinder API documentation will need to be updated to reflect the REST API changes.

References

None

List volumes and snapshots available for manage-existing

Sun, 21 Feb 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/list-manage-existing

Cinder currently has the ability to take over the management of existing volumes and snapshots (“manage existing”) and to relinquish management of volumes and snapshots (“unmanage”). The API to manage an existing volume takes a reference, which is a driver-specific string that is used to identify the volume on the storage backend. This process is currently confusing and error prone. This spec’s purpose is to detail APIs for listing volumes and snapshots available for management to make this flow more user-friendly. Allowing the listing of available volumes/snapshots will allow for the creation of an easy-to-use GUI to migrate existing volumes to a new OpenStack environment, or to recover in the case the Cinder database is lost.

Problem description

There is no Cinder API to list the volumes and snapshots available for managing, nor to know what reference string to use without referring to driver-specific documentation. This means that an admin must query the storage backend, get the appropriate references, and put them into the appropriate Cinder API. Without the proposed APIs this flow is confusing and error-prone.

Use Cases

Having Cinder manage an existing block storage volume or snapshot, using only Cinder APIs and without referring to external driver documentation.

Proposed change

Add new APIs to: 1) list volumes available for management in a given pool. 2) list snapshots available for management in a given pool.

The APIs would return information regarding the volumes/snapshots, provided by the driver. In addition, for each resource, an indication will be provided if it is safe to manage it. The cinder-volume manager will call the appropriate driver to get a list of resources. For each, the driver will return a safety indication, which includes if a volume is attached, or a Cinder ID if it suspects the resource is already managed by Cinder. The manager will look up any resources suspected already in use in the DB and mark them as not safe (this avoids the driver accessing the DB).

A list of items will be returned to the user, consisting of:

ref (string): a backend-specific reference to the volume that will be specified when the user wants to manage that volume.
size (integer): the size of the volume, rounded up to Gigabytes, as an OpenStack user would see it if the volume is managed.
actual_size (integer): the size of the volume in bytes, which may not be a precise Gigabyte multiple, as the volume was not created by OpenStack.
safe_to_manage (boolean): True if a call to manage the volume specified by host_ref is expected to be safe, False if it is known that it will not be (for example, if the driver detects that the volume is attached, or cinder-volume detects that it already manages the resource).
reason_not_safe (string): If “safe_to_manage” is False, specifies why.
driver-specific key-value pairs (list of string:string): Any additional information that a driver may wish to return to help the admin identify the volumes/snapshots, for example a description, timestamps, etc.

For listing snapshots, return these additional fields:

source_volume_ref (string): The backend-specific reference of the volume that is the source of the snapshot.
source_volume_id (string): The Cinder volume ID of the volume that owns the snapshot if the volume is already managed by Cinder.

Alternatives

Leave as is.

Data model impact

None

REST API impact

Add two additional API extensions.

Security impact

Cinder currently has the ability to create objects in the storage pool, and view or modify those objects. This allows the admin to view volumes not created by Cinder. On the other hand, the admin can take the storage credentials that Cinder has and perform the action on the storage backend itself, so there is no real impact.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Driver maintainers can add the two new calls to bring this functionality to their drivers.

Implementation

I propose that list_manageable be added as a new GET action on the os-volume-manage Cinder extension. It would be separately specifiable in Cinder’s policy.json file to restrict access appropriately (default being admin_api).

There would be new RPC calls to cinder-volume which would call two new driver APIs (list_manageable_volumes and list_manageable_snapshots). I will implement the LVM driver code as well, as a reference implementation.

Assignee(s)

Primary assignee:: avishay

Work Items

Create APIs
Create RPC calls
Create driver interface
Implement LVM reference
Implement tempest test

Dependencies

None

Testing

Standard unit tests and manual testing, as well as tempest test for these proposed APIs, as well as manage and unmanage.

Documentation Impact

The new APIs will be documented.

References

None

Delete multiple volume metadata keys

Thu, 11 Feb 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/delete-multiple-metadata-keys

Problem description

The current implementation of Cinder API functionality supports deleting of multiple volume metadata as multiple requests for deleting only one metadata key with a request. The more keys are necessary to remove, the more API requests and DB queries are needed. It would be more efficient to delete multiple metadata keys with a single request at a time.

Use Cases

Deleting multiple volume metadata keys with a single request.

Proposed change

To delete multiple metadata items without affecting the remaining ones, just update the metadata items with the updated complete list of ones (without items to delete) in the body of the request. On success, the server responds with a 200 status code.

Note

A PUT request should use etags to avoid the lost update problem.

Alternatives

Have a request that deletes all keys rather than a specified list as an option.

Data model impact

None

REST API impact

The existing API for update metadata will be used in this case. The API call will fail in the case some of the items that are specified to delete does not exist. The API call will send the set of keys that are supposed to be updated and ignore missing keys.

Security impact

None

Notifications impact

One delete notification will be emitted per metadata item deleted as today.

Other end user impact

A user will have new API.

Performance Impact

The deleting of a volume metadata keys with a single request allows to improve performance by reducing DB calls. It’s very important in case with many keys. Better performance due to fewer request round trips.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Yuriy Nesenenko(ynesenenko@mirantis.com)

Work Items

Extend python-cinderclient to support new API.
Add unit and tempest tests.

Dependencies

Depends on Cinder API microversion. Depends on API WG patch https://review.openstack.org/281511/ Depends on API WG patch https://review.openstack.org/301846/

Testing

Unit and functional tests are needed to ensure response is working correctly.

Documentation Impact

Documents concerning the API will need to reflect these changes.

References

None

Cheesecake

Fri, 29 Jan 2016 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/replication

This spec proposes further refinement to the Cinder replication. After more vendors have tried to implement replication and we’ve learned more lessons about the differences in backends and their semantics we’ve decided we should step back and look at simplifying this even further.

The goal of the new design is to address a large amount of confusion and differences in interpretation. Rather than try and cover multiple use cases in the first iteration, this spec aims to address a single fairly well defined use case. Then we can iterate and move on from there.

Problem description

The existing design is great for some backends, but is challenging for many devices to fit in to. It’s also filled with pitfalls with the question of managed/unmanaged, not to mention trying to deal with failing over some volumes and leaving others. The concept of failing over on a volume basis instead of on a device basis while nice for testing doesn’t fit well into the intended use case and results in quite a bit of complexity and also is not something that a number of backends can even support.

Use Cases

This is intended to be a DR mechanism. The model Use Case is a catastrophic event occurring on the backend storage device, but some or all volumes that were on the primary backend may have been replicated to another backend device in which case those volumes may still be accessible.

The flow of events are as follows: 1. Admin configures a backend device to enable replication. We have a configured cinder backend just as always (Backend-A) but we add config options for a replication target (Backend-B).

We no longer deal with differentiation between managed and unmanaged Now, to enable a replication target(s), the replication_target entry is the ONLY method allowed and is specified as a section in the driver.

Depending on the back-end device enabling this may mean that EVERY volume created on the device is replicated, or for those that have the capability and if admins choose to do so a Volume-Type of “replicated=True” can be created and used by tenants.

Note that if the backend only supports replicating “all” volumes, or if the Admin wants to set things up so that “all” volumes are replicated that the Type creation may or may not be necessary.

Tenant creates a Volume that is replicated (either by specifying appropriate Type, or by the nature of the backend device) Result in this example is a Volume we’ll call “Foo”
Backend-A is caught in the crossfire of a water balloon fight that shouldn’t have been taking place in the data center, and looses it’s magic smoke, “It’s dead Jim!”
Admin issues “cinder replication-failover” command with possible arguments a. Call propagates to Cinder Driver, which performs appropriate steps for that driver to now point to the secondary (target) device (Backend-B).
1. The Service Table in Cinder’s database is updated to indicate that a replication failover event has occurred, and the driver is currently pointing to an alternate target device.
In this scenario volumes that were replicated should still be accessible by tenants. The usage may or may not be restricted depending on options provided in the failover command. If no restrictions are set we expect to be able to continue using them as we would prior to the failure event.

Volumes that were attached/in-use are a special case in this scenario and will require additional steps. The Tenant will be required in this case to detach the volumes from any instances manually. Cinder does not have the ability to call Nova’s volume-detach methods, so this has to be done by the Tenant or the Admin.
1. Freeze option provided as an argument to Failover The failover command includes a “freeze” option. This option indicates that a volume may still be read or written to, HOWEVER that we will not allow any additional resource create or delete options until an admin issues a “thaw” command. This means that attempts to call snapshot-create, xxx-delete, resize, retype etc should return an InvalidCommand error. This is intended to try and keep things in as stable of a state as possible, to help in recovering from the catastrophic event. We think of this as the backend resources becoming ReadOnly from a management/control plan perspective. This does not mean you can’t R/W IO from an instance to the volume.
How to get back to “normal” a. If the original backend device is salvageable, the failover command should be used to switch back to the original primary device. This of course means that there should be some mechanism on the backend and operations performed by the Admin that ensures the resources still exist on the Primary (Backend-A) and that their data is updated based on what may have been written while they were hosted on Backend-B. This indicates that for backends to support this something like 2-way replication is going to be required. For backends that can’t support this, it’s likely that we’ll need to instead swap the primary and secondary configuration info (Reconfigure making Backend-B the Primary).

It’s important to emphasize, if the volume is not of type “replicated” it will NOT be accessible after the failover. This approach fails over the entire backend to another device.

Proposed change

One of the goals of this patch is to try and eliminate some of the challenges with the differences between manage and unmanaged replication targets. In this model we make this easier for backends. Rather than having some volumes on one backend and some on another and not doing things like stats update, we now fail over the entire backend including stats updates and everything.

This does mean that non-replicated type volumes will be left behind and inaccessible (unavailable), that’s an expectation in this use case (the device burst into flames). We should treat these volumes just like we currently treat volumes in a scenario where somebody disconnects a backend. That’s essentially what is happening here and it’s no different really.

For simplicity in the first iteration, we’re specifying the device as a driver parameter in the config file and we’re not trying to just read in a secondary configured backend device.

[driver-foo] volume_driver=xxxx valid_replication_devices=’remote_device={‘some unique access meta}’,…

NOTE That the remote_device access MUST be handled via the configured driver.

Add the following API calls replication-enable/disable ‘backend-name’

This will issue a command to the backend to update the capabilities being reported for replication.

replication-failover [–freeze] ‘backend-name’ This triggers the failover event, assuming that the current primary backend is no longer accessible.

replication-thaw ‘backend-name’ Thaw a backend that experienced a failover and is frozen

Special considerations

async vs sync This spec does not make any assumptions about what replication method the backend uses, nor does it care.
transport Implementation details and the how the backend performs replication is completely up to the backend. The requirements are that the interfaces and end results are consistent.
The Volume driver for the replicated backend MUST have the ability to communicate with the other backend and route the calls correctly based on what’s selected as the current primary. One example of an important detail here is the “update stats” call.

In the case of a failover, it is expected that the secondary/target device is now reporting stats/capabilities, NOT the now dead backend.
Tenant visibility The visibility by tenants is LIMITED!!! In other words the tenant should know very little about what’s going on. The only information that should really be propogated is that the backend and the volume is in a “failed-over” state, and if it’s “frozen”.

In the case of a failover where volumes are no longer available on the new backend, the driver should raise a NotFound Exception for an API calls that attempt to access them.

Alternatives

The result will be something that should be easier to implement and as an option will have less impact on the core code.

One appealing option would be to leave Cinder more cloud-like and not even offer replication.

Data model impact

We’ll need a new column in the host table that indicates “failed-over” and “frozen” status.

We’ll also need a new property for volumes, indicating if they’re failed-over and if they’re frozen or not.

Finally, to plan for cases where perhaps a backend has multiple replication targets, we need to provide them a mechanism to persist some ID info as to where the fail-over was sent to. In other words, make sure the driver has a way to set things back up correctly on an init.

REST API impact

replication-enable/disable ‘backend-name’: This will issue a command to the backend to update the capabilities being reported for replication.
replication-failover [–freeze] ‘backend-name’: This triggers the failover event, assuming that the current primary backend is no longer accessible.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?

Nope
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?

Nope, not that I know of
Does this change involve cryptography or hashing?

Nope, not that I know of
Does this change require the use of sudo or any elevated privileges?

Nope, not that I know of
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.

Nope, not that I know of
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Nope, not that I know of

Notifications impact

We’d certainly want to add a notification event that we “failed over”

Also freeze/thaw, as well as enable/disable events.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cinderclient? What does the user interface there look like?

TBD

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but when considering large scale deployments the proposed call may in fact be performed on hundreds of nodes.
Scheduler filters get called once per host for every volume being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries can have a profound impact on performance, especially in critical sections of code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other volume drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that targets (LVM) are stored in, how do we handle any used directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the volumes in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other volume drivers would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: john-griffith
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cinder, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cinder (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?
Need Horizon support

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

Obviously this is going to need docs and devref info in cinder docs tree

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. link to any vendor documentation)
Anything else you feel it is worthwhile to refer to

The specs process is a bit much, we should revisit it. It’s rather bloated, and while the first few sections are fantastic for requiring thought and planning, towards the end it just gets silly.

Extend a volume while volume is attached to an instance

Tue, 26 Jan 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/extend-attached-volume

This spec covers the Cinder changes in the volume extend volume flow that will allow volume drivers that support extending for attached volumes to extend the attached volume.

Extending an attached volume requires Nova side changes. A Nova spec covers the Nova changes necessary to extend a volume that is attached to one or more instances.

Cinder will interact with Nova via the existing os-server-external-events extension to notify Nova that the volume has changed size and react to the change in size.

After Nova has processed the extend size API call, the instance administrator might need to take action to make use of the new space associated with the volume. Within the instance the appropriate actions must be taken for the OS the instance is running to discover the new size of the volume. The instance administrator can then make use of the new space in the volume.

Problem description

Cinder requires that a volume be in the available state. This requires an attached volume to be detached from an instance before the volume can be extended in size. This means taking the volume offline from the standpoint of the application using the volume.

Use Cases

An operator wants to increase the size of a volume that is currently attached to an instance without detaching that volume from the instance.

Proposed change

In the Cinder extend API, allow the volume extend operation to continue processing when the volume status is ‘in‐use’ for volume drivers that can support extending an ‘in‐use’ volume. A volume could be extended if the volume status is ‘available’ or ‘in-use’ if the volume driver supports in-use extend. All other volume status values besides ‘available’ and ‘in-use’ would raise an exception.

A volume driver wishing to support in-use extend must add a new capability (in-use-extend) that indicates the driver supports extending a volume while that volume is ‘in-use’. If a driver does not have the capability to extend its volumes while they are ‘in-use’, the API will fail the extend request with a message indicating that the volume driver does not support extending a volume that is in the ‘in-use’ state.

A new policy will be introduced so a deployer can disable the volume extend operation on attached volume if the deployment does not fully support the feature. For example, Nova virt driver not supporting online volume size extension.

If the volume driver capabilities indicate that the driver supports in-use extend then the volume status will be changed to ‘extending’ and then call the driver extend_volume. After changing the volume status to ‘in‐use’, notify Nova that the volume has changed size. Nova will react to the change in size for all the instances that have the volume attached.

Alternatives

One alternative is to create a new volume and attach that volume to the instance. Once the new volume is discovered on the instance, use LVM to extend the space to the application LV. This is the method being used today to allow more space to be allocated. This is undesirable in the long run as each time a volume is extended the extension results in a new volume to be allocated, discovered on the instance, and added to a volume group, then extending the LV to include the volume. Operators have expressed concerns about using this approach as it eventually encounters limitations in the number of volumes that can be attached to a specific OS. For the volume controllers that have the ability to grow a volume while it is in‐use, it is desirable to use that ability.

Data model impact

None

REST API impact

No changes to the API interface itself, as no change is required to the arguments to the API. The API documentation will change to reflect that an in-use volume may now be extended, if the volume driver capabilities allow in-use extend.

Security impact

None

Notifications impact

TBD

Other end user impact

End user will be able to extend their volumes without having to detach them first.

Performance Impact

None

Other deployer impact

None

Developer impact

Driver owners whose backend controllers support extending a volume that is in-use by an instance may want to enable in-use extend.

Implementation

Assignee(s)

Primary assignee: * Mathieu Gagné

Other contributors: * Gerald McBrearty * Sam Matzek

Work Items

The Cinder extend volume API will check volume driver capability to see if the driver supports in-use-extend when the status of the volume is in-use.

The extend_volume rpcAPI will need to change the status of a volume back to ‘in-use’ if the volume is attached to an instance or ‘available’ if the volume is not attached to an instance, unless the driver raises an error which will cause the volume status to be changed ‘error-extending’.

Dependencies

Nova spec: Allow an attached volume to be extended [1] Adds the Nova external events support for volume-changed. It will call the os-brick extend_volume API to trigger the host kernel size information to be updated on the host where the volume has been discovered.

Testing

The following scenarios for in-use volumes will need to be covered in UT and FVT

volume driver without in-use-extend capability.
volume driver with in-use-extend capability with Nova not supporting the volume-changed external event.
volume driver with in-use-extend capability with Nova supporting the volume-changed external event.

Documentation Impact

The extend documentation will need to be updated to indicate that a volume that is attached to an instance can be extended if the driver has the capability ‘in-use-extend’.

References

Online DB schema upgrades

Thu, 07 Jan 2016 00:00:00

https://blueprints.launchpad.net/cinder/+spec/online-schema-upgrades

One of the priorities for Mitaka cycle is making Cinder able to perform rolling upgrades. This effort requires a lot of discipline to block any changes that can break interoperability of different services versions.

Some database migrations can break this interoperability. This spec aims to provide a guideline of how we handle such changes to make sure Cinder is able to do rolling upgrades.

Problem description

Non backward compatible database migrations include:

Non-additive migrations:
- Column, tables removals
- Column, tables renames
- Adding constraints
Long-running data migrations (for example from one column to another)

All of these should be run when all the services are down. This may not be a problem for small amount of data, but when we’re talking about thousands of rows, the downtime may be significant.

Use Cases

Deployer would like to be able to perform Cinder upgrade without downtime of control plane services.

User would like not to experience downtime of control plane when cloud he’s using is getting upgraded.

Proposed change

In general we want to build on Nova’s experiences and adapt their solution for our needs. Detailed description of how Nova is doing DB migrations in a live way is presented in Dan Smith’s blogpost.

At this moment please note that we’re not basing on older Nova’s approach called Online Schema Changes that was automatically doing additive and contracting DB migrations based on current and desired DB model described in models.py. This approach would eventually lead to developers not needing to write migrations manually. We will not adopt this solution as it was considered experimental, was known for destroying data, wasn’t working in late Liberty and was recently removed from Nova.

The main difference between Cinder and Nova that affects online DB schema upgrades solution is the fact that in Cinder all the services are querying the DB. In Nova only nova-conductor has access to the DB. Therefore we need to modify the approach slightly. The changes force us to span the migrations over 3 releases instead of 2 like in Nova.

First of all we should introduce a unit test that will block migrations that are contracting. Basically such test is failing when a migration using DROP or ALTER command is in tree. We will do it in a similar way like Nova did.

After adding such test we should allow only migrations that are expanding or additive, that is - are adding new columns or tables. Good part is that since Juno we’ve only merged migrations that are like that. This means that we will need the special way rarely.

And (a little complicated) special way goes as follows. Let’s use a real-world example to make this a little more useful. We’ll try to model live DB schema change that will change the relationship between consistency groups and volume types to be modelled correctly. This is a n-n relationship that currently is represented by volume_type_id column in consistencygroups table that is storing comma-separated ids of volume types related to the consistency group. Correct way to represent that is to use ConsistencyGroupVolumeTypeMapping table which maintains connections between CG and VT.

We will be working through 3 releases (+ the current one), so let’s identify them using letters:

L - current
M - in development
N
O

M release

In L release we have only the old representation of this relationship. In M we should merge a change that adds ConsistencyGroupVolumeTypeMapping model and required columns in ConsistencyGroup and VolumeType models that will form relationships to the table mentioned earlier. This additive migration can be easily executed on database while L services are running.

As on upgrade we will have L and M releases operating in the same moment we should modify M’s CG versioned object to:

Write data in both places.
Read data from old place.

That way we maintain compatibility with M->L (M is writing to old place) and L->M (M is reading from old place).

Once upgrade is completed admin should use a tool that will finish data migrations. This tool will be developed in cinder-manage and will operate on chunks of CG rows (to limit performance impact). It will simply iterate and set the relationship in new place.

In the end everything is running M services, so everythings writes to both places and reads from the old one. Also we should have all the data migrated at this point so for all CGs have relationship defined both in new and old way.

N release

Now on upgrade we will have M and N release versions. N’s versioned objects:

Write data in both places.
Read data from new place.

This way we maintain compatibility M->N (N is writing in place M is reading) and N->M (M is writing in place N is reading).

As N is reading from new place, before starting any N’s service we need to make sure that all the CGs are migrated. To enforce that, as a first N’s migration we should merge a dummy one that will block subsequent ones if there are items that weren’t migrated yet.

Please note that we still have old representation in SQLAlchemy model, so we would be unable to drop the column even if we managed to switch N services to write only to the new place.

O release

This time we’ll have N and O services cooperating.

In O we finally modify the SQLAlchemy model to remove the old ConsistencyGroup volume_type_id field. This time we’re able to, because it’s guaranteed that all services write and read from new place. We also remove it from O’s versioned object - it will now read and write data only to the new place.

After the upgrade we have everything writing and reading only from the new place. Moreover - SQLAlchemy models in O’s services doesn’t have a notion of volume_type_id column. Therefore we can have a post-upgrade migration script that finally drops the column (or we can simply add it as migration in P release if we want to limit number of steps needed to do upgrades.

Alternatives

We may simply block both contracting and data migrations. This however will prevent us from fixing bad decisions made in the past, which may hurt the project, as I don’t believe it is mature enough yet.

It’s possible to implement Online Schema Changes to make DB migrations to have two steps divided automatically. Nova was considering this as experimental. Moreover the code got removed in Mitaka-1. It also doesn’t solve long-running data migrations problem so doesn’t remove all the complications this spec is proposing.

We can also leave things as they are now and assume you need this downtime to upgrade. This probably isn’t what operators want and all the efforts we’ve already did (versioned objects, RPC compatibility) would be wasted.

Data model impact

We don’t expect impact to the data model itself, but to the way we’re doing migrations of the data model.

Changes in the model would need to be reviewed carefully to make sure they follow this guideline.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

We can expect a little performance impact on DB calls because we’ll be queuing and saving additional columns instead of dropping unused ones right away. This shouldn’t be very significant however.

There will be overhead on the DB when executing small chunks of DB migrations while the cloud is running. This is however a tradeoff, as alternative to that is downtime of the whole cloud.

Nova does this in a very similar way and they doesn’t report performance issues.

Other deployer impact

It will be needed for deployer to be aware of how to do DB upgrade in a correct way. This will be documented, but knowledge will need to be propagated.

Developer impact

There will be huge impact on developers and reviewers as both groups will need to be aware how Cinder is doing DB migrations and write/review the code accordingly to prevent rolling upgrades feature from breaking.

Implementation

Assignee(s)

Primary assignee:: michal-dulko-f (dulek)
Other contributors:: Engagement from whole core team is needed to make sure the code is reviewed also from the perspective of live upgrades.

Work Items

Add unit test blocking contracting migrations (needed in Mitaka).
Add online DB data migrations bits to cinder-manage.
Write developer guide on how to write DB migrations properly.
Add DB migrations bits to Cinder’s upgrade documentation.

Dependencies

None for itself. Whole rolling upgrades story depends on RPC compatibility (API versioning and versioned objects with compatibility mode).

Testing

Partial Grenade tests should be added to make sure upgrading services one-by-one doesn’t break rolling upgrades. This should also cover DB migrations step.

Documentation Impact

We need to write detailed developer guide on how to write DB migrations in new model to make sure that we have a clear resource which we can point developers to.

To educate administrators and deployers on how to do rolling upgrades DB migrations parts should be added to general upgrade instruction for Cinder.

References

Discussion at the Design Summit

Retype volumes with different encryptions

Mon, 23 Nov 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/retype-encrypted-volume

Enable the function that changing volume type of a volume [1] to another type with different encryptions.

Problem description

Currently Cinder prevents retyping volumes to a volume type with different encryptions.

Use Cases

Customers use unencrypted volumes, but later they would like to change the volumes to encrypted.
Customers use encrypted volumes and later want to change to unencrypted.
Customers want to change encryptions of a volume.

Proposed change

Allow retype between encrypted and unencrypted volumes. Same as current retype mechanism, it allows to retype volumes in available and in-use volumes.

If a volume is in available status, the detailed process will be:

Create a new volume according to new volume_type.
Map the two volumes to the volume host.
Open the device with dm-crypt if volumes are encrypted. This is done through os-brick/encryptors [2].
Copy data from original volume to new volume.
Close dm-crypt and detach the volumes.
Delete original volume in backend storage.

If a volume is in-use status, nothing needs to change except the bug fix [3].

Alternatives

None

Data model impact

None

REST API impact

With the feature, it allows users to retype a volume to different encryptions.

Security impact

Cinder needs to access encryption keys and decrypt the data.

Notifications impact

A flag will be added to current retype notification to show whether it needs encryption change.

Other end user impact

During retyping volumes with different encryptions, Cinder needs to get key. But Barbican can be configued only to give key materials to tenants, not admin. This may lead that admin can’t retype volumes successfully. In such cases, Cinder will catch the exception, log the error. The volume to retype will be set to original state. As os-brick/encryptors doesn’t work on RBD, Sheepdog volumes, the function to retype such volumes to different encryptions will fail, and volumes will be set to original state.

Performance Impact

It adds the step of encrypting/decrypting data during retype process, and the impact is dependent on the performance of encryption.

Other deployer impact

The feature is dependent on Castellan [4]. Meanwhile, Barbican [5] is currently the only key manager backend supported by Castellan. Both the two packages are needed.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: LisaLi <xiaoyan.li@intel.com>

Work Items

Remove current limitation which disallows the retype.
Attach/detach encrypted volume through dm-crypt.

Dependencies

None

Testing

Unit tests need to be created to cover the code change that mentioned in “Proposed change”. New tempest test cases will be added after current retype test [6].

Documentation Impact

The cinder API documentation will need to be updated to describe the change.

References

Integrate Castellan for Key Management

Thu, 19 Nov 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/use-castellan-key-manager

Castellan is a key manager interface library that is intended to be usable with multiple back ends, including Barbican. The Castellan code is based on the basic key manager interface that resides in Nova and Cinder. Now that the key manager interface lives in a separate library, the key manager code can be removed from Nova and Cinder, and Castellan can be used as the key manager interface instead.

Problem description

As encryption features in OpenStack projects are becoming more common, the projects typically need a way to interface with a key manager. Different deployers may have different requirements for key managers, so the key manager interface must also be configurable to have different back ends. The Castellan key manager interface was based off the key manager interfaces found in Cinder and Nova. Now that the shared key manager interface lives in a separate library, the original key manager interface embedded in Cinder can be removed and Castellan used instead.

Use Cases

Castellan supports existing features such as volume encryption.

Proposed change

Castellan must be added to the required libraries list.

Castellan by default pulls configuration options from a Castellan-specific configuration file in /etc/castellan, but can also take in configuration options if passed in directly. The configuration options for the key manager can still be specified in cinder.conf, and passed along to Castellan.

The old key manager interface code and back end implementations in cinder/keymgr and tests in cinder/tests/unit/keymgr can be removed. Any place in the Cinder code where the key manager interface was called will be replaced by calls to Castellan instead. Castellan does not include ConfKeyManager, an insecure fixed-key key manager that reads the key from the configuration file. The implementation for ConfKeyManager will remain in Cinder but will be converted to a Castellan plugin to exercise the relevant code paths.

Alternatives

The alternative is to leave the key manager interface as it is, but this means that Cinder’s key manager will not benefit from the updates, new features, and future additional back ends available in Castellan.

Data model impact

None

REST API impact

None

Security impact

Castellan behaves very similarly to the current Cinder key manager. Castellan has added improvements and bug fixes beyond what is currently in the Nova and Cinder key managers, making it more secure. The fixed-key key manager found in Nova and Cinder is insecure for deployments, but is useful for testing. Castellan doesn’t include the fixed-key key manager, so the ConfKeyManager will be converted to a Castellan plugin and will remain in the Cinder code.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

The deployer should be made aware of a change in the default key manager back end. The current default back end in Cinder is a fixed key, but Castellan uses Barbican as the default. This means that the deployer should ensure Barbican is running and the fixed key added to Barbican so it can continue to be used.

The options in the Cinder configuration file for volume encryption will change. The option ‘keymgr’ will be spelled out to ‘key_manager’. The key manager option group will still have an option ‘api_class’ to specify the desired back end. In the ‘barbican’ option group, a few new options will be available to increase the robustness of the back end, such as the number of times to check if a key has been successfully created.

To maintain backwards compatibility, the old options will still be listed as deprecated options. Standard deprecation policy will be followed, and these old options should be removed in the next release cycle.

Developer impact

Cinder developers should not be impacted by this change. If developers find more uses for a key manager, Castellan should be just as easy to use as the current Cinder key manager interface.

Implementation

Assignee(s)

Primary assignee:: Kaitlin Farr <kaitlin.farr@jhuapl.edu> kfarr on IRC
Other contributors:: None

Work Items

Add Castellan to Cinder requirements.

Replace calls to Cinder’s key manager with calls to Castellan.

Remove Cinder key manager code.

Update documentation.

Dependencies

This change depends on Castellan, version >= 0.2.0. Castellan is already in OpenStack’s global requirements.

Testing

This change can be unit tested using a simple in-memory back end. As actual deployments should be using Barbican, this feature should be tested using a Barbican back end, too.

Documentation Impact

These changes will be documented. Cinder documentation for volume encryption will be updated to reference Castellan [4].

References

[1] Castellan source code:: https://github.com/openstack/castellan
[2] Castellan in OpenStack’s global requirements:: https://github.com/openstack/requirements/blob/master/global-requirements.txt
[3] Current Cinder key manager implementation: https://github.com/openstack/cinder/tree/master/cinder/keymgr
[4] Volume encryption configuration reference: http://docs.openstack.org/liberty/config-reference/content/section_volume-encryption.html

Cinder Volume Active/Active support - Cleanup

Tue, 13 Oct 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support

Right now cinder-volume service can run only in Active/Passive HA fashion.

One of the reasons for this is that we have no concept of a cluster of nodes that handle the same storage back-end, and we assume only one volume service can access a specific storage back-end.

Given this premise, current code handles the cleanup for failed volume services as if no other service is working with resources from his back-end, and that is problematic when there are other volume services working with those resources, as is the case on an Active/Active configuration.

This spec introduces a new cleanup mechanism and modifies current cleanup mechanism so proper cleanup is done regardless of cinder configuration, Active/Passive or Active/Active.

Problem description

Current Cinder code only supports Active/Passive configurations, so the cleanup takes that into account and cleans up resources from ongoing operations accordingly, but that is incompatible with an Active/Active deployment.

The incompatibility comes from the fact that volume services on startup look on the DB for resources that are in the middle of an operation and are from their own storage back-end - detected by the host field - and proceed to clean them up depending on the state they are in. For example a downloading volume will be changed to error since the download was interrupted and we cannot recover from it.

With the new job distribution mechanism the host field will contain the host configuration of the volume service that created the resource, but that resource may now be in use by another volume service from the same cluster, so we cannot just rely on this host field for cleanup, as it may lead to cleaning wrong resources or skipping the ones we should be cleaning.

When we are working with an Active/Active system we cannot just clean all resources from our storage-backend that are in an ongoing state, since they may be legitimate undergoing jobs being handled by other volume services.

We are going to forget for a moment how we are doing the cleanup right now and focus on the different cleanup scenarios we have to cover. One is when a volume service “dies” -by that we mean that it really stops working, or it is fenced- and failover boots another volume service to replace it as if it were the same service -having the same host and cluster configurations-, and the other scenario is when the service dies and no other service takes its place, or the service that takes its place shares the cluster configuration but has a different host.

Those are the cases we have to solve to be able to support Active/Active and Active/Pasive configurations with proper cleanups.

Use Cases

Proposed change

Since checking for the status and the host field of the resource is no longer enough to know if it needs cleanup -because the host field will be referring to the host configuration of the volume service that created the resource and not the owner of the resource as explained in the Job Distribution specs- we will create a new table to track which service from the cluster is working on each resource.

We’ll call this new table workers and it will include all resources that are being processed with cleanable operations, and therefore would require cleanup if the service that is doing the operation crashed.

When a cleanable job is requested by the API or any of the services -for example a volume deletion can be requested by the API or by the c-vol service during a migration- we will create a new row in the workers table with the resource we are working on and who is working on it. And once the operation has been completed -successfully or unsuccessfully- this row will be deleted to indicate processing has concluded and a cleanup will no longer be needed if the service dies.

We will not be adding a row for non cleanable operations and resources that are used in cleanable operations but won’t require cleanup, as this would create a significant increase in DB operations that would end up affecting performance of all operations.

These workers rows serve as flags for the cleanup mechanism to know it must check that resource in case of a crash and see if it needs cleanup. There can only exist 1 cleanable operation at a time for a given resource.

To ensure that both scenarios mentioned above are taken care of, we will have cleanup code on cinder-volume and Scheduler services.

Cinder-volume service cleanups will be similar to the ones we currently have on startup -init_host method- but with small modifications to use the workers_table so services can tell which resources require cleanup because they were left in the middle of an operation. With this we take care of one of the scenarios, but we still have to consider the case where no replacement volume service comes up with the same host configuration, and for that we will add a mechanism on the scheduler that will take care of requesting other volume service from the cluster, that manage the same backend, to do the cleaning for the fallen service.

The cleanup mechanism implemented on the scheduler will have manual and automatic options, manual option will require the caller to specify which services should be cleaned up using filters, and automatic operation will let the scheduler decide which services should be cleaned up based on their status and how long they have been down.

Automatic cleanup mechanism will consist of a periodic task that will sample services that are down, with a frequency of service_down_time seconds, and will proceed to clean up resources that were left by those services that are down after auto_cleanup_checks x service_down_time seconds have passed since the service went down.

Since we can have multiple Scheduler services and the cinder-volume service all trying to do the cleanup simultaneously, code needs to be able handle these situations.

On one hand, to prevent multiple Schedulers from cleaning the same services’s resources they will be reporting all automatic cleanup operations requested to the cinder-volumes to the other Scheduler services and will ask other scheduler services which services have already been cleaned on service start.

On the other hand, to prevent cleanup concurrency issues if a cleanup is requested on a service that is already being cleaned up, we will issue all cleanup operations with a timestamp indicating that only workers entries before that should be cleaned up, so when a service starts doing the cleanup for a resource it updates the entry an prevents additional cleanup operations on the resource.

Row deletion operations in workers table will be a real deletions in the DB, not soft deletes like we do for other tables, because the number of operations, and therefore of rows, will be quite high and because we will be setting constraints on the rows that would not hold true if we had the same resource multiple times (there are workarounds, but it doesn’t seem to be worth it).

Since these will be big, complex changes, we will not be enabling any kind of automatic cleanup by default, and it will need to be either enabled in the configuration using auto_cleanup_enabled option or triggered using the manual cleanup API -using filters- or the automatic cleanup API.

It will be possible to trigger the automatic cleanup mechanism via the API even when it is disabled, as the disabling only prevents it from being automatically triggered.

It is important to mention that using “reset-state” operation on any resource will remove any existing workers table entry in the DB.

When proceeding with a cleanup we will ensure that no other service is working on that resource (claiming the worker’s entry) and that the data on the workers entry is still valid for the given resource (status matches) since a user may have forcefully issued another action on the resource in the meantime..

Alternatives

There are multiple alternatives to proposed change, the most appealing ones are:

Use Tooz with a DLM that allows Leader Election to prevent more than one scheduler from doing cleanup of down services. Downsides to this solution are considerable:
- Increased dependency on a DLM.
- Limiting DLM choices since now it needs to have Leader Election functionality.
- We will still need to let other schedulers know when the leader does cleanups because when electing a new leader will need this information to determine if down services have already been cleaned.
Create workers DB entries for every operation on a resource. Disadvantages of this alternative are:
- Considerable performance impact.
- Greatly increase cleanup mechanism complexity, as we would need to mark all entries as being processed by the service we are going to clean (this has its own complexity because multiple schedulers could be requesting it or a scheduler and the service itself), then see which of those resources would require cleanup according to the workers table and check if no other service is already working on that resource because a user decided to do a cleanup on his own (for example a force delete on a deleting resource) and if there’s no other service working on the resource and the resource has a status that is cleanable, then do the cleanup. Doing all this without races is quite complicated.

Data model impact

Create new workers table with following fields:

id: To uniquely identify each entry and speed up some operations
created_at: To mark when the job was started at the API
updated_at: To mark when the job was last touched (API, SCH, VOL)
deleted_at: Will not be used
resource_type: Resource type (Volume, Backup, Snapshot…)
resource_id: UUID of the resource
status: The status that should be cleaned on service failure
service_id: service working on the resource

REST API impact

Two new admin only API endpoint will be created, /workers/cleanup and /workers/auto_cleanup.

For /workers/cleanup endpoint we will be able to supply filtering parameters, but if no arguments are provided cleanup will issue a clean message for all services that are down. But we can restrict which services we want to be cleaned using parameters service_id, cluster_name, host, binary, disabled.

Cleaning specific resources is also possible using resource_type and resource_id parameters.

Cleanup cannot be triggered during a cloud upgrade, but a restarted service will still cleanup it’s own resources during an upgrade.

Both API endpoints will return a dictionary with 2 lists, one with services that have been issued a cleanup request (cleaning) and another list with services that cannot be cleaned right now because there is no alternative service to do the cleanup in that cluster (unavailable), that way the caller can know which services will be cleaned up.

Data returned for each service in the lists are id, name, and state fields.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Small impact on cleanable operations since we have to use the workers table to flag that we are working on the resource.

Other deployer impact

None

Developer impact

Any developer that wants to add new resources requiring cleanup or wants add cleanup for the status -new or existing- of an existing resource will have to use the new mechanism to mark the resource as cleanable, add which states are cleanable, and add the cleanup code.

Implementation

Assignee(s)

Primary assignee:: Gorka Eguileor (geguileo)
Other contributors:: Michal Dulko (dulek) Anyone is welcome to help

Work Items

Make DB changes to add the new workers table.
Implement adding rows to workers table.
Change host_init to use an RPC call for the cleanup.
Modify Scheduler code to do cleanups.
Create devref explaining requirements to add cleanup resources/statuses.

Dependencies

Job Distribution:

This depends on the job distribution mechanism so the cleanup can be done by any available service from the same cluster.

Testing

Unittests for new cleanup behavior.

Documentation Impact

Document new configuration option auto_cleanup_enabled and auto_cleanup_checks as well as the cleanup mechanism.

Document behavior of reset-state on Active-Active deployment.

References

General Description for HA A/A: https://review.openstack.org/232599

Job Distribution for HA A/A: https://review.openstack.org/327283

NFS Snapshots

Tue, 29 Sep 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/nfs-snapshots

Add support for snapshots (create, delete, clone from) to the NFS volume driver, so that it is as functional as other Cinder drivers.

Problem description

The NFS driver does not support many features expected in a modern Cinder driver.

create snapshot
delete snapshot
clone from snapshot

Use Cases

Proposed change

The basic model for how snapshots can be implemented here has been proven in the GlusterFS driver. The general idea here is to re-use that same model and much of the code for the NFS driver, which works the same as the GlusterFS driver in general.

In Juno, most of the snapshot-related code was refactored into a RemoteFSSnapshot class.

The NFS driver should inherit code from this base class (and probably make minor changes) to gain snapshot support.

Alternatives

No real alternative, baseline NFS servers do not have the ability to snapshot files. This method is already used in other Cinder drivers and works, so the only alternative is to decide we don’t want snapshots for this driver.

Data model impact

None

REST API impact

Nothing significant.

initialize_connection will return additional fields specifying the format of the volume file so that Nova knows how to attach it properly.

Security impact

None, see notes here.

This change needs to be tested with the NFS security infrastructure added in Kilo to ensure nothing breaks and that all files before, after, and during snapshot operations have the expected permissions. There is no reason to expect problems here.

In Juno a security concern related to qcow2 file handling was addressed. ( https://bugs.launchpad.net/cinder/+bug/1350504 )

This is now considered fixed from a security perspective as there are no known ways to exploit it, but is still being fixed in a more robust manner in the “Support storing volume format info” spec. This work is possible in parallel as there is no strict interdependency here. ( https://review.openstack.org/#/c/103750/ )

Notifications impact

None

Other end user impact

We do not yet support backups of qcow2-based volumes. (Unsure if targeted in Kilo elsewhere.)

Performance Impact

The NFS driver needs to have a per-volume-id lock for operations that manipulate volume/snapshot data. This does not cause any performance regression in existing functionality and should not be a large concern in most scenarios when using snapshots.

See https://review.openstack.org/#/c/106095/ (GlusterFS example)

Other deployer impact

New config option, nfs_qcow2_volumes, to specify if you would like to always create volume files as qcow2 images rather than raw.

Developer impact

Implementation

Assignee(s)

Primary assignee:: eharney and akerr (?)
Other contributors:: NetApp has indicated they will CI this driver. (bswartz)

Work Items

Try to inherit RemoteFSSnapshot code into NFS driver.
Iterate on issues there.
Test security concerns.

Dependencies

Storing volume format info: https://review.openstack.org/#/c/103750/

Not a strict dependency, and which lands first is not vital, but this effort also needs to land in Kilo along with this work and is strongly related.

I will submit another spec addressing the snapshot API between Cinder and Nova which will be used here, but nothing there blocks this work.

Testing

Standard third-party CI of the NFS driver

Consider testing this in-gate as a later effort

Documentation Impact

New config option, nfs_qcow2_volumes.

References

Original snapshot effort which this is based on:: https://blueprints.launchpad.net/cinder/+spec/qemu-assisted-snapshots
RemoteFS snapshot code refactoring, done to support this effort:: https://review.openstack.org/#/c/106066/
SMBFS volume driver, an example of another driver re-using this code:: https://review.openstack.org/#/c/106046/

Attach/detach volumes without Nova

Wed, 16 Sep 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/use-cinder-without-nova

Services like Ironic want to use Cinder w/o Nova. Also it would be useful to attach Cinder volumes not only for Nova/Ironic instances: e.g. make simple CLI to attach volume to some host (VM, workstation, etc) outside OpenStack cloud.

Problem description

Currently there is not a tool which could automate all the attach/detach process without using Nova.

Use Cases

Attach volume to Ironic instance
Attach volume to any virtual/baremetal host which is not provisioned by Nova or Ironic
Attach volume to Magnum container
Attach volume to Docker container

Proposed change

Provide a command-line (CLI) tool and Python API which could interact with Cinder and attach volumes to the local host. It will be implemented as python-cinderclient extension and won’t require to add any dependency if extension is not used. This extension won’t be installed by default. Users will be able to install it using PIP or OS package management tool:

pip install python-brick-cinderclient-ext

CLI interface e.g.:

cinder local-attach <volume_id>
cinder local-detach [--attachment-id attachment-id] <volume_id>
cinder get-connector

In case of Ironic when we don’t want to install any package into the user’s instances. Users could install package themselves via pip or OS package managers (apt, yum, etc.).

After this command cinder client will call Cinder API to change volume status to ‘in-use’ because we don’t know when user will attach volume to the host.

Volume attachment:

cinder local-attach [--mountpoint /mnt/disk]
                    [--multipath True]
                    [--enforce-multipath True]
                    [--mode rw]
                    <volume_id>

Volume detachment:

cinder local-detach [--attachment-id id]
                    [--multipath True]
                    [--enforce-multipath True]
                    [--device-info device-info]
                    <volume_id>

‘attachment_uuid’ is required option only for multi-attachment scenario.

Detach procedure should contain the following steps:

disconnect_volume
terminate_connection with a ‘force’ flag
detach

Force detach feature is out of scope of this spec and will be described in a separate spec.

Get connector details:

cinder get-connector

Since mount/unmount is admin only operation we need to run python-cinderclient as root to not add oslo.rootwrap dependency.

Alternatives

Cinder already has all needed APIs. Any API consumer could use existing methods to implement attach/detach actions without Nova.
We can implement needed API inside new python-brickclient.
We can introduce new binary (e.g.: brick) inside python-cinderclient project. In such case we’ll have two different clients inside single repository. From the packager’s point of view it will increase numbers of possible python-cinderclient packages. E.g.: python-cinderclient-iscsci, python-cinderclient-rbd, etc.

Data model impact

None

REST API impact

None

Security impact

TBD

Notifications impact

None

Other end user impact

This change will provide new CLI options and Python for python-cinderclient.

Performance Impact

None

Other deployer impact

Deployers could install new package to have new functionality

Developer impact

All drivers should implement initialize_connection which could be invoked several times without any side-effect. We’ve filed bugs for some of the drivers.

Implementation

Assignee(s)

Primary assignee:: Ivan Kolodyazhny <e0ne>

Work Items

Implement proof-of-concept based on a python-cinderclient
Get feedback and complete the spec
Fix typos in the spec
Make changes to Cinder API if needed
Unit-tests and Tempest tests are required
Functional tests on python-cinderclient gates will be implemented
New repository for python-brick-cinderclient-ext is needed

Dependencies

os-brick
python-cinderclient
Linux: open-iscsi, ceph-common, nfs-common or other tool to attach and detach volumes via drivers’ protocols Windows: tools for Windows to be able to attach and detach volume via drivers’ protocols.

Testing

Unit-tests should be implemented
New python-cinderclient API will be tested via functional tests on gates to test attach/detach features without Nova instances.
Functional tests for Ironic will be implemented to test attach/detach features to Ironic instances.

Documentation Impact

User manual will be updated to contain information about python-cinderclient extension.

References

https://review.openstack.org/254878 and related patches
http://lists.openstack.org/pipermail/openstack-dev/2015-July/068763.html
https://etherpad.openstack.org/p/liberty-cinder-ironic-integration
https://blueprints.launchpad.net/ironic/+spec/volume-connection-information
python-brickclient Proof-of-Concept implementation: https://github.com/e0ne/python-brickclient
Cinder Meeting Minutes: http://goo.gl/ztuYTa
http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-11-11-16.00.log.html

Update drivers to new base class structure

Fri, 28 Aug 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/abc-driver-update

The new abc structure was introduced by bp/abc-volume-drivers [1]. All drivers needs to get updated in order to benefit from the new structure.

Problem description

Instead of raising NotImplementedErrors during runtime this features allows to discovers the drivers feature set during startup and makes it discoverable for CI/code check systems.

Use Cases

The support matrix (see [2] for a draft implementation) can be extracted to see the graduation process of new functionality moving to a common function implemented by all drivers.

Proposed change

All cinder volume drivers needs to get updated with the following approach:

class FooDriver(driver.RetypeVD, driver.TransferVD, driver.ExtendVD,
                driver.CloneableVD, driver.CloneableImageVD,
                driver.SnapshotVD, driver.BaseVD)

A drivers must inherit from BaseVD and implement the basic functions. In order to mark that a driver does implement further feature sets it must inherit from the corresponding class.

If all drivers implement a certain feature set the functions will be moved to BasicVD at the end.

Alternatives

No porting at all, which would make the [1] pointless.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

See [1]

Other deployer impact

None.

Developer impact

This change will change all implemented drivers slightly. The functionality itself shouldn’t be changed at all but all driver need to be adopted to the new class model.

Implementation

Assignee(s)

Primary assignee:: Marc Koderer (m-koderer)
Other contributors:: All driver maintainers

Work Items

Etherpad if necessary.

Dependencies

None.

Testing

Individual driver unit tests needs to get adapted.

Documentation Impact

None.

References

[1]: https://review.openstack.org/#/c/114168/ [2]: https://review.openstack.org/#/c/160346/

Allow brick to support other iSCSI transports besides TCP

Fri, 28 Aug 2015 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/brick-add-open-iscsi-transport-support

Currently open-iscsi in openstack does not allow the use of offload iscsi transports (cxgb4i, qla4xx, bnx2i etc) or other software iscsi transports (like iser). This blueprint aims to add support for this

Problem description

open-iscsi has hardware iSCSI transport support. The interface argument specifies the hardware iSCSI interface to use for the operation. Any hardware vendors that currently provide hardware interfaces for open-iscsi cannot use them in openstack as the interfaces (iface) argument is not currently supported in the iSCSI driver.

Use Cases

Use of such iSCSI hardware transports requires providing a corresponding interface file (referred to as iface), which can be autogenerated via iscsiadm but need correctness checks or can also be generated on the fly, given the 3-tuple of ipaddress, hwaddress & transport name to be used. The iface format itself is generic and does not use any propreitary fields irrespective of transport being used.

These changes are already in nova, code can be seen here : https://review.openstack.org/#/c/146233/

This will also enable use of iSER directly through the iSCSI driver, though deprecating the iSER driver is not the aim of this spec, and will probably start with iSER disabled, (and also because iSER requires target support unlike hardware transports)

Proposed change

On the initiator side the only difference between using software iSCSI and hardware iSCSI transport is the interface flag (–interface=[iface])

e.g. “iscsiadm -m discovery -t st -p ip:port -I <iface file>”

iface files can automatically be generated for all hardware present on a system that is supported by open-iscsi by ‘iscsiadm -m iface’. The default format for iface file names is in the form <transport_name.hw_address>

The default iface name is ‘default’. Not using the -I parameter (which is how open-iscsi is currently being used in nova) is the same as specifying

‘iscsiadm -m discovery -t st -p ip:port -I default’

Required changes are :

A new parameter to specify which transport should be used while doing iscsi discovery/login, falling back to TCP if operation does not succeed, or if parameter is not provided (-I default)
Offloaded transports use the device path with a prefix pci-0000:00:00.0-, change related code to handle this

Alternatives

Use manual discovery of iscsi-targets, and provide -I parameter when doing discovery. Transport names are written in the resultant records generated by open-iscsi, however this does not guarantee hardware transports will be used, as this can be overridden with depending on how cli params are constructed.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Actual iSCSI performance can generally be improved by using transports over iscsi_tcp, no other impact.

Other deployer impact

This can be used to deprecate the separate iSER connector, as use of iser simply implies providing -I iser param to iscsiadm

Developer impact

Implementation

Assignee(s)

Primary assignee:: anish7 <anish@chelsio.com>

Work Items

Add -I parameter to all iscsiadm login/discovery commands currently being used in brick
Add prefix support to _get_device_path()

Dependencies

None as such. However, the minimum information required for using a tansport if it is not already configured is a 3 tuple of ip, transport name & hardware address.

Testing

Current iSCSI tests will work just fine, run separately using a faux transport (open-iscsi already supports this ability)

Documentation Impact

Use of transport ifaces will need relevant documentation.

References

https://bugs.launchpad.net/os-brick/+bug/1370226 (Launchpad bug this addresses)
https://review.openstack.org/#/c/146233/ (Corresponding code in nova)
http://www.open-iscsi.org/docs/README (Section 5.1 iSCSI iface setup)
http://red.ht/1BJxsjL (Configuring an iface for iSCSI Offload, RedHat documentation)

Extract Brick library from Cinder

Fri, 28 Aug 2015 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/extract-brick

This Specification talks about 2 points. First the extraction of the brick/cinder directory into it’s own standalone library. Second, changing cinder to use the external library.

Problem description

The idea of ‘brick’ was created back in the Havana timeframe. All along it was meant to be a standalone library that Cinder, Nova and any other project in OpenStack could use. Currently brick lives in a directory inside of Cinder, which means that only Cinder can use it.

Use Cases

Use the same library code both in Nova and Cinder.

Proposed change

We want to extract the brick directory and encapsulate it into it’s own pypi library that any python project can use.

So we need to do:

First create a separate python project and release it into pypi
Add brick to cinder’s requirements.txt
Remove the existing cinder/brick directory
Everywhere in Cinder that uses cinder/brick will need to be modified to use the new pypi library.

Alternatives

We could simply keep brick inside of Cinder and not share it’s code. The problem with this is that any changes/fixes to brick will then need to be backported into the same code in Nova. This is the existing problem.

Data model impact

This doesn’t change the data model for Cinder.

REST API impact

None

Security impact

Any security related issues that impact brick may impact anything in Cinder that would use the new library. Brick will be placed into stackforge for code reviews, so anyone can contribute fixes.

Notifications impact

None

Other end user impact

None

Performance Impact

There should be no performance impact, as the same code will exist in the pypi brick library as it does today in Cinder. The import will pull the library from a system installed location instead of the cinder codebase.

Other deployer impact

Brick will be a requirement for Cinder. So whoever builds distribution packages for cinder will also need to ensure that brick gets installed.

Developer impact

Currently, any Cinder developer can add new features and fix bugs in brick as part of the Cinder project. Going forward, developers will need to clone the source of brick from stackforge and make their changes there.

The downside to this is that If there are new features, APIs or interface changes in the brick library, a release will have to be published in order for Cinder to get those changes.

Implementation

Assignee(s)

Primary assignee:: Walter A. Boring IV - walter-boring
Other contributors:: e0ne

Work Items

I have already started the work of creating a standalone brick library here: https://github.com/hemna/cinder-brick

This repo will need to be registered into stackforge, so brick can benefit from the community process and gerrit.
Once in stackforge, an official release needs to be made so that the package exists in pypi.
Then a patch to Cinder can be made to add it to the requirements.txt
The another patch to follow that removes brick/cinder and adjusts all of Cinder to use the new library.

Dependencies

None

Testing

All of the existing unit tests in cinder for brick can remain to ensure that it works as advertised. The existing Cinder brick unit tests will be migrated to the new project and run there as well.

Documentation Impact

None.

References

Existing standalone project: https://github.com/hemna/cinder-brick
The original Cinder brick proposal https://wiki.openstack.org/wiki/CinderBrick

Standard Capabilities

Fri, 28 Aug 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/standard-capabilities

Problem description

For the Liberty release there is a proposal [1] to allow storage backends to push capabilities of their pools [1]. Eventually we would want some common capabilities to graduate into becoming a well defined capability. The point of this spec is to agree on the initial well defined capabilities.

Use Cases

Having the well defined capabilities will allow the deployer to see what common capabilities are shared beyond their deployed backends in Cinder.

Proposed change

The initial well defined capabilities are:

QOS
Compression
Replication
Thin provisioning

Keep in mind this is just an agreement that these are common features that a backend could support. As shown in the proposal of how this will work [1], backends will still be able to push up the specific keys they look for in volume types for these capabilities.

Alternatives

None

Data model impact

None. This information comes directly from the volume drivers, reported to the scheduler, to the Cinder API.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Volume driver maintainers will need report capabilities from their driver to the scheduler. They can get this information directly from the backend and pass it right up to the scheduler if it already follows the format specified earlier. If not, it’s up to the driver to parse the response from the backend in a format the scheduler will understand. If capabilities are not being reported, the default False on features will be done.

Implementation

Assignee(s)

Primary assignee:: thingee

Work Items

Standardize on the Capabilities here. Right here, right now.

Dependencies

None.

Testing

None

Documentation Impact

The developer docs for driver maintainers will need to be updated to include the list of common capabilities the maintainer needs to have their driver push that they support. By default capabilities are marked as False, as in not being supported by the driver.

References

[1] - https://review.openstack.org/#/c/183947/

Add Cinder Support for DB2 (v10.5+)

Mon, 24 Aug 2015 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/cinder/+spec/db2-database

The community currently supports MySQL and PostgreSQL production databases. Several other core projects already support DB2 (Keystone, Glance, Ceilometer, Heat). This blueprint adds support to Cinder for DB2 as a production database.

Problem description

Currently there is no support in the community for a deployer to run Cinder against a DB2 backend database.
For anyone running applications against an existing DB2 database that wants to move to OpenStack, they’d have to use a different database engine to run Cinder in OpenStack.
There is currently an inconsistent support matrix across the core projects since the majority of core projects support DB2 but Cinder does not yet.

Use Cases

Proposed change

The changes required to enable Cinder to run on DB2 are limited given that Cinder’s database structure is not complicated. No changes are currently required to the migration scripts to enable deployment on a DB2 database.

Unit test will need to be updated to support running tests against a DB2 backend with the ibm_db_sa driver and all Cinder patches will be tested against a Tempest full run with 3rd party CI running DB2, maintained by IBM. Note that these 3rd Party CI runs have already been enabled for Cinder.

There is already code in Oslo’s db.api layer to support common function with DB2 like duplicate entry error handling and connection trace, so that is not part of this spec. Cinder, however, will need to sync up to the latest Oslo DB code to enable DB2 support.

Alternatives

Deployers can use other supported database backends like MySQL or PostgreSQL, but this may not be an ideal option for customers already running applications with DB2 that want to integrate with OpenStack. In addition, you could run other core projects with multiple schemas in a single DB2 OpenStack database, but you’d have to run Cinder separately which is a maintenance/configuration problem.

Data model impact

There are no impacts on the current Cinder data model. Any issues that arise due to future changes will be uncovered by the 3rd Party CI tests and will be able to be addressed at that point in time.

The one exception to this statement is in the unit test path. I made two commits in Icehouse to start implementing the infrastructure to enable DB2 unit testing. The commits were to add a bool_type dictionary (2a7b11922bd9389287915c45de92ca5eed3d448e) and a time_type dictionary (a9527de9ed3a2eae951564c3c74b7319113e8bf5). I will need to add the appropriate types for DB2, as was done for MySQL, as part of the unit test changes I will be making for DB2.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

Execution of migration unit tests on DB2 may take longer than on other backend databases due to the time required to create the database schema. This impact, however, will only be seen when running unit tests for DB2.

Other deployer impact

There are no plans to migrate the Cinder database from a pre-existing installation (I.E. MySQL) to DB2. So deployers will need to be starting with a fresh Cinder installation to enable DB2 as a backend database.

Developer impact

The only impact on developers is if they are adding DB API code or migrations that don’t work with DB2, they will have to adjust those appropriately, just like we do today with MySQL and PostgreSQL. IBM ATCs would provide support/guidance on issues like this which require specific conditions for DB2, although for the most part, the DB2 InfoCenter provides adequate detail on how to work with the engine and provides details on error codes.

DB2 SQL error message explanations can be found here: http://pic.dhe.ibm.com/infocenter/db2luw/v10r5/index.jsp?topic=%2Fcom.ibm.db2.luw.messages.sql.doc%2Fdoc%2Frsqlmsg.html
Information on developing with DB2 using python can be found here: http://pic.dhe.ibm.com/infocenter/db2luw/v10r5/index.jsp?topic=%2Fcom.ibm.swg.im.dbclient.python.doc%2Fdoc%2Fc0054366.html
Main contacts for DB2 questions in OpenStack:
- Matt Riedemann (mriedem@us.ibm.com) - Nova core member
- Brant Knudson (bknudson@us.ibm.com) - Keystone core member
- Jay Bryant (jsbryant@us.ibm.com) - Cinder core member
- Rahul Priyadarshi (rahul.priyadarshi@in.ibm.com) - ibm_db_sa maintainer

Implementation

Assignee(s)

Primary assignee:: Email: jsbryant@us.ibm.com Launchpad ID: jsbryant

Work Items

Make the test_migrations.py module work with a configured DB2 backend for running unit tests.
1. Add the appropriate column types for DB2.
2. Add support for creating the DB2 schema for Cinder testing.

Dependencies

DB2 support was added to sqlalchemy-migrate 0.9 during Icehouse: https://blueprints.launchpad.net/sqlalchemy-migrate/+spec/add-db2-support
There are no requirements changes in Cinder for the unit tests to work. The runtime requirements are the ibm-db-sa and ibm_db modules, which are both available from pypi. sqlalchemy-migrate optionally imports ibm-db-sa. The ibm-db-sa module requires a natively compiled ibm_db which has the c binding that talks to the DB2 ODBC/CLI driver.
Note that only DB2 10.5+ is supported since that’s what added unique index support over nullable columns which is how sqlalchemy-migrate handles unique constraints over nullable columns.

Testing

IBM is already running 3rd party CI for DB2 on Cinder.
DB2 Unit Test for Cinder will be enabled.

Documentation Impact

The install guides in the community do not go into specifics about setting up the database. The RHEL/Fedora install guide says to use the openstack-db script provided by openstack-utils in RDO which uses MySQL. The other install guides just say that SQLite3, MySQL and PostgreSQL are widely used databases. So for the install guides, those generic statements about supported databases would be updated to add DB2 to the list. Similar generic statements are also made in the following places which would be updated as well:
There are database topics in the security guide, so there would be DB2 considerations there as well, specifically:
- http://docs.openstack.org/security-guide/databases/database-backend-considerations.html
- http://docs.openstack.org/security-guide/databases/database-transport-security.html

References

There are Chef cookbooks on stackforge which support configuring OpenStack to run with an existing DB2 installation: http://git.openstack.org/cgit/stackforge/cookbook-openstack-common/
A wiki document originally written to describe DB2 for OpenStack: https://wiki.openstack.org/wiki/DB2Enablement
DB2 10.5 InfoCenter: http://pic.dhe.ibm.com/infocenter/db2luw/v10r5/index.jsp
Some older manual setup instructions for DB2 with OpenStack: http://www.ibm.com/developerworks/cloud/library/cl-openstackdb2/index.html
ibm-db-sa: https://code.google.com/p/ibm-db/source/clones?repo=ibm-db-sa

Efficient volume copy for volume migration

Mon, 24 Aug 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/efficient-volume-copy-for-cinder-assisted-migration

Currently, cinder supports these 3 ways for volume migration.

#

Migration method

Data copy method

Handler of migration

1

storage driver

depend on vendor storage

inside storage array

2

copy_volume_data()

dd command

cinder volume node

3

swap_volume()

libvirt block rebase copy

nova compute node

#	Migration method	Data copy method	Handler of migration
1	storage driver	depend on vendor storage	inside storage array
2	copy_volume_data()	dd command	cinder volume node
3	swap_volume()	libvirt block rebase copy	nova compute node

During volume migration case #2, Cinder uses dd command for data copy of volume migration, but the copy always copies full blocks even if the source data contains many null and zero blocks. The dd command has an option conv=sparse to skip null or zero blocks for more efficient data copy.

The purpose of this request is to introduce a mechanism to handle sparse copy option properly for volume migration using dd command.

Problem description

If we create a volume from thin-provisioning pool, volume blocks are not pre-allocated, and the volume blocks will be allocated on demand. In this situation, if we migrate detached volume using dd command, dd will copy full block from source to destination volume even if the source volume contains many null or zero blocks. As a result, usage of the destination volume will be always 100%. Here is an example volume migration using thin LVM driver.

Before migration

LV            VG    Attr       LSize   Pool     Origin Data%
vg1-pool      vg1   twi-a-tz--   3.80g                10.28
volume-1234   vg1   Vwi-a-tz--   1.00g vg1-pool       19.53

After migration without conv=sparse option

LV            VG    Attr       LSize   Pool     Origin Data%
vg2-pool      vg2   twi-a-tz--   3.80g                31.45
volume-1234   vg2   Vwi-a-tz--   1.00g vg2-pool      100.00

Use Cases

Using sparse copy is able to reduce volume usage of destination storage array compared to using full block copy.

Proposed change

If the volume pre-initilization(zero cleared) is ensured beforehand at the destination c-vol, we can skip copy of null and zero blocks to destination volume by using conv=sparse option for dd.

However, if the destination volume is not zero cleared beforehand, we should disable sparse copy and copy full block from source to destination volume to avoid data corruption problem.

For example, following case, we should disable sparse copy and copy full block because new volume of thick LVM might not be initialized, we must not use conv=sparse option for similar cases.

Source c-vol driver: Thin LVM
Dest c-vol driver: Drivers who provides an uninitialized new volume. (ex. Thick LVM)

In order to handle sparse option properly, following changes are required.

Add cinder_sparse_copy_volume capability into capabilities list as a well-defined option.

'cinder_sparse_copy_volume': {
   'default': 'False',
   options: {}
},

Get capability list from the “destination cinder volume driver” using get_capabilities method before starting volume copy.

get_capabilities method is proposed following Spec.

Updating Get Volume Driver Capabilities Spec: https://review.openstack.org/#/c/183947/1

These are steps of migrate_volume related to sparse copy.

source c-vol driver                   dest c-vol driver
 |                                           |
 | ========================================> |
 |   Ask capability of sparse copy via       |
 |   get_capabilities                        |
 | <======================================== |
 |   Dest c-vol returns capabilities to      |
 |   source c-vol                            |
 | ========================================> |
 | Start volume copy with or without sparse  |
 | based on the retun from dest c-vol        |
 |                                           |

Alternatives

Copy full blocks from source volume to destination volume. And then, try fstrim or blkdiscard to release unused blocks to backend storage. However, if the backend storage doesn’t support UNMAP SCSI command, we can’t release unused blocks after migration.

Data model impact

None

REST API impact

Add cinder_sparse_copy_volume parameter as a well_defined capability.

'cinder_sparse_copy_volume': {
   'default': 'False',
   options: {}
},

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

This feature improves required time to copy data between source and destination volume.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: mitsuhiro-tanino

Work Items

Add cinder_sparse_copy_volume capability into capabilities list.
Add handler of get_capabilities in migrate_volume method.
Update LVM reference implementation with cinder_sparse_copy_volume capability.
Update NFS driver with cinder_sparse_copy_volume capability:

NFS driver already has _sparse_copy_volume_data option but this way doesn’t ask capability to destination cinder volume before volume copy. We should use same way to handle sparse copy.

Dependencies

None

Testing

Unit tests

Documentation Impact

Need to update openstack cloud administrator guide. http://docs.openstack.org/admin-guide/blockstorage_volume_migration.html

References

Related Cinder Spec

Updating Get Volume Driver Capabilities Spec: https://review.openstack.org/#/c/183947/1
Related fix

https://review.openstack.org/#/c/183633/

Incremental backup improvements for L

Mon, 24 Aug 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-incremental-backup-improvements-for-l

This specification proposes to improve the current incremental backup by adding is_incremental and has_dependent_backups flags to indicate the type of backup and enriching the notification system via adding parent_id to report.

Problem description

In Kilo release we supported incremental backup, but there are still some points that need to be improved. 1. From the API perspective, there is no place to show the backup is incremental or not. 2. User also doesn’t know if the incremental backup can be deleted or not, It’s important that Cinder doesn’t allow this backup to be deleted since ‘Incremental backups exist for this backup’. Currently, they must have a try to know it. So if there is a flag to indicate the backup can’t be deleted or not, it will bring more convenience to user and reduce API call. 3. Enriching the notification system via reporting to Ceilometer, add parent_id to report

Use Cases

It’s useful for 3rd party billing system to distinguish the full backup and incremental backup, as using different size of storage space, they could have different fee for full and incremental backups.

Proposed change

1. When show single backup detail, cinder-api needs to judge if this backup is a full backup or not checking backup[‘parent_id’]. 2. If it’s an incremental backup, judge if this backup has dependent backups like we do in process of delete backup. 3. Then add ‘is_incremental=True’ and ‘has_dependent_backups=True/False’ to response body. 4. Add parent_id to notification system.

Alternatives

None.

Data model impact

None.

REST API impact

The response body of show incremental backup detail is changed like this:

::

{

“backup”: {: ……, “is_incremental”: True/False, “has_dependent_backups”: True/False

}

}

If there is full backup, the is_incremental flag will be False. And has_dependent_backups will be True if the full backup has dependent backups.

Security impact

None

Notifications impact

Add parent_id to backup notification.

Other end user impact

End user can get more info about incremental backup. Enhance user experience.

Performance Impact

Because we add an additional judgment for dependent backups. We can eliminate performance impact by adding index to backup table and counting the number of dependent backups to make judgment in SQL.

IPv6 Impact

None.

Other deployer impact

None.

Developer impact

None.

Community Impact

None.

Implementation

Assignee(s)

wanghao

Work Items

Add querying and judging code in cinder-api.
Add parent_id to notification system.
Add unit tests.

Dependencies

None

Testing

Unit tests are needed to ensure response is working correctly.

Documentation Impact

1. Cloud admin documentations will be updated to introduce the changes: http://docs.openstack.org/admin-guide/blockstorage_volume_backups.html

2. API ref will be also updated for backups: http://developer.openstack.org/api-ref-blockstorage-v2.html

References

None

Add ability for Cinder backend to report discard/unmap/trim

Tue, 11 Aug 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-backend-report-discard

Currently, libvirt/qemu has support for a discard option when attaching a volume to an instance. With this feature, the unmap/trim command can be sent from guest to the physical storage device.

A cinder back-end should report a connection capability that can be used to instruct Nova how to attach a volume.

This spec aims to add the assist to support enabling discard for cinder volumes.

Problem description

Currently there is no way for Nova to know if a Cinder back end supports discard/trim/unmap functionality. This aims to provide that info to Nova.

Use Cases

If a Cinder backed uses media that can make use of discard functionality there should be a way to do this. This will improve long term performance of such back ends.

Proposed change

Any driver which wants to support this will just add an entry to the properties returned by intialize_connection.

“discard”: True,

An example from the Pure Storage driver would be

properties = {
    "driver_volume_type": "iscsi",
    "data": {
        "target_iqn": target_port["iqn"],
        "target_portal": target_port["portal"],
        "target_lun": connection["lun"],
        "target_discovered": True,
        "access_mode": "rw",
        "discard": True,
    },
}

Alternatives

Alternatives include adding a setting to cinder.conf for each backend that supports this. This includes a manual step that would be more error prone.

Data model impact

None

REST API impact

Documentation on the response to ‘initialize_connection’ will need to be updated with notes explaining that this new info is provided.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There will be a performance gain for back ends that benefit from having discard functionality.

See https://en.wikipedia.org/wiki/Trim_(computing) for more info.

Other deployer impact

Deployers will need to add some properties to Glance images to make this work. Once an instance is booted from the modified image the discard capability reported by the driver will be activated.

hw_scsi_model=virtio-scsi hw_disk_bus=scsi

glance image-update –property hw_scsi_model=virtio-scsi: –property hw_disk_bus=scsi <image-uuid>

This causes the correct controller to be created on the instance via code changes here https://review.openstack.org/#/c/70263/. Once the correct controller is in place the unmap functionality specified by the Cinder backend will be utilized.

A secondary volume attachment will be treated separately and will honor the discard settings provided at connection time.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

Daniel Wilson

Work Items

Add the “discard”: True line to the Pure Storage driver.

Implement the funcionality in Nova as well. https://review.openstack.org/#/c/205726/1

Dependencies

None

Testing

Tempest tests will not be needed for this spec. Pure Storage CI will test this code path with existing tempest tests.

Documentation Impact

Documentation on the response to ‘initialize_connection’ will need to be updated with notes explaining that this new info is provided.

References

https://en.wikipedia.org/wiki/Trim_(computing)

Brick fetch volume paths API

Tue, 21 Jul 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/brick-fetch-paths

The Brick (os-brick) library currently has the knowledge of how to discover and remove volumes from the scsi subsystem on a host. Every connector object has the ability to return a list of existing paths on the system for a given volume, as well as the location where those volume paths live. This spec’s purpose is to detail a formal API for fetching the existing paths for a volume, as well as the search path, and the list of all existing volumes for a connector. The purpose of this is to allow the verification of volume paths after attachment, as well as verification of a volume being removed after detachment. Once we have this API in place, then it’s possible to automate the verification of attaches and detaches for complex operations.

Problem description

Historically in OpenStack, we haven’t had a way of automatically verifying that the detach process doesn’t leave behind orphaned volumes. I define orphaned volumes as volume paths on a system that were associated with a volume, that has been detached.

This can cause problems later on when volumes are attached to that host. Some SAN arrays reuse iqn’s and or LUN ids for volumes. This can get a host in the state where a volume on the host conflicts with an orphaned volume path on the host, leading to corruption with i/o.

Use Cases

The primary use case is for fetching the list of existing paths for a volume on a system after attach and detach.

Proposed change

Add new APIs to each os-brick connector to 1) fetch the existing paths on the system for a given volume. 2) fetch all of the volume paths for a connector.

Flask based API script. Add a new flask script that exposes these API calls that can be run in a test environment. This flask script is needed in order to automatically verify the success of detach during nova live migration which happens on multiple compute hosts. This flask script is only intended on being used during jenkins tests and gate checking.

Alternatives

Someone can write individual verification scripts, outside of os-brick, for each type of volume (connector), but won’t benefit from future fixes in connectors.

Data model impact

None

REST API impact

None

Security impact

os-brick already returns a single path entry after connect_volume calls. What os-brick doesn’t do today is return a list of all volume paths associated with a connector.

It’s noted here that the flask script as described in the proposed change, should not be run/used on a production/live system. This is intended for check/gate testing only.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Connector developers will be required to implement the new APIs for their connector, even if they are stubs.

Implementation

Assignee(s)

Primary assignee:: walter-boring
Other contributors:: anthony-mic-lee

Work Items

I have a Work In Progress patch already up in gerrit for the API work. We need to add a follow up patch that creates the flask script for remotely calling the methods for the connector.

Dependencies

None

Testing

Standard unit tests. I have also done manual testing with it. Eventually, we would like to add verification in check/gate testing for volume attaches as well. For each connector object we should have 3rd party CI to test the new APIs as well.

Documentation Impact

The code is documented.

References

The existing patch in gerrit * https://review.openstack.org/#/c/199764/

Ability to update volume type public status

Mon, 20 Jul 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/volume-types-public-update

This proposal is to add the ability to update volume type is_public status.

Problem description

Currently, the v2 volume type update api doesn’t support updating volume type’s public status. All volume types created are public by default if not specified, It is not possible to update a existing volume_type is_public status. It is necessary to add updating public status for volume type. If a volume type updated from public to private, the volumes created with this type will not be affected, but the user without access will not be able to create volume with this type anymore.

Use Cases

Suppose the admin created a volume type. And he/she wants to make the volume type not public and add access to specified projects.

Proposed change

Modify volume_type update API adding is_public property support.

Alternatives

Data model impact

REST API impact

Volume type update change

Update volume type API * V2/<tenant id>/types/volume_type_id * Method: PUT * JSON schema definition for V2:
```
{
    "volume_type":
    {
        "name": "test_type",
        "description": "Test volume type",
        "is_public": "False" # new
    }
}
```
- In the existing update volume type API, add a new parameter “is_public” to allow updating public status for volume type.

Security impact

Notifications impact

Other end user impact

python-cinderclient needs to be changed to support the modified API.

Update volume type cinder type-update –name <name> –description <description> –is-public <is-public>

Performance Impact

Other deployer impact

Developer impact

Implementation

Assignee(s)

Primary assignee:: liyingjun

Other contributors:

Work Items

API change: * Modify Update Volume Type API.

Dependencies

Testing

New unit tests will be added to test the changed code.

Documentation Impact

Documentation changes are needed.

References

Cinder API WSGI application under Apache/Nginx

Wed, 17 Jun 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/non-eventlet-wsgi-app.

Cinder API uses eventelt as a webserver and WSGI application managing. Eventlet provides own WSGI application to provide web server functionality.

Problem description

Cinder API is deployed in other way as a common web application. Apache/Nginx is generally used web servers for REST API application.
Cinder API is runned as a separate service. It means that cloud operator need to configure some software to monitor that Cinder API is running.
Apache/Nginx works better under the real heavy load then eventlet.

Use Cases

Deploy Cinder API with Apache/Nginx like an any other web applicaction.
Deploy Cinder API with Apache/Nginx to have load balancing and web applicaion
monitoring out of the box. E.g. Nginx/uWSGI can restart backend service if it stopped by any case.

Proposed change

Provide WSGI application based on used web-framework instead of eventlet. Leave eventlet-based WSGI application as a default option and make it configurable.

Alternatives

Leave as is and use eventlet for REST API web serving. Use something like haproxy for API requests load balancing and some watchdog to restart Cinder API service after shutdown.

Data model impact

None.

REST API impact

None.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Potential performance impact could be present if we will have a lot of requests to Cinder API. Performance impact will be tested with Rally.

Other deployer impact

Deployers should configure Apache/Nginx/etc and WSGI module to handle requests to Cinder API. By default, Cinder API will use eventlet and no deployer impact will be.

No new configuration options for Cinder will be introduced.

New deployment mode should be supported by Chef cookbooks and Puppet manifests.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Ivan Kolodyazhny <e0ne>
Other contributors:: Anton Arefiev <aarefiev>

Work Items

Implement WSGI application based on webob framework.
Test performance impact with Rally.
Write documentation how to run Cinder API with Apache/Nginx.
Implement configuration option in Devstack to support new deployment mode.
Make sure usage of eventlet doesn’t break WSGI in Nginx/Apache.
Start cross-project initiative to implement this in oslo.

Dependencies

None

Testing

Functional tests for new deployment mode will be implemented. We need to test this feature on every commit on infra with CI.

Documentation Impact

Administrators Guide will be updated.

References

Volume state enforcer

Tue, 14 Apr 2015 00:00:00

https://blueprints.launchpad.net/cinder/+spec/cinder-state-enforcer

Concurrent resource access in cinder is a problem that has caused resource corruption when simultaneous resources are mutated on by multiple cinder entrypoints (api and manager for example). In Icehouse there has been some addition & usage of locks around manager functions to queue up those requests when a resource is being simultaneous worked on by multiple functions (this stops one of those operations from concurrently mutating the underlying resource). Sadly this is more of a sledgehammer approach and hides the symptoms of the problem and makes it non-obvious when debugging what other requests are queued up behind the lock (or why dead-locking is occurring, if and when it does).

To help alleviate and hopefully solve this problem we will try to attack some of these issues in a different manner, integrating a allowed state transition table into the create_volume workflow and doing strategic state transitions and aborting/erroring out when these state transitions are not allowed. In the future this will help create a concrete set of well defined states and transitions for other workflows as well (and will make it clear while looking at code and during debugging which transitions are allowed at the same time and what transitions are actively occurring).

Problem description

A high-level description of the problem:

Concurrent resource mutation, bad (EOM).

More detailed description:

Locks in cinder are being added to protect against simultaneous resource modification, for example in create_volume, attach_volume, delete_volume, detach_volume... a external lock is acquired in the manager with name volume_id, f.__name__. This has helped make the manager more safe to concurrent resource access but the initial goal of this was for it to only be a temporary solution to a wider problem. One of the issues with this mechanism is that it is not using a DLM (distributed lock manager) but only a local filesystem lock instead. This means that a cinder-api service can mutate the resource (or initiate a request to do this) while a second mutation is actively in flight. When a single manager is active this will work out (since one of the in flight requests will backup behind the external lock). This solves the problem when a single master manager is running; yet this is an atypical deployment pattern and should not be recommended as the way to deploy and run cinder (it should be horizontally scalable so that there can be X active managers, where X is > 1). We need some other type of solution that scales horizontally but also solves the same end goal (disallowing simultaneous resource mutation by X entities at the same time).

Since the scope of this problem is bigger (it applies to all/most operations that act on resources) we have to start somewhere so we will start by working through how this will look for the create_volume workflow. It does raise a larger question of how can this change be done in a piecemeal fashion since the other operations will still be lock dependent, and mixing state transitions and lock acquisition techniques will likely not end in a correct solution. We will have to explore how to do this in a way that is piecemeal but also does not destabilize cinder more.

Use Cases

Proposed change

Instead of acquiring local filesystem locks in the manager processes refactor the concept of a lock to instead be a set of allowed and disallowed state transitions (which is in concept similar to the internal mechanism that a lock uses anyway).

Lets take an abbreviated example of how this could work:

When a volume is requested to be created, a database record is created for this volume, in this database record there exists a field called status (in fact there exists multiple of these statuses fields, in the future these should maybe be removed?) that is used to report back to the user data about the status of the create volume request as it moves through the various components in cinder (api, scheduler, and manager).

This status itself has a expected transition diagram and itself is a starting point in determining the larger states transitions that a cinder volume create request goes through (and is allowed to go through). Instead of overriding this status field this proposal proposes to augment the data storage layer in cinder with a new resource_states table. It may be represented by something other than a table depending on where this data is stored (if zookeeper was used it would be represented as a resource tree), the only constraint that we must enforce is that we can atomically fetch and update the given state of a resource in a single atomic operation.

A potential schema could look like the following:

Resource	State	Transitioned on
7c92ee46-7a2e-4183-99c5-909f3d46a90e 7c92ee46-7a2e-4183-99c5-909f3d46a90e 7c92ee46-7a2e-4183-99c5-909f3d46a90e 7c92ee46-7a2e-4183-99c5-909f3d46a90e	CREATING_DB SCHEDULING CREATING_VOL NULL/None	2014-05-22T15 2014-05-23T15 2014-05-23T15 2014-05-24T15

This table structure will then be used (with NULL states to delimit when a request has fulfilled its set of allowed state transitions) to determine at the API level (before a request has been accepted) what a resource is currently being used for and the API server can then attempt to initiate a transition to a desired state (for example, DETACHING) and depending on if this transition is allowed (by looking at the last known state) it may fail or succeed at performing this transition. If it succeeds it continues with the rest of the workflow for the desired operation (subsequent transitions will also be made in the rest of the workflow, as needed, with the final transition being a transition to NULL/None, to denote that the operation has completed). If the transition is disallowed/fails the API request will be denied and the operation will not be allowed to make forward progress (in the future this model can be relaxed to allow for simultaneous state transitions for operations where this makes sense).

To accomplish this, in the create_volume operation there exists the usage of taskflow, which has helped decompose the workflows that volume creation goes through (it also makes it possible to resume from a prior state if the process crashes). This decomposition makes it obvious (or more obvious) where the transitions should occur and what the transitions are. The proposed path is to add in new nodes into the workflow that will perform & validate these state transitions (attempting to mutate the above resource state table) at a granular-enough level to be useful & meaningful (the transition table also can be useful for operators and developers attempting to determine what is happening inside cinder). When this is combined with notifications from taskflow about its own internal states (via notifications) the ability to decipher what is going on internally to cinder becomes very easy & provides invaluable information to users, developers and operators using & operating cinder.

Alternatives

One possibility for avoiding the above resource_states table is to use a DLM and use a similar approach that is being used with file locks in cinder. The usage of it would be similar to the usage of file locks, although there are scenarios at RPC boundaries where it would still require state transition validation. For example when a lock is released and an async RPC call is made there becomes the possibility for other async RPC calls to also be active at the same time and there would require a state transition and lock system to be used when the receivers of those RPC calls accept and perform the requested RPC operation.

Another possible solution that does not require state transitions is to not use async RPC calls but instead use sync RPC calls, and the sender would only release the DLM lock it owns after it has received confirmation that the receiver has started to process (or accepted the request). The receiver would then acquire the lock during this period when it accepts the request, ensuring that correct lock hand-off happens between the send and receiver. This would require a sensitive and hard to get correct lock hand-off code path & process (this path would need to be tested heavily to ensure correctness).

IMHO both of these alternative methods are too fragile and do not make the state transition process and diagram obvious to developers, operators, and users. This lack of information impedes cinder adoption, and makes it more difficult to recovery from (and understand) inevitable failures and operational issues.

What this does not solve

I would also like include a note to what the scope of this specification does not encompass.

It does not encompass cross-project resource usage and inconsistencies related to state transitions being done by a project using cinder (for example the initiation of a detach of a volume by nova will not be aborted early in the nova API flow, but instead will be aborted later in the workflow if cinder is performing other state transitions on that resource).
It also does not also stop cinder from deleting a volume underneath nova (aka a VM can be using a volume while cinder is deleting it).

These are larger cross-project consistency issues and will need to be solved at a higher level across the projects. It should be noted that once a project itself has a consistent set of states and transitions it becomes much easier to make cross-project consistency possible (without internal consistency cross-project resource usage might as well be discouraged/avoided).

Data model impact

See the above proposed table.

Cross-project impact

We must be careful to retain the existing API so that nova which is dependent on cinders currently visible states continues to work. This just means that we need to have a exposed mapping that nova is compatible with; while we have an internal mapping which is much more detailed and consistent.

REST API impact

Maybe in the future.

Security impact

N/A

Notifications impact

None currently, the state transition information could also be sent out to the notification system if this is desirable in the future to do so.

Other end user impact

End users should now expect more errors (or try again later) responses when performing operations concurrently on the same set of resources. Previously some of these operations may or may not have succeeded.

Performance Impact

A new table will be created in sqlalchemy and a new model will be created for this new schema. This table will be high read and write traffic (since all operations that occur in cinder will write data to it) so it might be recommended to alter the table type to a more friendly format that performs better for this tables limited usage. Since this table is relatively simple it should also be possible in the future (when correctness is achieved) to switch this table to some other backend that can optimize itself for small read/writes with little history (history is not as useful, except for operators and developers who wish to interrogate what has happened to a resource in the past).

Other deployer impact

N/A

Developer impact

Developers would likely get a lot of the benefit of this information to start since it will help them understand the states a workflow goes through (at the cinder level), combining this with the event stream that taskflow emits creates a lot of useful runtime information that can be used while running cinder or while developing cinder (where to add new state transitions in becomes more obvious when the state transitions that occur are well defined and understood).

Implementation

Assignee(s)

Primary assignee:

Harlowja

Other contributors:

DuncanT
Others?
You the person reading this?

Work Items

Determine state digram and debate what states should be used internally to cinder (the critical must-have states) and what states are more informational (DuncanT has apparently done some of this analysis).
Create database schema migration/addition for the decided upon new schema.
Create database models for new schema (and determine and discuss on how the atomic state update will be accomplished).
Identify key locations where these state transitions will occur (before or after which taskflow tasks) or at a layer outside of taskflow.
Add new tests that trigger these new state transitions and violation checks, ensuring that what is desired to occur actually occurs.
Simultaneously work on creating a model inside of taskflow that can help other projects avoid recreating chunks of the above code for there own similar needs/use-cases.
Test like crazy.
- Do load-testing/concurrency-testing (using rally or tempest) to verify the improvement has helped and not hurt cinder.

Milestones

J/3 into K (this is likely not a short-term specification).

Dependencies

N/A

Testing

Since this change affects how cinder operates at a low level, it will require a good amount of testing to verify that concurrent operations are disallowed. Currently tempest may not be the best way to test these concurrent operations since to my knowledge it does not run in parallel (and only when it runs in a controlled parallel process can u find these concurrency issues). So the way to test these concurrency issues needs to be determined (is rally the way to go here, using its concurrent scenarios to probe that this feature works?).

Documentation Impact

There may be new documentation required to explain why operations that were allowed to occur concurrently are no longer allowed to occur concurrently since this new state transition will be more strict as to what can and what can not occur at the same time.

It will also become possible to start to form documents like taskflow states that show exactly what the internals of cinder are doing and what the allowed state transitions (aka the cinder reference operation states) are.

References

Summit discussion/session:

https://etherpad.openstack.org/p/juno-cinder-state-and-workflow-management